[==========] Running 57 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 31 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765832506145157-29124-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Dec 15 21:01:46 dist-test-slave-bjsf krb5kdc[29131](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Dec 15 21:01:46 dist-test-slave-bjsf krb5kdc[29131](info): set up 2 sockets
Dec 15 21:01:46 dist-test-slave-bjsf krb5kdc[29131](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20251215 21:01:47.975855 29124 mini_kdc.cc:121] Time spent starting KDC: real 1.812s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765832506145157-29124-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765832506145157-29124-0/krb5kdc/test-user.keytab.
Dec 15 21:01:48 dist-test-slave-bjsf krb5kdc[29131](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765832508, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-12-15T21:01:48Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-12-15T21:01:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.113.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.113.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.113.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765832506145157-29124-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.28.113.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765832506145157-29124-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.28.113.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.28.113.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.28.113.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765832506145157-29124-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.28.113.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765832506145157-29124-0/krb5kdc/kudu.keytab.
I20251215 21:01:48.099022 29124 external_mini_cluster.cc:1366] Running /tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu
/tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765832506145157-29124-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765832506145157-29124-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765832506145157-29124-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765832506145157-29124-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.113.62:40209
--webserver_interface=127.28.113.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765832506145157-29124-0/krb5kdc/kudu.keytab
--principal=kudu/127.28.113.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.28.113.20:33253
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.113.62:40209
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765832506145157-29124-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765832506145157-29124-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765832506145157-29124-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251215 21:01:48.203341 29147 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251215 21:01:48.203603 29147 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251215 21:01:48.203660 29147 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251215 21:01:48.206844 29147 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251215 21:01:48.206912 29147 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251215 21:01:48.206938 29147 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251215 21:01:48.206969 29147 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251215 21:01:48.206996 29147 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251215 21:01:48.211550 29147 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.113.20:33253
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765832506145157-29124-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765832506145157-29124-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.113.62:40209
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765832506145157-29124-0/krb5kdc/kudu.keytab
--principal=kudu/127.28.113.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.113.62:40209
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765832506145157-29124-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.113.62
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29147
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765832506145157-29124-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision dc741732bb6a40213e11a13869fad06a49a209a3
build type DEBUG
built by None at 15 Dec 2025 20:43:14 UTC on 24a791456cd2
build id 9530
I20251215 21:01:48.212639 29147 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251215 21:01:48.213522 29147 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251215 21:01:48.219194 29155 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251215 21:01:48.219172 29153 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251215 21:01:48.219233 29147 server_base.cc:1047] running on GCE node
W20251215 21:01:48.219172 29152 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251215 21:01:48.219786 29147 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251215 21:01:48.220723 29147 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251215 21:01:48.221870 29147 hybrid_clock.cc:648] HybridClock initialized: now 1765832508221862 us; error 27 us; skew 500 ppm
Dec 15 21:01:48 dist-test-slave-bjsf krb5kdc[29131](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765832508, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.113.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251215 21:01:48.224325 29147 init.cc:377] Logged in from keytab as kudu/127.28.113.62@KRBTEST.COM (short username kudu)
I20251215 21:01:48.225320 29147 webserver.cc:492] Webserver started at http://127.28.113.62:34189/ using document root <none> and password file <none>
I20251215 21:01:48.225875 29147 fs_manager.cc:362] Metadata directory not provided
I20251215 21:01:48.225926 29147 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251215 21:01:48.226096 29147 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251215 21:01:48.227728 29147 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765832506145157-29124-0/minicluster-data/master-0/data/instance:
uuid: "bab2a0aa2e9842fea70f66aaa3237b25"
format_stamp: "Formatted at 2025-12-15 21:01:48 on dist-test-slave-bjsf"
server_key: "cbf900cab344ff4274cfc098034622a1"
server_key_iv: "b8af64fd968fceb9d1e1306de7c7aaff"
server_key_version: "encryptionkey@0"
I20251215 21:01:48.228191 29147 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765832506145157-29124-0/minicluster-data/master-0/wal/instance:
uuid: "bab2a0aa2e9842fea70f66aaa3237b25"
format_stamp: "Formatted at 2025-12-15 21:01:48 on dist-test-slave-bjsf"
server_key: "cbf900cab344ff4274cfc098034622a1"
server_key_iv: "b8af64fd968fceb9d1e1306de7c7aaff"
server_key_version: "encryptionkey@0"
I20251215 21:01:48.231334 29147 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.001s sys 0.001s
I20251215 21:01:48.233390 29162 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251215 21:01:48.234325 29147 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20251215 21:01:48.234454 29147 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765832506145157-29124-0/minicluster-data/master-0/data,/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765832506145157-29124-0/minicluster-data/master-0/wal
uuid: "bab2a0aa2e9842fea70f66aaa3237b25"
format_stamp: "Formatted at 2025-12-15 21:01:48 on dist-test-slave-bjsf"
server_key: "cbf900cab344ff4274cfc098034622a1"
server_key_iv: "b8af64fd968fceb9d1e1306de7c7aaff"
server_key_version: "encryptionkey@0"
I20251215 21:01:48.234534 29147 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765832506145157-29124-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765832506145157-29124-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765832506145157-29124-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251215 21:01:48.250823 29147 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251215 21:01:48.257253 29147 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251215 21:01:48.257437 29147 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251215 21:01:48.264286 29147 rpc_server.cc:307] RPC server started. Bound to: 127.28.113.62:40209
I20251215 21:01:48.264282 29214 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.28.113.62:40209 every 8 connection(s)
I20251215 21:01:48.265268 29147 server_base.cc:1179] Dumped server information to /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765832506145157-29124-0/minicluster-data/master-0/data/info.pb
I20251215 21:01:48.267902 29215 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 21:01:48.273265 29215 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bab2a0aa2e9842fea70f66aaa3237b25: Bootstrap starting.
I20251215 21:01:48.274049 29124 external_mini_cluster.cc:1428] Started /tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu as pid 29147
I20251215 21:01:48.274186 29124 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765832506145157-29124-0/minicluster-data/master-0/wal/instance
I20251215 21:01:48.275391 29124 external_mini_cluster.cc:1468] Setting key e1d32ae0996ed5685ee5eab2296c088b
I20251215 21:01:48.276032 29215 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bab2a0aa2e9842fea70f66aaa3237b25: Neither blocks nor log segments found. Creating new log.
I20251215 21:01:48.276859 29215 log.cc:826] T 00000000000000000000000000000000 P bab2a0aa2e9842fea70f66aaa3237b25: Log is configured to *not* fsync() on all Append() calls
I20251215 21:01:48.279012 29215 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bab2a0aa2e9842fea70f66aaa3237b25: No bootstrap required, opened a new log
I20251215 21:01:48.282199 29215 raft_consensus.cc:359] T 00000000000000000000000000000000 P bab2a0aa2e9842fea70f66aaa3237b25 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bab2a0aa2e9842fea70f66aaa3237b25" member_type: VOTER last_known_addr { host: "127.28.113.62" port: 40209 } }
I20251215 21:01:48.282416 29215 raft_consensus.cc:385] T 00000000000000000000000000000000 P bab2a0aa2e9842fea70f66aaa3237b25 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251215 21:01:48.282500 29215 raft_consensus.cc:740] T 00000000000000000000000000000000 P bab2a0aa2e9842fea70f66aaa3237b25 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bab2a0aa2e9842fea70f66aaa3237b25, State: Initialized, Role: FOLLOWER
I20251215 21:01:48.282977 29215 consensus_queue.cc:260] T 00000000000000000000000000000000 P bab2a0aa2e9842fea70f66aaa3237b25 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bab2a0aa2e9842fea70f66aaa3237b25" member_type: VOTER last_known_addr { host: "127.28.113.62" port: 40209 } }
I20251215 21:01:48.283118 29215 raft_consensus.cc:399] T 00000000000000000000000000000000 P bab2a0aa2e9842fea70f66aaa3237b25 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251215 21:01:48.283177 29215 raft_consensus.cc:493] T 00000000000000000000000000000000 P bab2a0aa2e9842fea70f66aaa3237b25 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251215 21:01:48.283265 29215 raft_consensus.cc:3060] T 00000000000000000000000000000000 P bab2a0aa2e9842fea70f66aaa3237b25 [term 0 FOLLOWER]: Advancing to term 1
I20251215 21:01:48.284325 29215 raft_consensus.cc:515] T 00000000000000000000000000000000 P bab2a0aa2e9842fea70f66aaa3237b25 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bab2a0aa2e9842fea70f66aaa3237b25" member_type: VOTER last_known_addr { host: "127.28.113.62" port: 40209 } }
I20251215 21:01:48.284691 29215 leader_election.cc:304] T 00000000000000000000000000000000 P bab2a0aa2e9842fea70f66aaa3237b25 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bab2a0aa2e9842fea70f66aaa3237b25; no voters:
I20251215 21:01:48.284994 29215 leader_election.cc:290] T 00000000000000000000000000000000 P bab2a0aa2e9842fea70f66aaa3237b25 [CANDIDATE]: Term 1 election: Requested vote from peers
I20251215 21:01:48.285089 29220 raft_consensus.cc:2804] T 00000000000000000000000000000000 P bab2a0aa2e9842fea70f66aaa3237b25 [term 1 FOLLOWER]: Leader election won for term 1
I20251215 21:01:48.285274 29220 raft_consensus.cc:697] T 00000000000000000000000000000000 P bab2a0aa2e9842fea70f66aaa3237b25 [term 1 LEADER]: Becoming Leader. State: Replica: bab2a0aa2e9842fea70f66aaa3237b25, State: Running, Role: LEADER
I20251215 21:01:48.285553 29220 consensus_queue.cc:237] T 00000000000000000000000000000000 P bab2a0aa2e9842fea70f66aaa3237b25 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bab2a0aa2e9842fea70f66aaa3237b25" member_type: VOTER last_known_addr { host: "127.28.113.62" port: 40209 } }
I20251215 21:01:48.286012 29215 sys_catalog.cc:565] T 00000000000000000000000000000000 P bab2a0aa2e9842fea70f66aaa3237b25 [sys.catalog]: configured and running, proceeding with master startup.
I20251215 21:01:48.287220 29221 sys_catalog.cc:455] T 00000000000000000000000000000000 P bab2a0aa2e9842fea70f66aaa3237b25 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bab2a0aa2e9842fea70f66aaa3237b25" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bab2a0aa2e9842fea70f66aaa3237b25" member_type: VOTER last_known_addr { host: "127.28.113.62" port: 40209 } } }
I20251215 21:01:48.287333 29221 sys_catalog.cc:458] T 00000000000000000000000000000000 P bab2a0aa2e9842fea70f66aaa3237b25 [sys.catalog]: This master's current role is: LEADER
I20251215 21:01:48.287595 29222 sys_catalog.cc:455] T 00000000000000000000000000000000 P bab2a0aa2e9842fea70f66aaa3237b25 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bab2a0aa2e9842fea70f66aaa3237b25. Latest consensus state: current_term: 1 leader_uuid: "bab2a0aa2e9842fea70f66aaa3237b25" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bab2a0aa2e9842fea70f66aaa3237b25" member_type: VOTER last_known_addr { host: "127.28.113.62" port: 40209 } } }
I20251215 21:01:48.287678 29222 sys_catalog.cc:458] T 00000000000000000000000000000000 P bab2a0aa2e9842fea70f66aaa3237b25 [sys.catalog]: This master's current role is: LEADER
I20251215 21:01:48.288002 29229 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
Dec 15 21:01:48 dist-test-slave-bjsf krb5kdc[29131](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765832508, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.113.62@KRBTEST.COM
I20251215 21:01:48.290688 29229 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251215 21:01:48.295410 29218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:01:48.277785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42048 (local address 127.28.113.62:40209)
1215 21:01:48.278207 (+ 422us) server_negotiation.cc:207] Beginning negotiation
1215 21:01:48.278220 (+ 13us) server_negotiation.cc:400] Waiting for connection header
1215 21:01:48.278842 (+ 622us) server_negotiation.cc:408] Connection header received
1215 21:01:48.281817 (+ 2975us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1215 21:01:48.281831 (+ 14us) server_negotiation.cc:462] Received NEGOTIATE request from client
1215 21:01:48.282100 (+ 269us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1215 21:01:48.282354 (+ 254us) server_negotiation.cc:227] Negotiated authn=SASL
1215 21:01:48.284535 (+ 2181us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:01:48.285658 (+ 1123us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1215 21:01:48.286493 (+ 835us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:01:48.286755 (+ 262us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:01:48.290820 (+ 4065us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1215 21:01:48.290837 (+ 17us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1215 21:01:48.290850 (+ 13us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1215 21:01:48.290888 (+ 38us) server_negotiation.cc:923] calling sasl_server_start()
1215 21:01:48.292853 (+ 1965us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1215 21:01:48.293292 (+ 439us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1215 21:01:48.293296 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1215 21:01:48.293300 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
1215 21:01:48.293365 (+ 65us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1215 21:01:48.293606 (+ 241us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1215 21:01:48.293609 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1215 21:01:48.293610 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1215 21:01:48.293904 (+ 294us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1215 21:01:48.294095 (+ 191us) server_negotiation.cc:1036] Waiting for connection context
1215 21:01:48.294516 (+ 421us) server_negotiation.cc:300] Negotiation successful
1215 21:01:48.294752 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":272,"thread_start_us":124,"threads_started":1}
I20251215 21:01:48.296159 29229 catalog_manager.cc:1357] Generated new cluster ID: ca3f752a03064ad8bacc856bf6b3e2f8
I20251215 21:01:48.296408 29229 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251215 21:01:48.312493 29229 catalog_manager.cc:1380] Generated new certificate authority record
I20251215 21:01:48.313480 29229 catalog_manager.cc:1514] Loading token signing keys...
I20251215 21:01:48.328469 29229 catalog_manager.cc:6027] T 00000000000000000000000000000000 P bab2a0aa2e9842fea70f66aaa3237b25: Generated new TSK 0
I20251215 21:01:48.329198 29229 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.113.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.113.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.113.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765832506145157-29124-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.28.113.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765832506145157-29124-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.28.113.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.28.113.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.28.113.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765832506145157-29124-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.28.113.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765832506145157-29124-0/krb5kdc/kudu.keytab.
I20251215 21:01:48.380111 29124 external_mini_cluster.cc:1366] Running /tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu
/tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765832506145157-29124-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765832506145157-29124-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765832506145157-29124-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765832506145157-29124-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.113.1:0
--local_ip_for_outbound_sockets=127.28.113.1
--webserver_interface=127.28.113.1
--webserver_port=0
--tserver_master_addrs=127.28.113.62:40209
--keytab_file=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765832506145157-29124-0/krb5kdc/kudu.keytab
--principal=kudu/127.28.113.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.28.113.20:33253
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765832506145157-29124-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765832506145157-29124-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765832506145157-29124-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251215 21:01:48.506871 29243 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251215 21:01:48.507095 29243 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251215 21:01:48.507154 29243 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251215 21:01:48.510383 29243 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251215 21:01:48.510447 29243 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251215 21:01:48.510538 29243 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.113.1
I20251215 21:01:48.514680 29243 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.113.20:33253
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765832506145157-29124-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765832506145157-29124-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765832506145157-29124-0/krb5kdc/kudu.keytab
--principal=kudu/127.28.113.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.113.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765832506145157-29124-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.113.1
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.28.113.62:40209
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29243
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.113.1
--log_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765832506145157-29124-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision dc741732bb6a40213e11a13869fad06a49a209a3
build type DEBUG
built by None at 15 Dec 2025 20:43:14 UTC on 24a791456cd2
build id 9530
I20251215 21:01:48.515631 29243 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251215 21:01:48.516445 29243 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251215 21:01:48.522511 29249 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251215 21:01:48.522483 29248 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251215 21:01:48.522593 29243 server_base.cc:1047] running on GCE node
W20251215 21:01:48.522477 29251 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251215 21:01:48.523037 29243 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251215 21:01:48.523535 29243 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251215 21:01:48.524693 29243 hybrid_clock.cc:648] HybridClock initialized: now 1765832508524675 us; error 33 us; skew 500 ppm
Dec 15 21:01:48 dist-test-slave-bjsf krb5kdc[29131](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765832508, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.113.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251215 21:01:48.527061 29243 init.cc:377] Logged in from keytab as kudu/127.28.113.1@KRBTEST.COM (short username kudu)
I20251215 21:01:48.528095 29243 webserver.cc:492] Webserver started at http://127.28.113.1:45885/ using document root <none> and password file <none>
I20251215 21:01:48.528599 29243 fs_manager.cc:362] Metadata directory not provided
I20251215 21:01:48.528645 29243 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251215 21:01:48.528810 29243 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251215 21:01:48.530556 29243 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765832506145157-29124-0/minicluster-data/ts-0/data/instance:
uuid: "2d6ed65f02c5489ca5098cb3cb4fbaba"
format_stamp: "Formatted at 2025-12-15 21:01:48 on dist-test-slave-bjsf"
server_key: "305ceb4c1fbaeed4d7c214267bbc3d8f"
server_key_iv: "3c67fbb775bfd4c06388c25fa52ce67a"
server_key_version: "encryptionkey@0"
I20251215 21:01:48.531023 29243 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765832506145157-29124-0/minicluster-data/ts-0/wal/instance:
uuid: "2d6ed65f02c5489ca5098cb3cb4fbaba"
format_stamp: "Formatted at 2025-12-15 21:01:48 on dist-test-slave-bjsf"
server_key: "305ceb4c1fbaeed4d7c214267bbc3d8f"
server_key_iv: "3c67fbb775bfd4c06388c25fa52ce67a"
server_key_version: "encryptionkey@0"
I20251215 21:01:48.534271 29243 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20251215 21:01:48.536347 29258 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251215 21:01:48.537281 29243 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20251215 21:01:48.537374 29243 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765832506145157-29124-0/minicluster-data/ts-0/data,/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765832506145157-29124-0/minicluster-data/ts-0/wal
uuid: "2d6ed65f02c5489ca5098cb3cb4fbaba"
format_stamp: "Formatted at 2025-12-15 21:01:48 on dist-test-slave-bjsf"
server_key: "305ceb4c1fbaeed4d7c214267bbc3d8f"
server_key_iv: "3c67fbb775bfd4c06388c25fa52ce67a"
server_key_version: "encryptionkey@0"
I20251215 21:01:48.537460 29243 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765832506145157-29124-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765832506145157-29124-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765832506145157-29124-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251215 21:01:48.579471 29243 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251215 21:01:48.582074 29243 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251215 21:01:48.582238 29243 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251215 21:01:48.582815 29243 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251215 21:01:48.583689 29243 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251215 21:01:48.583743 29243 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251215 21:01:48.583788 29243 ts_tablet_manager.cc:616] Registered 0 tablets
I20251215 21:01:48.583817 29243 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251215 21:01:48.592445 29243 rpc_server.cc:307] RPC server started. Bound to: 127.28.113.1:38669
I20251215 21:01:48.592480 29371 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.28.113.1:38669 every 8 connection(s)
I20251215 21:01:48.593355 29243 server_base.cc:1179] Dumped server information to /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765832506145157-29124-0/minicluster-data/ts-0/data/info.pb
I20251215 21:01:48.595733 29124 external_mini_cluster.cc:1428] Started /tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu as pid 29243
I20251215 21:01:48.595824 29124 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765832506145157-29124-0/minicluster-data/ts-0/wal/instance
I20251215 21:01:48.596089 29124 external_mini_cluster.cc:1468] Setting key 1a76c1663590c4fefde83e0c519617a5
Dec 15 21:01:48 dist-test-slave-bjsf krb5kdc[29131](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765832508, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.113.1@KRBTEST.COM for kudu/127.28.113.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.28.113.2@KRBTEST.COM; defaulting to no policy
I20251215 21:01:48.607501 29218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:01:48.594935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.113.1:49049 (local address 127.28.113.62:40209)
1215 21:01:48.595080 (+ 145us) server_negotiation.cc:207] Beginning negotiation
1215 21:01:48.595085 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1215 21:01:48.595758 (+ 673us) server_negotiation.cc:408] Connection header received
1215 21:01:48.596646 (+ 888us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1215 21:01:48.596649 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1215 21:01:48.596697 (+ 48us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1215 21:01:48.596769 (+ 72us) server_negotiation.cc:227] Negotiated authn=SASL
1215 21:01:48.598781 (+ 2012us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:01:48.599438 (+ 657us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1215 21:01:48.600262 (+ 824us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:01:48.600522 (+ 260us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:01:48.603197 (+ 2675us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1215 21:01:48.603219 (+ 22us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1215 21:01:48.603223 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1215 21:01:48.603256 (+ 33us) server_negotiation.cc:923] calling sasl_server_start()
1215 21:01:48.605158 (+ 1902us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1215 21:01:48.605798 (+ 640us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1215 21:01:48.605802 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1215 21:01:48.605805 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
1215 21:01:48.605864 (+ 59us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1215 21:01:48.606291 (+ 427us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1215 21:01:48.606295 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1215 21:01:48.606298 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
1215 21:01:48.606491 (+ 193us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1215 21:01:48.606583 (+ 92us) server_negotiation.cc:1036] Waiting for connection context
1215 21:01:48.607208 (+ 625us) server_negotiation.cc:300] Negotiation successful
1215 21:01:48.607338 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20251215 21:01:48.608139 29374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:01:48.595222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.113.62:40209 (local address 127.28.113.1:49049)
1215 21:01:48.595633 (+ 411us) negotiation.cc:107] Waiting for socket to connect
1215 21:01:48.595667 (+ 34us) client_negotiation.cc:174] Beginning negotiation
1215 21:01:48.596450 (+ 783us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1215 21:01:48.596920 (+ 470us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1215 21:01:48.596927 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
1215 21:01:48.597408 (+ 481us) client_negotiation.cc:189] Negotiated authn=SASL
1215 21:01:48.598582 (+ 1174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:01:48.598599 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:01:48.599559 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1215 21:01:48.599563 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1215 21:01:48.600133 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:01:48.600142 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:01:48.600350 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:01:48.601011 (+ 661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1215 21:01:48.601046 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
1215 21:01:48.603025 (+ 1979us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1215 21:01:48.605312 (+ 2287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1215 21:01:48.605319 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1215 21:01:48.605332 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
1215 21:01:48.605653 (+ 321us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1215 21:01:48.606049 (+ 396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1215 21:01:48.606053 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1215 21:01:48.606055 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1215 21:01:48.606195 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1215 21:01:48.606609 (+ 414us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1215 21:01:48.606616 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1215 21:01:48.606965 (+ 349us) client_negotiation.cc:769] Sending connection context
1215 21:01:48.607158 (+ 193us) client_negotiation.cc:240] Negotiation successful
1215 21:01:48.607422 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":105,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.113.2@KRBTEST.COM" created.
I20251215 21:01:48.609222 29372 heartbeater.cc:344] Connected to a master server at 127.28.113.62:40209
I20251215 21:01:48.609534 29372 heartbeater.cc:461] Registering TS with master...
I20251215 21:01:48.610096 29372 heartbeater.cc:507] Master 127.28.113.62:40209 requested a full tablet report, sending...
I20251215 21:01:48.611730 29179 ts_manager.cc:194] Registered new tserver with Master: 2d6ed65f02c5489ca5098cb3cb4fbaba (127.28.113.1:38669)
I20251215 21:01:48.613371 29179 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.113.1@KRBTEST.COM'} at 127.28.113.1:49049
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.113.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765832506145157-29124-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.28.113.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765832506145157-29124-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.28.113.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.28.113.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.28.113.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765832506145157-29124-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.28.113.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765832506145157-29124-0/krb5kdc/kudu.keytab.
I20251215 21:01:48.639691 29124 external_mini_cluster.cc:1366] Running /tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu
/tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765832506145157-29124-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765832506145157-29124-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765832506145157-29124-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765832506145157-29124-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.113.2:0
--local_ip_for_outbound_sockets=127.28.113.2
--webserver_interface=127.28.113.2
--webserver_port=0
--tserver_master_addrs=127.28.113.62:40209
--keytab_file=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765832506145157-29124-0/krb5kdc/kudu.keytab
--principal=kudu/127.28.113.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.28.113.20:33253
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765832506145157-29124-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765832506145157-29124-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765832506145157-29124-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251215 21:01:48.743839 29379 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251215 21:01:48.744088 29379 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251215 21:01:48.744150 29379 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251215 21:01:48.747488 29379 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251215 21:01:48.747561 29379 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251215 21:01:48.747640 29379 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.113.2
I20251215 21:01:48.751756 29379 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.113.20:33253
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765832506145157-29124-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765832506145157-29124-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765832506145157-29124-0/krb5kdc/kudu.keytab
--principal=kudu/127.28.113.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.113.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765832506145157-29124-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.113.2
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.28.113.62:40209
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29379
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.113.2
--log_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765832506145157-29124-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision dc741732bb6a40213e11a13869fad06a49a209a3
build type DEBUG
built by None at 15 Dec 2025 20:43:14 UTC on 24a791456cd2
build id 9530
I20251215 21:01:48.752789 29379 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251215 21:01:48.753563 29379 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251215 21:01:48.759864 29385 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251215 21:01:48.759848 29387 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251215 21:01:48.760202 29379 server_base.cc:1047] running on GCE node
W20251215 21:01:48.759848 29384 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251215 21:01:48.760927 29379 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251215 21:01:48.761502 29379 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251215 21:01:48.762676 29379 hybrid_clock.cc:648] HybridClock initialized: now 1765832508762653 us; error 40 us; skew 500 ppm
Dec 15 21:01:48 dist-test-slave-bjsf krb5kdc[29131](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765832508, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.113.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251215 21:01:48.765303 29379 init.cc:377] Logged in from keytab as kudu/127.28.113.2@KRBTEST.COM (short username kudu)
I20251215 21:01:48.766266 29379 webserver.cc:492] Webserver started at http://127.28.113.2:38139/ using document root <none> and password file <none>
I20251215 21:01:48.766788 29379 fs_manager.cc:362] Metadata directory not provided
I20251215 21:01:48.766836 29379 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251215 21:01:48.766999 29379 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251215 21:01:48.768677 29379 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765832506145157-29124-0/minicluster-data/ts-1/data/instance:
uuid: "ba28b5d21b814505887f8fc5ae120e3c"
format_stamp: "Formatted at 2025-12-15 21:01:48 on dist-test-slave-bjsf"
server_key: "5bc18534b513937e303fe8313003141d"
server_key_iv: "7828850e10b45d0649313dfd65c6150c"
server_key_version: "encryptionkey@0"
I20251215 21:01:48.769110 29379 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765832506145157-29124-0/minicluster-data/ts-1/wal/instance:
uuid: "ba28b5d21b814505887f8fc5ae120e3c"
format_stamp: "Formatted at 2025-12-15 21:01:48 on dist-test-slave-bjsf"
server_key: "5bc18534b513937e303fe8313003141d"
server_key_iv: "7828850e10b45d0649313dfd65c6150c"
server_key_version: "encryptionkey@0"
I20251215 21:01:48.772238 29379 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20251215 21:01:48.774312 29394 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251215 21:01:48.775249 29379 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20251215 21:01:48.775344 29379 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765832506145157-29124-0/minicluster-data/ts-1/data,/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765832506145157-29124-0/minicluster-data/ts-1/wal
uuid: "ba28b5d21b814505887f8fc5ae120e3c"
format_stamp: "Formatted at 2025-12-15 21:01:48 on dist-test-slave-bjsf"
server_key: "5bc18534b513937e303fe8313003141d"
server_key_iv: "7828850e10b45d0649313dfd65c6150c"
server_key_version: "encryptionkey@0"
I20251215 21:01:48.775425 29379 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765832506145157-29124-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765832506145157-29124-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765832506145157-29124-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251215 21:01:48.793179 29379 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251215 21:01:48.795722 29379 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251215 21:01:48.795890 29379 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251215 21:01:48.796422 29379 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251215 21:01:48.797256 29379 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251215 21:01:48.797303 29379 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251215 21:01:48.797353 29379 ts_tablet_manager.cc:616] Registered 0 tablets
I20251215 21:01:48.797381 29379 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251215 21:01:48.805913 29379 rpc_server.cc:307] RPC server started. Bound to: 127.28.113.2:40175
I20251215 21:01:48.805938 29507 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.28.113.2:40175 every 8 connection(s)
I20251215 21:01:48.806843 29379 server_base.cc:1179] Dumped server information to /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765832506145157-29124-0/minicluster-data/ts-1/data/info.pb
I20251215 21:01:48.814278 29124 external_mini_cluster.cc:1428] Started /tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu as pid 29379
I20251215 21:01:48.814363 29124 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765832506145157-29124-0/minicluster-data/ts-1/wal/instance
I20251215 21:01:48.814544 29124 external_mini_cluster.cc:1468] Setting key 71ebaf1e9f39b9541a15c21b1a293e37
Dec 15 21:01:48 dist-test-slave-bjsf krb5kdc[29131](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765832508, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.113.2@KRBTEST.COM for kudu/127.28.113.62@KRBTEST.COM
I20251215 21:01:48.818784 29218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:01:48.808550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.113.2:42505 (local address 127.28.113.62:40209)
1215 21:01:48.808687 (+ 137us) server_negotiation.cc:207] Beginning negotiation
1215 21:01:48.808691 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1215 21:01:48.809342 (+ 651us) server_negotiation.cc:408] Connection header received
1215 21:01:48.810195 (+ 853us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1215 21:01:48.810199 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1215 21:01:48.810259 (+ 60us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1215 21:01:48.810331 (+ 72us) server_negotiation.cc:227] Negotiated authn=SASL
1215 21:01:48.812020 (+ 1689us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:01:48.812525 (+ 505us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1215 21:01:48.813151 (+ 626us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:01:48.813323 (+ 172us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:01:48.815435 (+ 2112us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1215 21:01:48.815449 (+ 14us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1215 21:01:48.815451 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1215 21:01:48.815476 (+ 25us) server_negotiation.cc:923] calling sasl_server_start()
1215 21:01:48.816960 (+ 1484us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1215 21:01:48.817411 (+ 451us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1215 21:01:48.817414 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1215 21:01:48.817415 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1215 21:01:48.817459 (+ 44us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1215 21:01:48.817733 (+ 274us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1215 21:01:48.817736 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1215 21:01:48.817738 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1215 21:01:48.817880 (+ 142us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1215 21:01:48.817964 (+ 84us) server_negotiation.cc:1036] Waiting for connection context
1215 21:01:48.818567 (+ 603us) server_negotiation.cc:300] Negotiation successful
1215 21:01:48.818667 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20251215 21:01:48.819619 29510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:01:48.808809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.113.62:40209 (local address 127.28.113.2:42505)
1215 21:01:48.809212 (+ 403us) negotiation.cc:107] Waiting for socket to connect
1215 21:01:48.809248 (+ 36us) client_negotiation.cc:174] Beginning negotiation
1215 21:01:48.810004 (+ 756us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1215 21:01:48.810501 (+ 497us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1215 21:01:48.810509 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
1215 21:01:48.810910 (+ 401us) client_negotiation.cc:189] Negotiated authn=SASL
1215 21:01:48.811844 (+ 934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:01:48.811859 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:01:48.812638 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1215 21:01:48.812641 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1215 21:01:48.813046 (+ 405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:01:48.813053 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:01:48.813261 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:01:48.813820 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1215 21:01:48.813843 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
1215 21:01:48.815277 (+ 1434us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1215 21:01:48.817069 (+ 1792us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1215 21:01:48.817075 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1215 21:01:48.817085 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
1215 21:01:48.817307 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1215 21:01:48.817539 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1215 21:01:48.817542 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1215 21:01:48.817544 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1215 21:01:48.817641 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1215 21:01:48.817975 (+ 334us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1215 21:01:48.817982 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1215 21:01:48.818284 (+ 302us) client_negotiation.cc:769] Sending connection context
1215 21:01:48.818517 (+ 233us) client_negotiation.cc:240] Negotiation successful
1215 21:01:48.818788 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":107,"threads_started":1}
I20251215 21:01:48.820716 29508 heartbeater.cc:344] Connected to a master server at 127.28.113.62:40209
I20251215 21:01:48.820992 29508 heartbeater.cc:461] Registering TS with master...
I20251215 21:01:48.821633 29508 heartbeater.cc:507] Master 127.28.113.62:40209 requested a full tablet report, sending...
I20251215 21:01:48.822708 29179 ts_manager.cc:194] Registered new tserver with Master: ba28b5d21b814505887f8fc5ae120e3c (127.28.113.2:40175)
WARNING: no policy specified for kudu/127.28.113.3@KRBTEST.COM; defaulting to no policy
I20251215 21:01:48.823246 29179 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.113.2@KRBTEST.COM'} at 127.28.113.2:42505
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.113.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.113.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765832506145157-29124-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.28.113.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765832506145157-29124-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.28.113.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.28.113.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.28.113.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765832506145157-29124-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.28.113.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765832506145157-29124-0/krb5kdc/kudu.keytab.
I20251215 21:01:48.855837 29124 external_mini_cluster.cc:1366] Running /tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu
/tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765832506145157-29124-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765832506145157-29124-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765832506145157-29124-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765832506145157-29124-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.113.3:0
--local_ip_for_outbound_sockets=127.28.113.3
--webserver_interface=127.28.113.3
--webserver_port=0
--tserver_master_addrs=127.28.113.62:40209
--keytab_file=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765832506145157-29124-0/krb5kdc/kudu.keytab
--principal=kudu/127.28.113.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.28.113.20:33253
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765832506145157-29124-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765832506145157-29124-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765832506145157-29124-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251215 21:01:48.956045 29515 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251215 21:01:48.956236 29515 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251215 21:01:48.956291 29515 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251215 21:01:48.959337 29515 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251215 21:01:48.959391 29515 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251215 21:01:48.959472 29515 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.113.3
I20251215 21:01:48.963438 29515 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.113.20:33253
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765832506145157-29124-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765832506145157-29124-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765832506145157-29124-0/krb5kdc/kudu.keytab
--principal=kudu/127.28.113.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.113.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765832506145157-29124-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.113.3
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.28.113.62:40209
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29515
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.113.3
--log_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765832506145157-29124-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision dc741732bb6a40213e11a13869fad06a49a209a3
build type DEBUG
built by None at 15 Dec 2025 20:43:14 UTC on 24a791456cd2
build id 9530
I20251215 21:01:48.964399 29515 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251215 21:01:48.965133 29515 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251215 21:01:48.971012 29523 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251215 21:01:48.970999 29520 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251215 21:01:48.971176 29515 server_base.cc:1047] running on GCE node
W20251215 21:01:48.970999 29521 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251215 21:01:48.971635 29515 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251215 21:01:48.972162 29515 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251215 21:01:48.973354 29515 hybrid_clock.cc:648] HybridClock initialized: now 1765832508973336 us; error 31 us; skew 500 ppm
Dec 15 21:01:48 dist-test-slave-bjsf krb5kdc[29131](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765832508, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.113.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251215 21:01:48.975752 29515 init.cc:377] Logged in from keytab as kudu/127.28.113.3@KRBTEST.COM (short username kudu)
I20251215 21:01:48.976722 29515 webserver.cc:492] Webserver started at http://127.28.113.3:38187/ using document root <none> and password file <none>
I20251215 21:01:48.977197 29515 fs_manager.cc:362] Metadata directory not provided
I20251215 21:01:48.977239 29515 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251215 21:01:48.977380 29515 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251215 21:01:48.978917 29515 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765832506145157-29124-0/minicluster-data/ts-2/data/instance:
uuid: "335746fec0e74ce6a7b04f3e29632141"
format_stamp: "Formatted at 2025-12-15 21:01:48 on dist-test-slave-bjsf"
server_key: "964c3e9211b550abebc7915c4b8e5859"
server_key_iv: "f0aa492a751c5d2f8f65987d81357184"
server_key_version: "encryptionkey@0"
I20251215 21:01:48.979311 29515 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765832506145157-29124-0/minicluster-data/ts-2/wal/instance:
uuid: "335746fec0e74ce6a7b04f3e29632141"
format_stamp: "Formatted at 2025-12-15 21:01:48 on dist-test-slave-bjsf"
server_key: "964c3e9211b550abebc7915c4b8e5859"
server_key_iv: "f0aa492a751c5d2f8f65987d81357184"
server_key_version: "encryptionkey@0"
I20251215 21:01:48.982378 29515 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20251215 21:01:48.984361 29530 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251215 21:01:48.985245 29515 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20251215 21:01:48.985330 29515 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765832506145157-29124-0/minicluster-data/ts-2/data,/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765832506145157-29124-0/minicluster-data/ts-2/wal
uuid: "335746fec0e74ce6a7b04f3e29632141"
format_stamp: "Formatted at 2025-12-15 21:01:48 on dist-test-slave-bjsf"
server_key: "964c3e9211b550abebc7915c4b8e5859"
server_key_iv: "f0aa492a751c5d2f8f65987d81357184"
server_key_version: "encryptionkey@0"
I20251215 21:01:48.985400 29515 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765832506145157-29124-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765832506145157-29124-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765832506145157-29124-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251215 21:01:49.007894 29515 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251215 21:01:49.010818 29515 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251215 21:01:49.010952 29515 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251215 21:01:49.011420 29515 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251215 21:01:49.012244 29515 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251215 21:01:49.012290 29515 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251215 21:01:49.012328 29515 ts_tablet_manager.cc:616] Registered 0 tablets
I20251215 21:01:49.012347 29515 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251215 21:01:49.021134 29515 rpc_server.cc:307] RPC server started. Bound to: 127.28.113.3:37385
I20251215 21:01:49.021168 29643 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.28.113.3:37385 every 8 connection(s)
I20251215 21:01:49.022030 29515 server_base.cc:1179] Dumped server information to /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765832506145157-29124-0/minicluster-data/ts-2/data/info.pb
Dec 15 21:01:49 dist-test-slave-bjsf krb5kdc[29131](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765832508, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.113.3@KRBTEST.COM for kudu/127.28.113.62@KRBTEST.COM
I20251215 21:01:49.030710 29124 external_mini_cluster.cc:1428] Started /tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu as pid 29515
I20251215 21:01:49.030822 29124 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765832506145157-29124-0/minicluster-data/ts-2/wal/instance
I20251215 21:01:49.031064 29124 external_mini_cluster.cc:1468] Setting key bc6614b83b9f7a81c1edbb7661a47273
I20251215 21:01:49.033881 29218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:01:49.023673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.113.3:39513 (local address 127.28.113.62:40209)
1215 21:01:49.023811 (+ 138us) server_negotiation.cc:207] Beginning negotiation
1215 21:01:49.023814 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1215 21:01:49.024482 (+ 668us) server_negotiation.cc:408] Connection header received
1215 21:01:49.025315 (+ 833us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1215 21:01:49.025328 (+ 13us) server_negotiation.cc:462] Received NEGOTIATE request from client
1215 21:01:49.025388 (+ 60us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1215 21:01:49.025459 (+ 71us) server_negotiation.cc:227] Negotiated authn=SASL
1215 21:01:49.027058 (+ 1599us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:01:49.027561 (+ 503us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1215 21:01:49.028213 (+ 652us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:01:49.028394 (+ 181us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:01:49.030396 (+ 2002us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1215 21:01:49.030414 (+ 18us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1215 21:01:49.030417 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1215 21:01:49.030443 (+ 26us) server_negotiation.cc:923] calling sasl_server_start()
1215 21:01:49.031909 (+ 1466us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1215 21:01:49.032452 (+ 543us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1215 21:01:49.032457 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1215 21:01:49.032459 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1215 21:01:49.032519 (+ 60us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1215 21:01:49.032842 (+ 323us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1215 21:01:49.032846 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1215 21:01:49.032848 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1215 21:01:49.033005 (+ 157us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1215 21:01:49.033088 (+ 83us) server_negotiation.cc:1036] Waiting for connection context
1215 21:01:49.033631 (+ 543us) server_negotiation.cc:300] Negotiation successful
1215 21:01:49.033761 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20251215 21:01:49.034488 29646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:01:49.023924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.113.62:40209 (local address 127.28.113.3:39513)
1215 21:01:49.024342 (+ 418us) negotiation.cc:107] Waiting for socket to connect
1215 21:01:49.024375 (+ 33us) client_negotiation.cc:174] Beginning negotiation
1215 21:01:49.025124 (+ 749us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1215 21:01:49.025629 (+ 505us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1215 21:01:49.025637 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
1215 21:01:49.026020 (+ 383us) client_negotiation.cc:189] Negotiated authn=SASL
1215 21:01:49.026911 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:01:49.026925 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:01:49.027671 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1215 21:01:49.027674 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1215 21:01:49.028097 (+ 423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:01:49.028105 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:01:49.028315 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:01:49.028884 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1215 21:01:49.028908 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
1215 21:01:49.030249 (+ 1341us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1215 21:01:49.032026 (+ 1777us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1215 21:01:49.032032 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1215 21:01:49.032044 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
1215 21:01:49.032348 (+ 304us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1215 21:01:49.032608 (+ 260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1215 21:01:49.032611 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1215 21:01:49.032613 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1215 21:01:49.032754 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1215 21:01:49.033099 (+ 345us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1215 21:01:49.033106 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1215 21:01:49.033365 (+ 259us) client_negotiation.cc:769] Sending connection context
1215 21:01:49.033581 (+ 216us) client_negotiation.cc:240] Negotiation successful
1215 21:01:49.033811 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":102,"threads_started":1}
I20251215 21:01:49.035504 29644 heartbeater.cc:344] Connected to a master server at 127.28.113.62:40209
I20251215 21:01:49.035748 29644 heartbeater.cc:461] Registering TS with master...
I20251215 21:01:49.036290 29644 heartbeater.cc:507] Master 127.28.113.62:40209 requested a full tablet report, sending...
I20251215 21:01:49.037223 29179 ts_manager.cc:194] Registered new tserver with Master: 335746fec0e74ce6a7b04f3e29632141 (127.28.113.3:37385)
I20251215 21:01:49.037773 29179 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.113.3@KRBTEST.COM'} at 127.28.113.3:39513
I20251215 21:01:49.045233 29124 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
Dec 15 21:01:49 dist-test-slave-bjsf krb5kdc[29131](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765832509, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Dec 15 21:01:49 dist-test-slave-bjsf krb5kdc[29131](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765832509, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.28.113.1@KRBTEST.COM
I20251215 21:01:49.068521 29651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:01:49.060012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39772 (local address 127.28.113.1:38669)
1215 21:01:49.060311 (+ 299us) server_negotiation.cc:207] Beginning negotiation
1215 21:01:49.060317 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1215 21:01:49.060339 (+ 22us) server_negotiation.cc:408] Connection header received
1215 21:01:49.060390 (+ 51us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1215 21:01:49.060394 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1215 21:01:49.060556 (+ 162us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1215 21:01:49.060676 (+ 120us) server_negotiation.cc:227] Negotiated authn=SASL
1215 21:01:49.061396 (+ 720us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:01:49.062090 (+ 694us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1215 21:01:49.062796 (+ 706us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:01:49.062968 (+ 172us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:01:49.065328 (+ 2360us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1215 21:01:49.065353 (+ 25us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1215 21:01:49.065365 (+ 12us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1215 21:01:49.065391 (+ 26us) server_negotiation.cc:923] calling sasl_server_start()
1215 21:01:49.067033 (+ 1642us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1215 21:01:49.067409 (+ 376us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1215 21:01:49.067414 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1215 21:01:49.067416 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1215 21:01:49.067464 (+ 48us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1215 21:01:49.067662 (+ 198us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1215 21:01:49.067665 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1215 21:01:49.067668 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
1215 21:01:49.067846 (+ 178us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1215 21:01:49.068053 (+ 207us) server_negotiation.cc:1036] Waiting for connection context
1215 21:01:49.068226 (+ 173us) server_negotiation.cc:300] Negotiation successful
1215 21:01:49.068367 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":202,"thread_start_us":109,"threads_started":1}
W20251215 21:01:49.069382 29286 server_base.cc:1129] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39772
Dec 15 21:01:49 dist-test-slave-bjsf krb5kdc[29131](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765832509, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Dec 15 21:01:49 dist-test-slave-bjsf krb5kdc[29131](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765832509, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.113.1@KRBTEST.COM
I20251215 21:01:49.090766 29651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:01:49.083792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39786 (local address 127.28.113.1:38669)
1215 21:01:49.083935 (+ 143us) server_negotiation.cc:207] Beginning negotiation
1215 21:01:49.083939 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1215 21:01:49.084018 (+ 79us) server_negotiation.cc:408] Connection header received
1215 21:01:49.084160 (+ 142us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1215 21:01:49.084163 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1215 21:01:49.084208 (+ 45us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1215 21:01:49.084275 (+ 67us) server_negotiation.cc:227] Negotiated authn=SASL
1215 21:01:49.085023 (+ 748us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:01:49.085504 (+ 481us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1215 21:01:49.086109 (+ 605us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:01:49.086298 (+ 189us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:01:49.088262 (+ 1964us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1215 21:01:49.088280 (+ 18us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1215 21:01:49.088283 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1215 21:01:49.088305 (+ 22us) server_negotiation.cc:923] calling sasl_server_start()
1215 21:01:49.089425 (+ 1120us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1215 21:01:49.089827 (+ 402us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1215 21:01:49.089831 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1215 21:01:49.089833 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1215 21:01:49.089875 (+ 42us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1215 21:01:49.090066 (+ 191us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1215 21:01:49.090069 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1215 21:01:49.090072 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
1215 21:01:49.090195 (+ 123us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1215 21:01:49.090308 (+ 113us) server_negotiation.cc:1036] Waiting for connection context
1215 21:01:49.090503 (+ 195us) server_negotiation.cc:300] Negotiation successful
1215 21:01:49.090618 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20251215 21:01:49.091670 29124 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu with pid 29243
I20251215 21:01:49.097077 29124 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu with pid 29379
I20251215 21:01:49.103554 29124 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu with pid 29515
I20251215 21:01:49.108390 29124 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu with pid 29147
2025-12-15T21:01:49Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (2972 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765832506145157-29124-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Dec 15 21:01:49 dist-test-slave-bjsf krb5kdc[29658](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Dec 15 21:01:49 dist-test-slave-bjsf krb5kdc[29658](info): set up 2 sockets
Dec 15 21:01:49 dist-test-slave-bjsf krb5kdc[29658](info): commencing operation
krb5kdc: starting...
W20251215 21:01:50.943708 29124 mini_kdc.cc:121] Time spent starting KDC: real 1.815s user 0.005s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765832506145157-29124-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765832506145157-29124-0/krb5kdc/test-user.keytab.
Dec 15 21:01:51 dist-test-slave-bjsf krb5kdc[29658](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765832511, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-12-15T21:01:51Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-12-15T21:01:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.113.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.113.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.113.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765832506145157-29124-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.28.113.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765832506145157-29124-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.28.113.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.28.113.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.28.113.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765832506145157-29124-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.28.113.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765832506145157-29124-0/krb5kdc/kudu.keytab.
I20251215 21:01:51.066524 29124 external_mini_cluster.cc:1366] Running /tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu
/tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765832506145157-29124-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765832506145157-29124-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765832506145157-29124-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765832506145157-29124-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.113.62:46049
--webserver_interface=127.28.113.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765832506145157-29124-0/krb5kdc/kudu.keytab
--principal=kudu/127.28.113.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.28.113.20:42723
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.113.62:46049
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765832506145157-29124-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765832506145157-29124-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765832506145157-29124-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251215 21:01:51.165395 29674 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251215 21:01:51.165623 29674 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251215 21:01:51.165692 29674 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251215 21:01:51.168848 29674 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251215 21:01:51.168907 29674 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251215 21:01:51.168938 29674 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251215 21:01:51.168963 29674 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251215 21:01:51.168994 29674 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251215 21:01:51.173018 29674 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.113.20:42723
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765832506145157-29124-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765832506145157-29124-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.113.62:46049
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765832506145157-29124-0/krb5kdc/kudu.keytab
--principal=kudu/127.28.113.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.113.62:46049
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765832506145157-29124-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.113.62
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29674
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765832506145157-29124-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision dc741732bb6a40213e11a13869fad06a49a209a3
build type DEBUG
built by None at 15 Dec 2025 20:43:14 UTC on 24a791456cd2
build id 9530
I20251215 21:01:51.173964 29674 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251215 21:01:51.174728 29674 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251215 21:01:51.179867 29682 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251215 21:01:51.180066 29674 server_base.cc:1047] running on GCE node
W20251215 21:01:51.179899 29679 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251215 21:01:51.179946 29680 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251215 21:01:51.180619 29674 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251215 21:01:51.181466 29674 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251215 21:01:51.182616 29674 hybrid_clock.cc:648] HybridClock initialized: now 1765832511182607 us; error 24 us; skew 500 ppm
Dec 15 21:01:51 dist-test-slave-bjsf krb5kdc[29658](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765832511, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.113.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251215 21:01:51.185035 29674 init.cc:377] Logged in from keytab as kudu/127.28.113.62@KRBTEST.COM (short username kudu)
I20251215 21:01:51.186012 29674 webserver.cc:492] Webserver started at http://127.28.113.62:42155/ using document root <none> and password file <none>
I20251215 21:01:51.186509 29674 fs_manager.cc:362] Metadata directory not provided
I20251215 21:01:51.186558 29674 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251215 21:01:51.186723 29674 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251215 21:01:51.188380 29674 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765832506145157-29124-0/minicluster-data/master-0/data/instance:
uuid: "76c3ef1e78ac48abbca1d1bf4138f963"
format_stamp: "Formatted at 2025-12-15 21:01:51 on dist-test-slave-bjsf"
server_key: "99f5503e01bf2bfb4eb761a825df0542"
server_key_iv: "b82d88f20a9e79bc8422579c197eaa07"
server_key_version: "encryptionkey@0"
I20251215 21:01:51.188827 29674 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765832506145157-29124-0/minicluster-data/master-0/wal/instance:
uuid: "76c3ef1e78ac48abbca1d1bf4138f963"
format_stamp: "Formatted at 2025-12-15 21:01:51 on dist-test-slave-bjsf"
server_key: "99f5503e01bf2bfb4eb761a825df0542"
server_key_iv: "b82d88f20a9e79bc8422579c197eaa07"
server_key_version: "encryptionkey@0"
I20251215 21:01:51.191864 29674 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20251215 21:01:51.193877 29689 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251215 21:01:51.194794 29674 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20251215 21:01:51.194900 29674 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765832506145157-29124-0/minicluster-data/master-0/data,/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765832506145157-29124-0/minicluster-data/master-0/wal
uuid: "76c3ef1e78ac48abbca1d1bf4138f963"
format_stamp: "Formatted at 2025-12-15 21:01:51 on dist-test-slave-bjsf"
server_key: "99f5503e01bf2bfb4eb761a825df0542"
server_key_iv: "b82d88f20a9e79bc8422579c197eaa07"
server_key_version: "encryptionkey@0"
I20251215 21:01:51.194978 29674 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765832506145157-29124-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765832506145157-29124-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765832506145157-29124-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251215 21:01:51.211820 29674 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251215 21:01:51.214407 29674 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251215 21:01:51.214569 29674 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251215 21:01:51.221465 29674 rpc_server.cc:307] RPC server started. Bound to: 127.28.113.62:46049
I20251215 21:01:51.221462 29741 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.28.113.62:46049 every 8 connection(s)
I20251215 21:01:51.222437 29674 server_base.cc:1179] Dumped server information to /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765832506145157-29124-0/minicluster-data/master-0/data/info.pb
I20251215 21:01:51.225078 29742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 21:01:51.230237 29742 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 76c3ef1e78ac48abbca1d1bf4138f963: Bootstrap starting.
I20251215 21:01:51.230728 29124 external_mini_cluster.cc:1428] Started /tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu as pid 29674
I20251215 21:01:51.230823 29124 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765832506145157-29124-0/minicluster-data/master-0/wal/instance
I20251215 21:01:51.231014 29124 external_mini_cluster.cc:1468] Setting key b3df7a142b9501d1649d4b820ff52f68
I20251215 21:01:51.232540 29742 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 76c3ef1e78ac48abbca1d1bf4138f963: Neither blocks nor log segments found. Creating new log.
I20251215 21:01:51.233186 29742 log.cc:826] T 00000000000000000000000000000000 P 76c3ef1e78ac48abbca1d1bf4138f963: Log is configured to *not* fsync() on all Append() calls
I20251215 21:01:51.234956 29742 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 76c3ef1e78ac48abbca1d1bf4138f963: No bootstrap required, opened a new log
Dec 15 21:01:51 dist-test-slave-bjsf krb5kdc[29658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765832511, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.113.62@KRBTEST.COM
I20251215 21:01:51.237828 29742 raft_consensus.cc:359] T 00000000000000000000000000000000 P 76c3ef1e78ac48abbca1d1bf4138f963 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76c3ef1e78ac48abbca1d1bf4138f963" member_type: VOTER last_known_addr { host: "127.28.113.62" port: 46049 } }
I20251215 21:01:51.238015 29742 raft_consensus.cc:385] T 00000000000000000000000000000000 P 76c3ef1e78ac48abbca1d1bf4138f963 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251215 21:01:51.238080 29742 raft_consensus.cc:740] T 00000000000000000000000000000000 P 76c3ef1e78ac48abbca1d1bf4138f963 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76c3ef1e78ac48abbca1d1bf4138f963, State: Initialized, Role: FOLLOWER
I20251215 21:01:51.238516 29742 consensus_queue.cc:260] T 00000000000000000000000000000000 P 76c3ef1e78ac48abbca1d1bf4138f963 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76c3ef1e78ac48abbca1d1bf4138f963" member_type: VOTER last_known_addr { host: "127.28.113.62" port: 46049 } }
I20251215 21:01:51.238663 29742 raft_consensus.cc:399] T 00000000000000000000000000000000 P 76c3ef1e78ac48abbca1d1bf4138f963 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251215 21:01:51.238724 29742 raft_consensus.cc:493] T 00000000000000000000000000000000 P 76c3ef1e78ac48abbca1d1bf4138f963 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251215 21:01:51.238811 29742 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 76c3ef1e78ac48abbca1d1bf4138f963 [term 0 FOLLOWER]: Advancing to term 1
I20251215 21:01:51.239794 29742 raft_consensus.cc:515] T 00000000000000000000000000000000 P 76c3ef1e78ac48abbca1d1bf4138f963 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76c3ef1e78ac48abbca1d1bf4138f963" member_type: VOTER last_known_addr { host: "127.28.113.62" port: 46049 } }
I20251215 21:01:51.240170 29742 leader_election.cc:304] T 00000000000000000000000000000000 P 76c3ef1e78ac48abbca1d1bf4138f963 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 76c3ef1e78ac48abbca1d1bf4138f963; no voters:
I20251215 21:01:51.240461 29742 leader_election.cc:290] T 00000000000000000000000000000000 P 76c3ef1e78ac48abbca1d1bf4138f963 [CANDIDATE]: Term 1 election: Requested vote from peers
I20251215 21:01:51.240818 29747 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 76c3ef1e78ac48abbca1d1bf4138f963 [term 1 FOLLOWER]: Leader election won for term 1
I20251215 21:01:51.241024 29747 raft_consensus.cc:697] T 00000000000000000000000000000000 P 76c3ef1e78ac48abbca1d1bf4138f963 [term 1 LEADER]: Becoming Leader. State: Replica: 76c3ef1e78ac48abbca1d1bf4138f963, State: Running, Role: LEADER
I20251215 21:01:51.241353 29747 consensus_queue.cc:237] T 00000000000000000000000000000000 P 76c3ef1e78ac48abbca1d1bf4138f963 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76c3ef1e78ac48abbca1d1bf4138f963" member_type: VOTER last_known_addr { host: "127.28.113.62" port: 46049 } }
I20251215 21:01:51.241550 29742 sys_catalog.cc:565] T 00000000000000000000000000000000 P 76c3ef1e78ac48abbca1d1bf4138f963 [sys.catalog]: configured and running, proceeding with master startup.
I20251215 21:01:51.242969 29748 sys_catalog.cc:455] T 00000000000000000000000000000000 P 76c3ef1e78ac48abbca1d1bf4138f963 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "76c3ef1e78ac48abbca1d1bf4138f963" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76c3ef1e78ac48abbca1d1bf4138f963" member_type: VOTER last_known_addr { host: "127.28.113.62" port: 46049 } } }
I20251215 21:01:51.243065 29748 sys_catalog.cc:458] T 00000000000000000000000000000000 P 76c3ef1e78ac48abbca1d1bf4138f963 [sys.catalog]: This master's current role is: LEADER
I20251215 21:01:51.243129 29745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:01:51.232075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57508 (local address 127.28.113.62:46049)
1215 21:01:51.232414 (+ 339us) server_negotiation.cc:207] Beginning negotiation
1215 21:01:51.232422 (+ 8us) server_negotiation.cc:400] Waiting for connection header
1215 21:01:51.232458 (+ 36us) server_negotiation.cc:408] Connection header received
1215 21:01:51.232976 (+ 518us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1215 21:01:51.232994 (+ 18us) server_negotiation.cc:462] Received NEGOTIATE request from client
1215 21:01:51.233268 (+ 274us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1215 21:01:51.233545 (+ 277us) server_negotiation.cc:227] Negotiated authn=SASL
1215 21:01:51.234343 (+ 798us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:01:51.235034 (+ 691us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1215 21:01:51.235839 (+ 805us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:01:51.236079 (+ 240us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:01:51.238329 (+ 2250us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1215 21:01:51.238349 (+ 20us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1215 21:01:51.238360 (+ 11us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1215 21:01:51.238397 (+ 37us) server_negotiation.cc:923] calling sasl_server_start()
1215 21:01:51.240375 (+ 1978us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1215 21:01:51.240862 (+ 487us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1215 21:01:51.240868 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1215 21:01:51.240873 (+ 5us) server_negotiation.cc:968] calling sasl_server_step()
1215 21:01:51.240967 (+ 94us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1215 21:01:51.241220 (+ 253us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1215 21:01:51.241224 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1215 21:01:51.241225 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1215 21:01:51.241617 (+ 392us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1215 21:01:51.241836 (+ 219us) server_negotiation.cc:1036] Waiting for connection context
1215 21:01:51.242112 (+ 276us) server_negotiation.cc:300] Negotiation successful
1215 21:01:51.242378 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":221,"thread_start_us":96,"threads_started":1}
I20251215 21:01:51.243556 29749 sys_catalog.cc:455] T 00000000000000000000000000000000 P 76c3ef1e78ac48abbca1d1bf4138f963 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 76c3ef1e78ac48abbca1d1bf4138f963. Latest consensus state: current_term: 1 leader_uuid: "76c3ef1e78ac48abbca1d1bf4138f963" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76c3ef1e78ac48abbca1d1bf4138f963" member_type: VOTER last_known_addr { host: "127.28.113.62" port: 46049 } } }
I20251215 21:01:51.244112 29749 sys_catalog.cc:458] T 00000000000000000000000000000000 P 76c3ef1e78ac48abbca1d1bf4138f963 [sys.catalog]: This master's current role is: LEADER
W20251215 21:01:51.246407 29762 catalog_manager.cc:1568] T 00000000000000000000000000000000 P 76c3ef1e78ac48abbca1d1bf4138f963: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20251215 21:01:51.246467 29762 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20251215 21:01:51.246519 29758 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251215 21:01:51.247169 29758 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251215 21:01:51.252076 29758 catalog_manager.cc:1357] Generated new cluster ID: edb035d76716441885d1a3ef7e064392
I20251215 21:01:51.252128 29758 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251215 21:01:51.268746 29758 catalog_manager.cc:1380] Generated new certificate authority record
I20251215 21:01:51.269588 29758 catalog_manager.cc:1514] Loading token signing keys...
I20251215 21:01:51.283849 29758 catalog_manager.cc:6027] T 00000000000000000000000000000000 P 76c3ef1e78ac48abbca1d1bf4138f963: Generated new TSK 0
I20251215 21:01:51.284534 29758 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.113.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.113.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.113.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765832506145157-29124-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.28.113.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765832506145157-29124-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.28.113.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.28.113.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.28.113.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765832506145157-29124-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.28.113.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765832506145157-29124-0/krb5kdc/kudu.keytab.
I20251215 21:01:51.334374 29124 external_mini_cluster.cc:1366] Running /tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu
/tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765832506145157-29124-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765832506145157-29124-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765832506145157-29124-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765832506145157-29124-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.113.1:0
--local_ip_for_outbound_sockets=127.28.113.1
--webserver_interface=127.28.113.1
--webserver_port=0
--tserver_master_addrs=127.28.113.62:46049
--keytab_file=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765832506145157-29124-0/krb5kdc/kudu.keytab
--principal=kudu/127.28.113.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.28.113.20:42723
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765832506145157-29124-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765832506145157-29124-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765832506145157-29124-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251215 21:01:51.454077 29770 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251215 21:01:51.454264 29770 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251215 21:01:51.454320 29770 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251215 21:01:51.457387 29770 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251215 21:01:51.457434 29770 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251215 21:01:51.457510 29770 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.113.1
I20251215 21:01:51.461381 29770 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.113.20:42723
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765832506145157-29124-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765832506145157-29124-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765832506145157-29124-0/krb5kdc/kudu.keytab
--principal=kudu/127.28.113.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.113.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765832506145157-29124-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.113.1
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.28.113.62:46049
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29770
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.113.1
--log_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765832506145157-29124-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision dc741732bb6a40213e11a13869fad06a49a209a3
build type DEBUG
built by None at 15 Dec 2025 20:43:14 UTC on 24a791456cd2
build id 9530
I20251215 21:01:51.462273 29770 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251215 21:01:51.463027 29770 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251215 21:01:51.468758 29775 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251215 21:01:51.468780 29778 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251215 21:01:51.468799 29776 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251215 21:01:51.468997 29770 server_base.cc:1047] running on GCE node
I20251215 21:01:51.469365 29770 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251215 21:01:51.469907 29770 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251215 21:01:51.471055 29770 hybrid_clock.cc:648] HybridClock initialized: now 1765832511471042 us; error 27 us; skew 500 ppm
Dec 15 21:01:51 dist-test-slave-bjsf krb5kdc[29658](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765832511, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.113.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251215 21:01:51.473425 29770 init.cc:377] Logged in from keytab as kudu/127.28.113.1@KRBTEST.COM (short username kudu)
I20251215 21:01:51.474362 29770 webserver.cc:492] Webserver started at http://127.28.113.1:44137/ using document root <none> and password file <none>
I20251215 21:01:51.474866 29770 fs_manager.cc:362] Metadata directory not provided
I20251215 21:01:51.474913 29770 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251215 21:01:51.475075 29770 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251215 21:01:51.476681 29770 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765832506145157-29124-0/minicluster-data/ts-0/data/instance:
uuid: "cf1e48f4eb75456c803332841127d851"
format_stamp: "Formatted at 2025-12-15 21:01:51 on dist-test-slave-bjsf"
server_key: "848496ee2bf1113203f3370280037f8a"
server_key_iv: "66a1edeefcf55dc2d56b0cd19661fa68"
server_key_version: "encryptionkey@0"
I20251215 21:01:51.477244 29770 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765832506145157-29124-0/minicluster-data/ts-0/wal/instance:
uuid: "cf1e48f4eb75456c803332841127d851"
format_stamp: "Formatted at 2025-12-15 21:01:51 on dist-test-slave-bjsf"
server_key: "848496ee2bf1113203f3370280037f8a"
server_key_iv: "66a1edeefcf55dc2d56b0cd19661fa68"
server_key_version: "encryptionkey@0"
I20251215 21:01:51.480324 29770 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20251215 21:01:51.482295 29785 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251215 21:01:51.483223 29770 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20251215 21:01:51.483318 29770 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765832506145157-29124-0/minicluster-data/ts-0/data,/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765832506145157-29124-0/minicluster-data/ts-0/wal
uuid: "cf1e48f4eb75456c803332841127d851"
format_stamp: "Formatted at 2025-12-15 21:01:51 on dist-test-slave-bjsf"
server_key: "848496ee2bf1113203f3370280037f8a"
server_key_iv: "66a1edeefcf55dc2d56b0cd19661fa68"
server_key_version: "encryptionkey@0"
I20251215 21:01:51.483395 29770 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765832506145157-29124-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765832506145157-29124-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765832506145157-29124-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251215 21:01:51.509593 29770 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251215 21:01:51.512329 29770 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251215 21:01:51.512514 29770 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251215 21:01:51.513042 29770 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251215 21:01:51.513919 29770 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251215 21:01:51.513967 29770 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251215 21:01:51.514017 29770 ts_tablet_manager.cc:616] Registered 0 tablets
I20251215 21:01:51.514042 29770 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251215 21:01:51.522233 29770 rpc_server.cc:307] RPC server started. Bound to: 127.28.113.1:42247
I20251215 21:01:51.522261 29898 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.28.113.1:42247 every 8 connection(s)
I20251215 21:01:51.523131 29770 server_base.cc:1179] Dumped server information to /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765832506145157-29124-0/minicluster-data/ts-0/data/info.pb
I20251215 21:01:51.529925 29124 external_mini_cluster.cc:1428] Started /tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu as pid 29770
I20251215 21:01:51.530026 29124 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765832506145157-29124-0/minicluster-data/ts-0/wal/instance
I20251215 21:01:51.530267 29124 external_mini_cluster.cc:1468] Setting key aeaebcc401db3b1829d91d28aa2955a0
Dec 15 21:01:51 dist-test-slave-bjsf krb5kdc[29658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765832511, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.113.1@KRBTEST.COM for kudu/127.28.113.62@KRBTEST.COM
I20251215 21:01:51.534672 29745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:01:51.524769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.113.1:42913 (local address 127.28.113.62:46049)
1215 21:01:51.524918 (+ 149us) server_negotiation.cc:207] Beginning negotiation
1215 21:01:51.524923 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1215 21:01:51.525575 (+ 652us) server_negotiation.cc:408] Connection header received
1215 21:01:51.526316 (+ 741us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1215 21:01:51.526319 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1215 21:01:51.526364 (+ 45us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1215 21:01:51.526435 (+ 71us) server_negotiation.cc:227] Negotiated authn=SASL
1215 21:01:51.527950 (+ 1515us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:01:51.528476 (+ 526us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1215 21:01:51.529038 (+ 562us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:01:51.529164 (+ 126us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:01:51.531455 (+ 2291us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1215 21:01:51.531470 (+ 15us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1215 21:01:51.531472 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1215 21:01:51.531496 (+ 24us) server_negotiation.cc:923] calling sasl_server_start()
1215 21:01:51.532910 (+ 1414us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1215 21:01:51.533415 (+ 505us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1215 21:01:51.533417 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1215 21:01:51.533419 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1215 21:01:51.533458 (+ 39us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1215 21:01:51.533744 (+ 286us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1215 21:01:51.533746 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1215 21:01:51.533747 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1215 21:01:51.533889 (+ 142us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1215 21:01:51.533954 (+ 65us) server_negotiation.cc:1036] Waiting for connection context
1215 21:01:51.534461 (+ 507us) server_negotiation.cc:300] Negotiation successful
1215 21:01:51.534557 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20251215 21:01:51.535516 29901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:01:51.525039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.113.62:46049 (local address 127.28.113.1:42913)
1215 21:01:51.525444 (+ 405us) negotiation.cc:107] Waiting for socket to connect
1215 21:01:51.525477 (+ 33us) client_negotiation.cc:174] Beginning negotiation
1215 21:01:51.526158 (+ 681us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1215 21:01:51.526563 (+ 405us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1215 21:01:51.526570 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
1215 21:01:51.526942 (+ 372us) client_negotiation.cc:189] Negotiated authn=SASL
1215 21:01:51.527803 (+ 861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:01:51.527824 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:01:51.528559 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1215 21:01:51.528562 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1215 21:01:51.528957 (+ 395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:01:51.528963 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:01:51.529119 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:01:51.529662 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1215 21:01:51.529684 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
1215 21:01:51.531316 (+ 1632us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1215 21:01:51.533014 (+ 1698us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1215 21:01:51.533021 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1215 21:01:51.533031 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
1215 21:01:51.533326 (+ 295us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1215 21:01:51.533540 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1215 21:01:51.533544 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1215 21:01:51.533546 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1215 21:01:51.533669 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1215 21:01:51.533975 (+ 306us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1215 21:01:51.533982 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1215 21:01:51.534242 (+ 260us) client_negotiation.cc:769] Sending connection context
1215 21:01:51.534447 (+ 205us) client_negotiation.cc:240] Negotiation successful
1215 21:01:51.534733 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":119,"threads_started":1}
I20251215 21:01:51.536661 29899 heartbeater.cc:344] Connected to a master server at 127.28.113.62:46049
I20251215 21:01:51.536888 29899 heartbeater.cc:461] Registering TS with master...
I20251215 21:01:51.537361 29899 heartbeater.cc:507] Master 127.28.113.62:46049 requested a full tablet report, sending...
I20251215 21:01:51.538653 29706 ts_manager.cc:194] Registered new tserver with Master: cf1e48f4eb75456c803332841127d851 (127.28.113.1:42247)
WARNING: no policy specified for kudu/127.28.113.2@KRBTEST.COM; defaulting to no policy
I20251215 21:01:51.539784 29706 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.113.1@KRBTEST.COM'} at 127.28.113.1:42913
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.113.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.113.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765832506145157-29124-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.28.113.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765832506145157-29124-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.28.113.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.28.113.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.28.113.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765832506145157-29124-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.28.113.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765832506145157-29124-0/krb5kdc/kudu.keytab.
I20251215 21:01:51.571602 29124 external_mini_cluster.cc:1366] Running /tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu
/tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765832506145157-29124-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765832506145157-29124-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765832506145157-29124-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765832506145157-29124-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.113.2:0
--local_ip_for_outbound_sockets=127.28.113.2
--webserver_interface=127.28.113.2
--webserver_port=0
--tserver_master_addrs=127.28.113.62:46049
--keytab_file=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765832506145157-29124-0/krb5kdc/kudu.keytab
--principal=kudu/127.28.113.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.28.113.20:42723
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765832506145157-29124-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765832506145157-29124-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765832506145157-29124-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251215 21:01:51.671886 29906 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251215 21:01:51.672130 29906 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251215 21:01:51.672188 29906 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251215 21:01:51.675302 29906 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251215 21:01:51.675356 29906 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251215 21:01:51.675432 29906 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.113.2
I20251215 21:01:51.679433 29906 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.113.20:42723
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765832506145157-29124-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765832506145157-29124-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765832506145157-29124-0/krb5kdc/kudu.keytab
--principal=kudu/127.28.113.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.113.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765832506145157-29124-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.113.2
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.28.113.62:46049
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29906
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.113.2
--log_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765832506145157-29124-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision dc741732bb6a40213e11a13869fad06a49a209a3
build type DEBUG
built by None at 15 Dec 2025 20:43:14 UTC on 24a791456cd2
build id 9530
I20251215 21:01:51.680423 29906 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251215 21:01:51.681176 29906 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251215 21:01:51.687150 29914 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251215 21:01:51.687163 29912 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251215 21:01:51.687291 29906 server_base.cc:1047] running on GCE node
W20251215 21:01:51.687420 29911 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251215 21:01:51.687788 29906 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251215 21:01:51.688295 29906 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251215 21:01:51.689442 29906 hybrid_clock.cc:648] HybridClock initialized: now 1765832511689427 us; error 32 us; skew 500 ppm
Dec 15 21:01:51 dist-test-slave-bjsf krb5kdc[29658](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765832511, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.113.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251215 21:01:51.691838 29906 init.cc:377] Logged in from keytab as kudu/127.28.113.2@KRBTEST.COM (short username kudu)
I20251215 21:01:51.692824 29906 webserver.cc:492] Webserver started at http://127.28.113.2:37629/ using document root <none> and password file <none>
I20251215 21:01:51.693326 29906 fs_manager.cc:362] Metadata directory not provided
I20251215 21:01:51.693374 29906 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251215 21:01:51.693569 29906 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251215 21:01:51.695168 29906 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765832506145157-29124-0/minicluster-data/ts-1/data/instance:
uuid: "c67c0ec489404bada63db9ac634264b7"
format_stamp: "Formatted at 2025-12-15 21:01:51 on dist-test-slave-bjsf"
server_key: "ab43c30bee19a656fd2409e717e74133"
server_key_iv: "29be96e0d5f4cd01db6501886f407364"
server_key_version: "encryptionkey@0"
I20251215 21:01:51.695602 29906 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765832506145157-29124-0/minicluster-data/ts-1/wal/instance:
uuid: "c67c0ec489404bada63db9ac634264b7"
format_stamp: "Formatted at 2025-12-15 21:01:51 on dist-test-slave-bjsf"
server_key: "ab43c30bee19a656fd2409e717e74133"
server_key_iv: "29be96e0d5f4cd01db6501886f407364"
server_key_version: "encryptionkey@0"
I20251215 21:01:51.698746 29906 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20251215 21:01:51.700773 29921 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251215 21:01:51.701669 29906 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20251215 21:01:51.701771 29906 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765832506145157-29124-0/minicluster-data/ts-1/data,/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765832506145157-29124-0/minicluster-data/ts-1/wal
uuid: "c67c0ec489404bada63db9ac634264b7"
format_stamp: "Formatted at 2025-12-15 21:01:51 on dist-test-slave-bjsf"
server_key: "ab43c30bee19a656fd2409e717e74133"
server_key_iv: "29be96e0d5f4cd01db6501886f407364"
server_key_version: "encryptionkey@0"
I20251215 21:01:51.701854 29906 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765832506145157-29124-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765832506145157-29124-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765832506145157-29124-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251215 21:01:51.721117 29906 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251215 21:01:51.723937 29906 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251215 21:01:51.724148 29906 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251215 21:01:51.724656 29906 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251215 21:01:51.725489 29906 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251215 21:01:51.725534 29906 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251215 21:01:51.725584 29906 ts_tablet_manager.cc:616] Registered 0 tablets
I20251215 21:01:51.725608 29906 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251215 21:01:51.734041 29906 rpc_server.cc:307] RPC server started. Bound to: 127.28.113.2:36931
I20251215 21:01:51.734066 30034 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.28.113.2:36931 every 8 connection(s)
I20251215 21:01:51.734944 29906 server_base.cc:1179] Dumped server information to /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765832506145157-29124-0/minicluster-data/ts-1/data/info.pb
I20251215 21:01:51.736187 29124 external_mini_cluster.cc:1428] Started /tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu as pid 29906
I20251215 21:01:51.736275 29124 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765832506145157-29124-0/minicluster-data/ts-1/wal/instance
I20251215 21:01:51.736474 29124 external_mini_cluster.cc:1468] Setting key 8169e921c4338c7cd70e23cd3dcd6b19
Dec 15 21:01:51 dist-test-slave-bjsf krb5kdc[29658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765832511, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.113.2@KRBTEST.COM for kudu/127.28.113.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.28.113.3@KRBTEST.COM; defaulting to no policy
I20251215 21:01:51.747370 29745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:01:51.736592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.113.2:56753 (local address 127.28.113.62:46049)
1215 21:01:51.736748 (+ 156us) server_negotiation.cc:207] Beginning negotiation
1215 21:01:51.736752 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1215 21:01:51.737424 (+ 672us) server_negotiation.cc:408] Connection header received
1215 21:01:51.738398 (+ 974us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1215 21:01:51.738401 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1215 21:01:51.738446 (+ 45us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1215 21:01:51.738519 (+ 73us) server_negotiation.cc:227] Negotiated authn=SASL
1215 21:01:51.740375 (+ 1856us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:01:51.740891 (+ 516us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1215 21:01:51.741724 (+ 833us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:01:51.741876 (+ 152us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:01:51.744347 (+ 2471us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1215 21:01:51.744365 (+ 18us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1215 21:01:51.744367 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1215 21:01:51.744393 (+ 26us) server_negotiation.cc:923] calling sasl_server_start()
1215 21:01:51.745764 (+ 1371us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1215 21:01:51.746190 (+ 426us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1215 21:01:51.746193 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1215 21:01:51.746194 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1215 21:01:51.746233 (+ 39us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1215 21:01:51.746495 (+ 262us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1215 21:01:51.746497 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1215 21:01:51.746498 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1215 21:01:51.746641 (+ 143us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1215 21:01:51.746709 (+ 68us) server_negotiation.cc:1036] Waiting for connection context
1215 21:01:51.747125 (+ 416us) server_negotiation.cc:300] Negotiation successful
1215 21:01:51.747249 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.113.3@KRBTEST.COM" created.
I20251215 21:01:51.748121 30037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:01:51.736897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.113.62:46049 (local address 127.28.113.2:56753)
1215 21:01:51.737306 (+ 409us) negotiation.cc:107] Waiting for socket to connect
1215 21:01:51.737339 (+ 33us) client_negotiation.cc:174] Beginning negotiation
1215 21:01:51.738191 (+ 852us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1215 21:01:51.738677 (+ 486us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1215 21:01:51.738686 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
1215 21:01:51.739200 (+ 514us) client_negotiation.cc:189] Negotiated authn=SASL
1215 21:01:51.740233 (+ 1033us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:01:51.740247 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:01:51.741013 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1215 21:01:51.741017 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1215 21:01:51.741617 (+ 600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:01:51.741628 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:01:51.741846 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:01:51.742496 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1215 21:01:51.742530 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
1215 21:01:51.744209 (+ 1679us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1215 21:01:51.745879 (+ 1670us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1215 21:01:51.745883 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1215 21:01:51.745893 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
1215 21:01:51.746108 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1215 21:01:51.746301 (+ 193us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1215 21:01:51.746304 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1215 21:01:51.746306 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1215 21:01:51.746426 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1215 21:01:51.746711 (+ 285us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1215 21:01:51.746716 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1215 21:01:51.746930 (+ 214us) client_negotiation.cc:769] Sending connection context
1215 21:01:51.747114 (+ 184us) client_negotiation.cc:240] Negotiation successful
1215 21:01:51.747334 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":114,"threads_started":1}
I20251215 21:01:51.749189 30035 heartbeater.cc:344] Connected to a master server at 127.28.113.62:46049
I20251215 21:01:51.749418 30035 heartbeater.cc:461] Registering TS with master...
I20251215 21:01:51.749928 30035 heartbeater.cc:507] Master 127.28.113.62:46049 requested a full tablet report, sending...
I20251215 21:01:51.750906 29706 ts_manager.cc:194] Registered new tserver with Master: c67c0ec489404bada63db9ac634264b7 (127.28.113.2:36931)
I20251215 21:01:51.751447 29706 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.113.2@KRBTEST.COM'} at 127.28.113.2:56753
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.113.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765832506145157-29124-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.28.113.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765832506145157-29124-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.28.113.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.28.113.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.28.113.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765832506145157-29124-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.28.113.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765832506145157-29124-0/krb5kdc/kudu.keytab.
I20251215 21:01:51.777878 29124 external_mini_cluster.cc:1366] Running /tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu
/tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765832506145157-29124-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765832506145157-29124-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765832506145157-29124-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765832506145157-29124-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.113.3:0
--local_ip_for_outbound_sockets=127.28.113.3
--webserver_interface=127.28.113.3
--webserver_port=0
--tserver_master_addrs=127.28.113.62:46049
--keytab_file=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765832506145157-29124-0/krb5kdc/kudu.keytab
--principal=kudu/127.28.113.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.28.113.20:42723
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765832506145157-29124-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765832506145157-29124-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765832506145157-29124-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251215 21:01:51.879784 30042 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251215 21:01:51.880020 30042 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251215 21:01:51.880084 30042 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251215 21:01:51.883188 30042 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251215 21:01:51.883239 30042 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251215 21:01:51.883313 30042 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.113.3
I20251215 21:01:51.887300 30042 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.113.20:42723
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765832506145157-29124-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765832506145157-29124-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765832506145157-29124-0/krb5kdc/kudu.keytab
--principal=kudu/127.28.113.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.113.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765832506145157-29124-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.113.3
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.28.113.62:46049
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30042
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.113.3
--log_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765832506145157-29124-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision dc741732bb6a40213e11a13869fad06a49a209a3
build type DEBUG
built by None at 15 Dec 2025 20:43:14 UTC on 24a791456cd2
build id 9530
I20251215 21:01:51.888299 30042 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251215 21:01:51.889058 30042 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251215 21:01:51.894984 30048 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251215 21:01:51.894999 30050 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251215 21:01:51.894979 30047 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251215 21:01:51.895232 30042 server_base.cc:1047] running on GCE node
I20251215 21:01:51.895535 30042 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251215 21:01:51.896059 30042 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251215 21:01:51.897198 30042 hybrid_clock.cc:648] HybridClock initialized: now 1765832511897189 us; error 29 us; skew 500 ppm
Dec 15 21:01:51 dist-test-slave-bjsf krb5kdc[29658](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765832511, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.113.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251215 21:01:51.899649 30042 init.cc:377] Logged in from keytab as kudu/127.28.113.3@KRBTEST.COM (short username kudu)
I20251215 21:01:51.900615 30042 webserver.cc:492] Webserver started at http://127.28.113.3:37705/ using document root <none> and password file <none>
I20251215 21:01:51.901098 30042 fs_manager.cc:362] Metadata directory not provided
I20251215 21:01:51.901140 30042 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251215 21:01:51.901288 30042 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251215 21:01:51.902882 30042 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765832506145157-29124-0/minicluster-data/ts-2/data/instance:
uuid: "2678bae19c5c4037bb880f8c01d8954f"
format_stamp: "Formatted at 2025-12-15 21:01:51 on dist-test-slave-bjsf"
server_key: "0ca1d3fde625fee1ab44f67dcebc784e"
server_key_iv: "e2f234251c9d4e0525191e5bcffde354"
server_key_version: "encryptionkey@0"
I20251215 21:01:51.903287 30042 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765832506145157-29124-0/minicluster-data/ts-2/wal/instance:
uuid: "2678bae19c5c4037bb880f8c01d8954f"
format_stamp: "Formatted at 2025-12-15 21:01:51 on dist-test-slave-bjsf"
server_key: "0ca1d3fde625fee1ab44f67dcebc784e"
server_key_iv: "e2f234251c9d4e0525191e5bcffde354"
server_key_version: "encryptionkey@0"
I20251215 21:01:51.906519 30042 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20251215 21:01:51.908620 30057 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251215 21:01:51.909581 30042 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20251215 21:01:51.909674 30042 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765832506145157-29124-0/minicluster-data/ts-2/data,/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765832506145157-29124-0/minicluster-data/ts-2/wal
uuid: "2678bae19c5c4037bb880f8c01d8954f"
format_stamp: "Formatted at 2025-12-15 21:01:51 on dist-test-slave-bjsf"
server_key: "0ca1d3fde625fee1ab44f67dcebc784e"
server_key_iv: "e2f234251c9d4e0525191e5bcffde354"
server_key_version: "encryptionkey@0"
I20251215 21:01:51.909760 30042 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765832506145157-29124-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765832506145157-29124-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765832506145157-29124-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251215 21:01:51.926064 30042 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251215 21:01:51.929193 30042 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251215 21:01:51.929356 30042 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251215 21:01:51.929884 30042 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251215 21:01:51.930758 30042 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251215 21:01:51.930809 30042 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251215 21:01:51.930862 30042 ts_tablet_manager.cc:616] Registered 0 tablets
I20251215 21:01:51.930886 30042 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251215 21:01:51.940088 30042 rpc_server.cc:307] RPC server started. Bound to: 127.28.113.3:46035
I20251215 21:01:51.940114 30170 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.28.113.3:46035 every 8 connection(s)
I20251215 21:01:51.941004 30042 server_base.cc:1179] Dumped server information to /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765832506145157-29124-0/minicluster-data/ts-2/data/info.pb
I20251215 21:01:51.942179 29124 external_mini_cluster.cc:1428] Started /tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu as pid 30042
I20251215 21:01:51.942286 29124 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765832506145157-29124-0/minicluster-data/ts-2/wal/instance
I20251215 21:01:51.942584 29124 external_mini_cluster.cc:1468] Setting key 268bf9d7cc0fd4cb816edc57e4965264
Dec 15 21:01:51 dist-test-slave-bjsf krb5kdc[29658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765832511, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.113.3@KRBTEST.COM for kudu/127.28.113.62@KRBTEST.COM
I20251215 21:01:51.951861 29745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:01:51.942682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.113.3:53423 (local address 127.28.113.62:46049)
1215 21:01:51.942806 (+ 124us) server_negotiation.cc:207] Beginning negotiation
1215 21:01:51.942810 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1215 21:01:51.943675 (+ 865us) server_negotiation.cc:408] Connection header received
1215 21:01:51.944513 (+ 838us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1215 21:01:51.944516 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1215 21:01:51.944563 (+ 47us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1215 21:01:51.944631 (+ 68us) server_negotiation.cc:227] Negotiated authn=SASL
1215 21:01:51.946158 (+ 1527us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:01:51.946644 (+ 486us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1215 21:01:51.947203 (+ 559us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:01:51.947321 (+ 118us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:01:51.949310 (+ 1989us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1215 21:01:51.949325 (+ 15us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1215 21:01:51.949327 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1215 21:01:51.949350 (+ 23us) server_negotiation.cc:923] calling sasl_server_start()
1215 21:01:51.950453 (+ 1103us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1215 21:01:51.950833 (+ 380us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1215 21:01:51.950835 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1215 21:01:51.950836 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1215 21:01:51.950870 (+ 34us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1215 21:01:51.951070 (+ 200us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1215 21:01:51.951072 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1215 21:01:51.951073 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1215 21:01:51.951202 (+ 129us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1215 21:01:51.951263 (+ 61us) server_negotiation.cc:1036] Waiting for connection context
1215 21:01:51.951659 (+ 396us) server_negotiation.cc:300] Negotiation successful
1215 21:01:51.951748 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20251215 21:01:51.952472 30173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:01:51.943080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.113.62:46049 (local address 127.28.113.3:53423)
1215 21:01:51.943537 (+ 457us) negotiation.cc:107] Waiting for socket to connect
1215 21:01:51.943577 (+ 40us) client_negotiation.cc:174] Beginning negotiation
1215 21:01:51.944350 (+ 773us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1215 21:01:51.944769 (+ 419us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1215 21:01:51.944776 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
1215 21:01:51.945152 (+ 376us) client_negotiation.cc:189] Negotiated authn=SASL
1215 21:01:51.946045 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:01:51.946060 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:01:51.946728 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1215 21:01:51.946730 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1215 21:01:51.947123 (+ 393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:01:51.947129 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:01:51.947281 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:01:51.947821 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1215 21:01:51.947843 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
1215 21:01:51.949185 (+ 1342us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1215 21:01:51.950546 (+ 1361us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1215 21:01:51.950551 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1215 21:01:51.950558 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
1215 21:01:51.950764 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1215 21:01:51.950926 (+ 162us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1215 21:01:51.950928 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1215 21:01:51.950930 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1215 21:01:51.951014 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1215 21:01:51.951263 (+ 249us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1215 21:01:51.951268 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1215 21:01:51.951460 (+ 192us) client_negotiation.cc:769] Sending connection context
1215 21:01:51.951634 (+ 174us) client_negotiation.cc:240] Negotiation successful
1215 21:01:51.951831 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":108,"threads_started":1}
I20251215 21:01:51.953392 30171 heartbeater.cc:344] Connected to a master server at 127.28.113.62:46049
I20251215 21:01:51.953609 30171 heartbeater.cc:461] Registering TS with master...
I20251215 21:01:51.954064 30171 heartbeater.cc:507] Master 127.28.113.62:46049 requested a full tablet report, sending...
I20251215 21:01:51.955094 29706 ts_manager.cc:194] Registered new tserver with Master: 2678bae19c5c4037bb880f8c01d8954f (127.28.113.3:46035)
I20251215 21:01:51.955695 29706 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.113.3@KRBTEST.COM'} at 127.28.113.3:53423
I20251215 21:01:51.955912 29124 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20251215 21:01:51.964619 29745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:01:51.958481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57524 (local address 127.28.113.62:46049)
1215 21:01:51.958607 (+ 126us) server_negotiation.cc:207] Beginning negotiation
1215 21:01:51.958611 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1215 21:01:51.958742 (+ 131us) server_negotiation.cc:408] Connection header received
1215 21:01:51.958861 (+ 119us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1215 21:01:51.958864 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1215 21:01:51.958901 (+ 37us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1215 21:01:51.958957 (+ 56us) server_negotiation.cc:227] Negotiated authn=SASL
1215 21:01:51.959697 (+ 740us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:01:51.960198 (+ 501us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1215 21:01:51.960831 (+ 633us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:01:51.961005 (+ 174us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:01:51.962069 (+ 1064us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1215 21:01:51.962088 (+ 19us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1215 21:01:51.962090 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1215 21:01:51.962115 (+ 25us) server_negotiation.cc:923] calling sasl_server_start()
1215 21:01:51.963221 (+ 1106us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1215 21:01:51.963584 (+ 363us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1215 21:01:51.963587 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1215 21:01:51.963588 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1215 21:01:51.963630 (+ 42us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1215 21:01:51.963822 (+ 192us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1215 21:01:51.963826 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1215 21:01:51.963828 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1215 21:01:51.964052 (+ 224us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1215 21:01:51.964175 (+ 123us) server_negotiation.cc:1036] Waiting for connection context
1215 21:01:51.964365 (+ 190us) server_negotiation.cc:300] Negotiation successful
1215 21:01:51.964470 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20251215 21:01:51.968456 29706 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57524:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251215 21:01:51.970630 29706 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251215 21:01:51.982649 30184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:01:51.978276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.113.3:46035 (local address 127.0.0.1:57236)
1215 21:01:51.978661 (+ 385us) negotiation.cc:107] Waiting for socket to connect
1215 21:01:51.978675 (+ 14us) client_negotiation.cc:174] Beginning negotiation
1215 21:01:51.978791 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1215 21:01:51.979666 (+ 875us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1215 21:01:51.979671 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
1215 21:01:51.979693 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1215 21:01:51.980038 (+ 345us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:01:51.980046 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:01:51.981469 (+ 1423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1215 21:01:51.981472 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1215 21:01:51.982239 (+ 767us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:01:51.982248 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:01:51.982365 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:01:51.982385 (+ 20us) client_negotiation.cc:769] Sending connection context
1215 21:01:51.982433 (+ 48us) client_negotiation.cc:240] Negotiation successful
1215 21:01:51.982502 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":310,"thread_start_us":109,"threads_started":1}
I20251215 21:01:51.982652 30182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:01:51.978056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.113.1:42247 (local address 127.0.0.1:50744)
1215 21:01:51.978625 (+ 569us) negotiation.cc:107] Waiting for socket to connect
1215 21:01:51.978662 (+ 37us) client_negotiation.cc:174] Beginning negotiation
1215 21:01:51.978789 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1215 21:01:51.979840 (+ 1051us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1215 21:01:51.979843 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1215 21:01:51.979859 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1215 21:01:51.980081 (+ 222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:01:51.980087 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:01:51.981424 (+ 1337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1215 21:01:51.981427 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1215 21:01:51.982220 (+ 793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:01:51.982227 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:01:51.982302 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:01:51.982313 (+ 11us) client_negotiation.cc:769] Sending connection context
1215 21:01:51.982402 (+ 89us) client_negotiation.cc:240] Negotiation successful
1215 21:01:51.982489 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":401,"thread_start_us":59,"threads_started":1}
I20251215 21:01:51.982811 30185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:01:51.978692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.113.2:36931 (local address 127.0.0.1:59580)
1215 21:01:51.979120 (+ 428us) negotiation.cc:107] Waiting for socket to connect
1215 21:01:51.979133 (+ 13us) client_negotiation.cc:174] Beginning negotiation
1215 21:01:51.979213 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1215 21:01:51.980097 (+ 884us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1215 21:01:51.980100 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1215 21:01:51.980118 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1215 21:01:51.980312 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:01:51.980318 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:01:51.981189 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1215 21:01:51.981192 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1215 21:01:51.982139 (+ 947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:01:51.982149 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:01:51.982248 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:01:51.982302 (+ 54us) client_negotiation.cc:769] Sending connection context
1215 21:01:51.982402 (+ 100us) client_negotiation.cc:240] Negotiation successful
1215 21:01:51.982489 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":80,"threads_started":1}
I20251215 21:01:51.983421 30186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:01:51.978662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57236 (local address 127.28.113.3:46035)
1215 21:01:51.979172 (+ 510us) server_negotiation.cc:207] Beginning negotiation
1215 21:01:51.979183 (+ 11us) server_negotiation.cc:400] Waiting for connection header
1215 21:01:51.979206 (+ 23us) server_negotiation.cc:408] Connection header received
1215 21:01:51.979287 (+ 81us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1215 21:01:51.979292 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1215 21:01:51.979478 (+ 186us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1215 21:01:51.979667 (+ 189us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1215 21:01:51.980172 (+ 505us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:01:51.981331 (+ 1159us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1215 21:01:51.982373 (+ 1042us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:01:51.982883 (+ 510us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:01:51.983157 (+ 274us) server_negotiation.cc:1036] Waiting for connection context
1215 21:01:51.983227 (+ 70us) server_negotiation.cc:300] Negotiation successful
1215 21:01:51.983317 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":410,"thread_start_us":202,"threads_started":1}
I20251215 21:01:51.983456 30183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:01:51.978181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50744 (local address 127.28.113.1:42247)
1215 21:01:51.979449 (+ 1268us) server_negotiation.cc:207] Beginning negotiation
1215 21:01:51.979455 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1215 21:01:51.979475 (+ 20us) server_negotiation.cc:408] Connection header received
1215 21:01:51.979550 (+ 75us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1215 21:01:51.979555 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1215 21:01:51.979719 (+ 164us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1215 21:01:51.979856 (+ 137us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1215 21:01:51.980195 (+ 339us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:01:51.981321 (+ 1126us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1215 21:01:51.982572 (+ 1251us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:01:51.983083 (+ 511us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:01:51.983181 (+ 98us) server_negotiation.cc:1036] Waiting for connection context
1215 21:01:51.983242 (+ 61us) server_negotiation.cc:300] Negotiation successful
1215 21:01:51.983325 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1166,"thread_start_us":125,"threads_started":1}
I20251215 21:01:51.984261 30187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:01:51.979104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59580 (local address 127.28.113.2:36931)
1215 21:01:51.979695 (+ 591us) server_negotiation.cc:207] Beginning negotiation
1215 21:01:51.979701 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1215 21:01:51.979719 (+ 18us) server_negotiation.cc:408] Connection header received
1215 21:01:51.979792 (+ 73us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1215 21:01:51.979797 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1215 21:01:51.979945 (+ 148us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1215 21:01:51.980092 (+ 147us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1215 21:01:51.980420 (+ 328us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:01:51.981089 (+ 669us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1215 21:01:51.982870 (+ 1781us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:01:51.983869 (+ 999us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:01:51.983998 (+ 129us) server_negotiation.cc:1036] Waiting for connection context
1215 21:01:51.984055 (+ 57us) server_negotiation.cc:300] Negotiation successful
1215 21:01:51.984134 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":500,"thread_start_us":143,"threads_started":1}
I20251215 21:01:51.986045 30105 tablet_service.cc:1505] Processing CreateTablet for tablet c2da738c54d44670891311ca43c38eba (DEFAULT_TABLE table=test-table [id=896ed83cfb6344fdb98e2006479a2e4f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251215 21:01:51.986070 29833 tablet_service.cc:1505] Processing CreateTablet for tablet c2da738c54d44670891311ca43c38eba (DEFAULT_TABLE table=test-table [id=896ed83cfb6344fdb98e2006479a2e4f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251215 21:01:51.986407 29969 tablet_service.cc:1505] Processing CreateTablet for tablet c2da738c54d44670891311ca43c38eba (DEFAULT_TABLE table=test-table [id=896ed83cfb6344fdb98e2006479a2e4f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251215 21:01:51.986853 30105 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c2da738c54d44670891311ca43c38eba. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 21:01:51.986922 29833 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c2da738c54d44670891311ca43c38eba. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 21:01:51.987131 29969 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c2da738c54d44670891311ca43c38eba. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 21:01:51.992025 30188 tablet_bootstrap.cc:492] T c2da738c54d44670891311ca43c38eba P 2678bae19c5c4037bb880f8c01d8954f: Bootstrap starting.
I20251215 21:01:51.993153 30189 tablet_bootstrap.cc:492] T c2da738c54d44670891311ca43c38eba P c67c0ec489404bada63db9ac634264b7: Bootstrap starting.
I20251215 21:01:51.993153 30190 tablet_bootstrap.cc:492] T c2da738c54d44670891311ca43c38eba P cf1e48f4eb75456c803332841127d851: Bootstrap starting.
I20251215 21:01:51.994588 30188 tablet_bootstrap.cc:654] T c2da738c54d44670891311ca43c38eba P 2678bae19c5c4037bb880f8c01d8954f: Neither blocks nor log segments found. Creating new log.
I20251215 21:01:51.995098 30190 tablet_bootstrap.cc:654] T c2da738c54d44670891311ca43c38eba P cf1e48f4eb75456c803332841127d851: Neither blocks nor log segments found. Creating new log.
I20251215 21:01:51.995098 30189 tablet_bootstrap.cc:654] T c2da738c54d44670891311ca43c38eba P c67c0ec489404bada63db9ac634264b7: Neither blocks nor log segments found. Creating new log.
I20251215 21:01:51.995275 30188 log.cc:826] T c2da738c54d44670891311ca43c38eba P 2678bae19c5c4037bb880f8c01d8954f: Log is configured to *not* fsync() on all Append() calls
I20251215 21:01:51.995873 30189 log.cc:826] T c2da738c54d44670891311ca43c38eba P c67c0ec489404bada63db9ac634264b7: Log is configured to *not* fsync() on all Append() calls
I20251215 21:01:51.995873 30190 log.cc:826] T c2da738c54d44670891311ca43c38eba P cf1e48f4eb75456c803332841127d851: Log is configured to *not* fsync() on all Append() calls
I20251215 21:01:51.996950 30188 tablet_bootstrap.cc:492] T c2da738c54d44670891311ca43c38eba P 2678bae19c5c4037bb880f8c01d8954f: No bootstrap required, opened a new log
I20251215 21:01:51.997119 30188 ts_tablet_manager.cc:1403] T c2da738c54d44670891311ca43c38eba P 2678bae19c5c4037bb880f8c01d8954f: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20251215 21:01:51.997467 30189 tablet_bootstrap.cc:492] T c2da738c54d44670891311ca43c38eba P c67c0ec489404bada63db9ac634264b7: No bootstrap required, opened a new log
I20251215 21:01:51.997467 30190 tablet_bootstrap.cc:492] T c2da738c54d44670891311ca43c38eba P cf1e48f4eb75456c803332841127d851: No bootstrap required, opened a new log
I20251215 21:01:51.997639 30190 ts_tablet_manager.cc:1403] T c2da738c54d44670891311ca43c38eba P cf1e48f4eb75456c803332841127d851: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20251215 21:01:51.997639 30189 ts_tablet_manager.cc:1403] T c2da738c54d44670891311ca43c38eba P c67c0ec489404bada63db9ac634264b7: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20251215 21:01:52.000260 30188 raft_consensus.cc:359] T c2da738c54d44670891311ca43c38eba P 2678bae19c5c4037bb880f8c01d8954f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf1e48f4eb75456c803332841127d851" member_type: VOTER last_known_addr { host: "127.28.113.1" port: 42247 } } peers { permanent_uuid: "2678bae19c5c4037bb880f8c01d8954f" member_type: VOTER last_known_addr { host: "127.28.113.3" port: 46035 } } peers { permanent_uuid: "c67c0ec489404bada63db9ac634264b7" member_type: VOTER last_known_addr { host: "127.28.113.2" port: 36931 } }
I20251215 21:01:52.000437 30188 raft_consensus.cc:385] T c2da738c54d44670891311ca43c38eba P 2678bae19c5c4037bb880f8c01d8954f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251215 21:01:52.000512 30188 raft_consensus.cc:740] T c2da738c54d44670891311ca43c38eba P 2678bae19c5c4037bb880f8c01d8954f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2678bae19c5c4037bb880f8c01d8954f, State: Initialized, Role: FOLLOWER
I20251215 21:01:52.000916 30188 consensus_queue.cc:260] T c2da738c54d44670891311ca43c38eba P 2678bae19c5c4037bb880f8c01d8954f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf1e48f4eb75456c803332841127d851" member_type: VOTER last_known_addr { host: "127.28.113.1" port: 42247 } } peers { permanent_uuid: "2678bae19c5c4037bb880f8c01d8954f" member_type: VOTER last_known_addr { host: "127.28.113.3" port: 46035 } } peers { permanent_uuid: "c67c0ec489404bada63db9ac634264b7" member_type: VOTER last_known_addr { host: "127.28.113.2" port: 36931 } }
I20251215 21:01:52.000994 30190 raft_consensus.cc:359] T c2da738c54d44670891311ca43c38eba P cf1e48f4eb75456c803332841127d851 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf1e48f4eb75456c803332841127d851" member_type: VOTER last_known_addr { host: "127.28.113.1" port: 42247 } } peers { permanent_uuid: "2678bae19c5c4037bb880f8c01d8954f" member_type: VOTER last_known_addr { host: "127.28.113.3" port: 46035 } } peers { permanent_uuid: "c67c0ec489404bada63db9ac634264b7" member_type: VOTER last_known_addr { host: "127.28.113.2" port: 36931 } }
I20251215 21:01:52.000996 30189 raft_consensus.cc:359] T c2da738c54d44670891311ca43c38eba P c67c0ec489404bada63db9ac634264b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf1e48f4eb75456c803332841127d851" member_type: VOTER last_known_addr { host: "127.28.113.1" port: 42247 } } peers { permanent_uuid: "2678bae19c5c4037bb880f8c01d8954f" member_type: VOTER last_known_addr { host: "127.28.113.3" port: 46035 } } peers { permanent_uuid: "c67c0ec489404bada63db9ac634264b7" member_type: VOTER last_known_addr { host: "127.28.113.2" port: 36931 } }
I20251215 21:01:52.001178 30189 raft_consensus.cc:385] T c2da738c54d44670891311ca43c38eba P c67c0ec489404bada63db9ac634264b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251215 21:01:52.001260 30190 raft_consensus.cc:385] T c2da738c54d44670891311ca43c38eba P cf1e48f4eb75456c803332841127d851 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251215 21:01:52.001259 30189 raft_consensus.cc:740] T c2da738c54d44670891311ca43c38eba P c67c0ec489404bada63db9ac634264b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c67c0ec489404bada63db9ac634264b7, State: Initialized, Role: FOLLOWER
I20251215 21:01:52.001349 30190 raft_consensus.cc:740] T c2da738c54d44670891311ca43c38eba P cf1e48f4eb75456c803332841127d851 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf1e48f4eb75456c803332841127d851, State: Initialized, Role: FOLLOWER
I20251215 21:01:52.001528 30171 heartbeater.cc:499] Master 127.28.113.62:46049 was elected leader, sending a full tablet report...
I20251215 21:01:52.001714 30188 ts_tablet_manager.cc:1434] T c2da738c54d44670891311ca43c38eba P 2678bae19c5c4037bb880f8c01d8954f: Time spent starting tablet: real 0.005s user 0.002s sys 0.003s
I20251215 21:01:52.001796 30190 consensus_queue.cc:260] T c2da738c54d44670891311ca43c38eba P cf1e48f4eb75456c803332841127d851 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf1e48f4eb75456c803332841127d851" member_type: VOTER last_known_addr { host: "127.28.113.1" port: 42247 } } peers { permanent_uuid: "2678bae19c5c4037bb880f8c01d8954f" member_type: VOTER last_known_addr { host: "127.28.113.3" port: 46035 } } peers { permanent_uuid: "c67c0ec489404bada63db9ac634264b7" member_type: VOTER last_known_addr { host: "127.28.113.2" port: 36931 } }
I20251215 21:01:52.002063 30189 consensus_queue.cc:260] T c2da738c54d44670891311ca43c38eba P c67c0ec489404bada63db9ac634264b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf1e48f4eb75456c803332841127d851" member_type: VOTER last_known_addr { host: "127.28.113.1" port: 42247 } } peers { permanent_uuid: "2678bae19c5c4037bb880f8c01d8954f" member_type: VOTER last_known_addr { host: "127.28.113.3" port: 46035 } } peers { permanent_uuid: "c67c0ec489404bada63db9ac634264b7" member_type: VOTER last_known_addr { host: "127.28.113.2" port: 36931 } }
I20251215 21:01:52.002470 29899 heartbeater.cc:499] Master 127.28.113.62:46049 was elected leader, sending a full tablet report...
I20251215 21:01:52.002880 30189 ts_tablet_manager.cc:1434] T c2da738c54d44670891311ca43c38eba P c67c0ec489404bada63db9ac634264b7: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20251215 21:01:52.003185 30035 heartbeater.cc:499] Master 127.28.113.62:46049 was elected leader, sending a full tablet report...
I20251215 21:01:52.003520 30190 ts_tablet_manager.cc:1434] T c2da738c54d44670891311ca43c38eba P cf1e48f4eb75456c803332841127d851: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
W20251215 21:01:52.024173 29900 tablet.cc:2378] T c2da738c54d44670891311ca43c38eba P cf1e48f4eb75456c803332841127d851: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20251215 21:01:52.191881 30172 tablet.cc:2378] T c2da738c54d44670891311ca43c38eba P 2678bae19c5c4037bb880f8c01d8954f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20251215 21:01:52.235920 30036 tablet.cc:2378] T c2da738c54d44670891311ca43c38eba P c67c0ec489404bada63db9ac634264b7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20251215 21:01:52.262317 30195 raft_consensus.cc:493] T c2da738c54d44670891311ca43c38eba P cf1e48f4eb75456c803332841127d851 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 21:01:52.262514 30195 raft_consensus.cc:515] T c2da738c54d44670891311ca43c38eba P cf1e48f4eb75456c803332841127d851 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf1e48f4eb75456c803332841127d851" member_type: VOTER last_known_addr { host: "127.28.113.1" port: 42247 } } peers { permanent_uuid: "2678bae19c5c4037bb880f8c01d8954f" member_type: VOTER last_known_addr { host: "127.28.113.3" port: 46035 } } peers { permanent_uuid: "c67c0ec489404bada63db9ac634264b7" member_type: VOTER last_known_addr { host: "127.28.113.2" port: 36931 } }
I20251215 21:01:52.263602 30195 leader_election.cc:290] T c2da738c54d44670891311ca43c38eba P cf1e48f4eb75456c803332841127d851 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2678bae19c5c4037bb880f8c01d8954f (127.28.113.3:46035), c67c0ec489404bada63db9ac634264b7 (127.28.113.2:36931)
I20251215 21:01:52.266743 30197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:01:52.263858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.113.2:36931 (local address 127.28.113.1:53287)
1215 21:01:52.264300 (+ 442us) negotiation.cc:107] Waiting for socket to connect
1215 21:01:52.264313 (+ 13us) client_negotiation.cc:174] Beginning negotiation
1215 21:01:52.264415 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1215 21:01:52.264759 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1215 21:01:52.264763 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
1215 21:01:52.264780 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1215 21:01:52.265085 (+ 305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:01:52.265094 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:01:52.265786 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1215 21:01:52.265790 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1215 21:01:52.266355 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:01:52.266362 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:01:52.266462 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:01:52.266484 (+ 22us) client_negotiation.cc:769] Sending connection context
1215 21:01:52.266532 (+ 48us) client_negotiation.cc:240] Negotiation successful
1215 21:01:52.266588 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":374,"mutex_wait_us":10,"thread_start_us":121,"threads_started":1}
I20251215 21:01:52.266881 30198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:01:52.263858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.113.3:46035 (local address 127.28.113.1:45873)
1215 21:01:52.264212 (+ 354us) negotiation.cc:107] Waiting for socket to connect
1215 21:01:52.264227 (+ 15us) client_negotiation.cc:174] Beginning negotiation
1215 21:01:52.264343 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1215 21:01:52.264659 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1215 21:01:52.264662 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1215 21:01:52.264681 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1215 21:01:52.265088 (+ 407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:01:52.265103 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:01:52.265817 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1215 21:01:52.265821 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1215 21:01:52.266422 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:01:52.266430 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:01:52.266540 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:01:52.266559 (+ 19us) client_negotiation.cc:769] Sending connection context
1215 21:01:52.266607 (+ 48us) client_negotiation.cc:240] Negotiation successful
1215 21:01:52.266659 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":282,"thread_start_us":98,"threads_started":1}
I20251215 21:01:52.267166 30187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:01:52.263951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.113.1:53287 (local address 127.28.113.2:36931)
1215 21:01:52.264117 (+ 166us) server_negotiation.cc:207] Beginning negotiation
1215 21:01:52.264122 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1215 21:01:52.264375 (+ 253us) server_negotiation.cc:408] Connection header received
1215 21:01:52.264570 (+ 195us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1215 21:01:52.264574 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1215 21:01:52.264629 (+ 55us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1215 21:01:52.264701 (+ 72us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1215 21:01:52.265201 (+ 500us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:01:52.265683 (+ 482us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1215 21:01:52.266466 (+ 783us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:01:52.266920 (+ 454us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:01:52.266959 (+ 39us) server_negotiation.cc:1036] Waiting for connection context
1215 21:01:52.267009 (+ 50us) server_negotiation.cc:300] Negotiation successful
1215 21:01:52.267066 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20251215 21:01:52.267280 30186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:01:52.263950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.113.1:45873 (local address 127.28.113.3:46035)
1215 21:01:52.264192 (+ 242us) server_negotiation.cc:207] Beginning negotiation
1215 21:01:52.264196 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1215 21:01:52.264301 (+ 105us) server_negotiation.cc:408] Connection header received
1215 21:01:52.264498 (+ 197us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1215 21:01:52.264502 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1215 21:01:52.264558 (+ 56us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1215 21:01:52.264633 (+ 75us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1215 21:01:52.265213 (+ 580us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:01:52.265700 (+ 487us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1215 21:01:52.266537 (+ 837us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:01:52.267021 (+ 484us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:01:52.267059 (+ 38us) server_negotiation.cc:1036] Waiting for connection context
1215 21:01:52.267109 (+ 50us) server_negotiation.cc:300] Negotiation successful
1215 21:01:52.267166 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113}
I20251215 21:01:52.267745 29989 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c2da738c54d44670891311ca43c38eba" candidate_uuid: "cf1e48f4eb75456c803332841127d851" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c67c0ec489404bada63db9ac634264b7" is_pre_election: true
I20251215 21:01:52.267789 30125 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c2da738c54d44670891311ca43c38eba" candidate_uuid: "cf1e48f4eb75456c803332841127d851" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2678bae19c5c4037bb880f8c01d8954f" is_pre_election: true
I20251215 21:01:52.268035 29989 raft_consensus.cc:2468] T c2da738c54d44670891311ca43c38eba P c67c0ec489404bada63db9ac634264b7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cf1e48f4eb75456c803332841127d851 in term 0.
I20251215 21:01:52.268046 30125 raft_consensus.cc:2468] T c2da738c54d44670891311ca43c38eba P 2678bae19c5c4037bb880f8c01d8954f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cf1e48f4eb75456c803332841127d851 in term 0.
I20251215 21:01:52.268453 29786 leader_election.cc:304] T c2da738c54d44670891311ca43c38eba P cf1e48f4eb75456c803332841127d851 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2678bae19c5c4037bb880f8c01d8954f, cf1e48f4eb75456c803332841127d851; no voters:
I20251215 21:01:52.268677 30195 raft_consensus.cc:2804] T c2da738c54d44670891311ca43c38eba P cf1e48f4eb75456c803332841127d851 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251215 21:01:52.268743 30195 raft_consensus.cc:493] T c2da738c54d44670891311ca43c38eba P cf1e48f4eb75456c803332841127d851 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 21:01:52.268774 30195 raft_consensus.cc:3060] T c2da738c54d44670891311ca43c38eba P cf1e48f4eb75456c803332841127d851 [term 0 FOLLOWER]: Advancing to term 1
I20251215 21:01:52.269646 30195 raft_consensus.cc:515] T c2da738c54d44670891311ca43c38eba P cf1e48f4eb75456c803332841127d851 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf1e48f4eb75456c803332841127d851" member_type: VOTER last_known_addr { host: "127.28.113.1" port: 42247 } } peers { permanent_uuid: "2678bae19c5c4037bb880f8c01d8954f" member_type: VOTER last_known_addr { host: "127.28.113.3" port: 46035 } } peers { permanent_uuid: "c67c0ec489404bada63db9ac634264b7" member_type: VOTER last_known_addr { host: "127.28.113.2" port: 36931 } }
I20251215 21:01:52.269979 30195 leader_election.cc:290] T c2da738c54d44670891311ca43c38eba P cf1e48f4eb75456c803332841127d851 [CANDIDATE]: Term 1 election: Requested vote from peers 2678bae19c5c4037bb880f8c01d8954f (127.28.113.3:46035), c67c0ec489404bada63db9ac634264b7 (127.28.113.2:36931)
I20251215 21:01:52.270319 30125 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c2da738c54d44670891311ca43c38eba" candidate_uuid: "cf1e48f4eb75456c803332841127d851" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2678bae19c5c4037bb880f8c01d8954f"
I20251215 21:01:52.270329 29989 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c2da738c54d44670891311ca43c38eba" candidate_uuid: "cf1e48f4eb75456c803332841127d851" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c67c0ec489404bada63db9ac634264b7"
I20251215 21:01:52.270421 30125 raft_consensus.cc:3060] T c2da738c54d44670891311ca43c38eba P 2678bae19c5c4037bb880f8c01d8954f [term 0 FOLLOWER]: Advancing to term 1
I20251215 21:01:52.270428 29989 raft_consensus.cc:3060] T c2da738c54d44670891311ca43c38eba P c67c0ec489404bada63db9ac634264b7 [term 0 FOLLOWER]: Advancing to term 1
I20251215 21:01:52.271289 29989 raft_consensus.cc:2468] T c2da738c54d44670891311ca43c38eba P c67c0ec489404bada63db9ac634264b7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cf1e48f4eb75456c803332841127d851 in term 1.
I20251215 21:01:52.271333 30125 raft_consensus.cc:2468] T c2da738c54d44670891311ca43c38eba P 2678bae19c5c4037bb880f8c01d8954f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cf1e48f4eb75456c803332841127d851 in term 1.
I20251215 21:01:52.271572 29788 leader_election.cc:304] T c2da738c54d44670891311ca43c38eba P cf1e48f4eb75456c803332841127d851 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c67c0ec489404bada63db9ac634264b7, cf1e48f4eb75456c803332841127d851; no voters:
I20251215 21:01:52.271726 30195 raft_consensus.cc:2804] T c2da738c54d44670891311ca43c38eba P cf1e48f4eb75456c803332841127d851 [term 1 FOLLOWER]: Leader election won for term 1
I20251215 21:01:52.271939 30195 raft_consensus.cc:697] T c2da738c54d44670891311ca43c38eba P cf1e48f4eb75456c803332841127d851 [term 1 LEADER]: Becoming Leader. State: Replica: cf1e48f4eb75456c803332841127d851, State: Running, Role: LEADER
I20251215 21:01:52.272258 30195 consensus_queue.cc:237] T c2da738c54d44670891311ca43c38eba P cf1e48f4eb75456c803332841127d851 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf1e48f4eb75456c803332841127d851" member_type: VOTER last_known_addr { host: "127.28.113.1" port: 42247 } } peers { permanent_uuid: "2678bae19c5c4037bb880f8c01d8954f" member_type: VOTER last_known_addr { host: "127.28.113.3" port: 46035 } } peers { permanent_uuid: "c67c0ec489404bada63db9ac634264b7" member_type: VOTER last_known_addr { host: "127.28.113.2" port: 36931 } }
I20251215 21:01:52.275131 29706 catalog_manager.cc:5654] T c2da738c54d44670891311ca43c38eba P cf1e48f4eb75456c803332841127d851 reported cstate change: term changed from 0 to 1, leader changed from <none> to cf1e48f4eb75456c803332841127d851 (127.28.113.1). New cstate: current_term: 1 leader_uuid: "cf1e48f4eb75456c803332841127d851" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf1e48f4eb75456c803332841127d851" member_type: VOTER last_known_addr { host: "127.28.113.1" port: 42247 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2678bae19c5c4037bb880f8c01d8954f" member_type: VOTER last_known_addr { host: "127.28.113.3" port: 46035 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c67c0ec489404bada63db9ac634264b7" member_type: VOTER last_known_addr { host: "127.28.113.2" port: 36931 } health_report { overall_health: UNKNOWN } } }
Dec 15 21:01:52 dist-test-slave-bjsf krb5kdc[29658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765832511, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.113.1@KRBTEST.COM
I20251215 21:01:52.344250 30183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:01:52.336997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50750 (local address 127.28.113.1:42247)
1215 21:01:52.337149 (+ 152us) server_negotiation.cc:207] Beginning negotiation
1215 21:01:52.337153 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1215 21:01:52.337167 (+ 14us) server_negotiation.cc:408] Connection header received
1215 21:01:52.337279 (+ 112us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1215 21:01:52.337282 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1215 21:01:52.337325 (+ 43us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1215 21:01:52.337391 (+ 66us) server_negotiation.cc:227] Negotiated authn=SASL
1215 21:01:52.338129 (+ 738us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:01:52.338619 (+ 490us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1215 21:01:52.339180 (+ 561us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:01:52.339301 (+ 121us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:01:52.341373 (+ 2072us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1215 21:01:52.341397 (+ 24us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1215 21:01:52.341410 (+ 13us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1215 21:01:52.341433 (+ 23us) server_negotiation.cc:923] calling sasl_server_start()
1215 21:01:52.342968 (+ 1535us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1215 21:01:52.343306 (+ 338us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1215 21:01:52.343311 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1215 21:01:52.343313 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1215 21:01:52.343360 (+ 47us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1215 21:01:52.343525 (+ 165us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1215 21:01:52.343527 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1215 21:01:52.343528 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1215 21:01:52.343691 (+ 163us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1215 21:01:52.343861 (+ 170us) server_negotiation.cc:1036] Waiting for connection context
1215 21:01:52.343999 (+ 138us) server_negotiation.cc:300] Negotiation successful
1215 21:01:52.344093 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
Dec 15 21:01:52 dist-test-slave-bjsf krb5kdc[29658](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765832512, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Dec 15 21:01:52 dist-test-slave-bjsf krb5kdc[29658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765832512, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.28.113.1@KRBTEST.COM
I20251215 21:01:52.366704 30183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:01:52.359427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50756 (local address 127.28.113.1:42247)
1215 21:01:52.359586 (+ 159us) server_negotiation.cc:207] Beginning negotiation
1215 21:01:52.359592 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1215 21:01:52.359657 (+ 65us) server_negotiation.cc:408] Connection header received
1215 21:01:52.359813 (+ 156us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1215 21:01:52.359817 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1215 21:01:52.359865 (+ 48us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1215 21:01:52.359941 (+ 76us) server_negotiation.cc:227] Negotiated authn=SASL
1215 21:01:52.360776 (+ 835us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:01:52.361247 (+ 471us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1215 21:01:52.361897 (+ 650us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:01:52.362072 (+ 175us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:01:52.364059 (+ 1987us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1215 21:01:52.364080 (+ 21us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1215 21:01:52.364083 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1215 21:01:52.364108 (+ 25us) server_negotiation.cc:923] calling sasl_server_start()
1215 21:01:52.365256 (+ 1148us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1215 21:01:52.365650 (+ 394us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1215 21:01:52.365654 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1215 21:01:52.365655 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1215 21:01:52.365699 (+ 44us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1215 21:01:52.365914 (+ 215us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1215 21:01:52.365918 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1215 21:01:52.365920 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1215 21:01:52.366054 (+ 134us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1215 21:01:52.366188 (+ 134us) server_negotiation.cc:1036] Waiting for connection context
1215 21:01:52.366420 (+ 232us) server_negotiation.cc:300] Negotiation successful
1215 21:01:52.366533 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
Dec 15 21:01:52 dist-test-slave-bjsf krb5kdc[29658](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765832512, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Dec 15 21:01:52 dist-test-slave-bjsf krb5kdc[29658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765832512, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.113.1@KRBTEST.COM
I20251215 21:01:52.388851 30183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:01:52.381510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50768 (local address 127.28.113.1:42247)
1215 21:01:52.381647 (+ 137us) server_negotiation.cc:207] Beginning negotiation
1215 21:01:52.381652 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1215 21:01:52.381743 (+ 91us) server_negotiation.cc:408] Connection header received
1215 21:01:52.381879 (+ 136us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1215 21:01:52.381884 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1215 21:01:52.381931 (+ 47us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1215 21:01:52.382006 (+ 75us) server_negotiation.cc:227] Negotiated authn=SASL
1215 21:01:52.382805 (+ 799us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:01:52.383269 (+ 464us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1215 21:01:52.383931 (+ 662us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:01:52.384154 (+ 223us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:01:52.386079 (+ 1925us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1215 21:01:52.386098 (+ 19us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1215 21:01:52.386100 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1215 21:01:52.386127 (+ 27us) server_negotiation.cc:923] calling sasl_server_start()
1215 21:01:52.387296 (+ 1169us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1215 21:01:52.387737 (+ 441us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1215 21:01:52.387741 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1215 21:01:52.387742 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1215 21:01:52.387790 (+ 48us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1215 21:01:52.388077 (+ 287us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1215 21:01:52.388081 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1215 21:01:52.388083 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1215 21:01:52.388223 (+ 140us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1215 21:01:52.388348 (+ 125us) server_negotiation.cc:1036] Waiting for connection context
1215 21:01:52.388566 (+ 218us) server_negotiation.cc:300] Negotiation successful
1215 21:01:52.388673 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
W20251215 21:01:52.390164 29813 tablet_service.cc:3012] Rejecting scan request for tablet c2da738c54d44670891311ca43c38eba: Uninitialized: clean time has not yet been initialized
I20251215 21:01:52.391695 29124 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu with pid 29770
I20251215 21:01:52.397466 29124 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu with pid 29906
I20251215 21:01:52.402056 29124 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu with pid 30042
I20251215 21:01:52.406632 29124 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu with pid 29674
2025-12-15T21:01:52Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (3299 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765832506145157-29124-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Dec 15 21:01:52 dist-test-slave-bjsf krb5kdc[30213](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Dec 15 21:01:52 dist-test-slave-bjsf krb5kdc[30213](info): set up 2 sockets
Dec 15 21:01:52 dist-test-slave-bjsf krb5kdc[30213](info): commencing operation
krb5kdc: starting...
W20251215 21:01:54.232149 29124 mini_kdc.cc:121] Time spent starting KDC: real 1.804s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765832506145157-29124-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765832506145157-29124-0/krb5kdc/test-user.keytab.
Dec 15 21:01:54 dist-test-slave-bjsf krb5kdc[30213](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765832514, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-12-15T21:01:54Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-12-15T21:01:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.113.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.113.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.113.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765832506145157-29124-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.28.113.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765832506145157-29124-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.28.113.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.28.113.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.28.113.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765832506145157-29124-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.28.113.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765832506145157-29124-0/krb5kdc/kudu.keytab.
I20251215 21:01:54.354501 29124 external_mini_cluster.cc:1366] Running /tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu
/tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765832506145157-29124-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765832506145157-29124-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765832506145157-29124-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765832506145157-29124-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.113.62:40865
--webserver_interface=127.28.113.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765832506145157-29124-0/krb5kdc/kudu.keytab
--principal=kudu/127.28.113.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.28.113.20:38907
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.113.62:40865
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765832506145157-29124-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765832506145157-29124-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765832506145157-29124-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251215 21:01:54.457362 30229 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251215 21:01:54.457619 30229 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251215 21:01:54.457695 30229 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251215 21:01:54.461004 30229 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251215 21:01:54.461076 30229 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20251215 21:01:54.461102 30229 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251215 21:01:54.461127 30229 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251215 21:01:54.461161 30229 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251215 21:01:54.461192 30229 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251215 21:01:54.465484 30229 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.113.20:38907
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765832506145157-29124-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765832506145157-29124-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.113.62:40865
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765832506145157-29124-0/krb5kdc/kudu.keytab
--principal=kudu/127.28.113.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.113.62:40865
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765832506145157-29124-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.113.62
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30229
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765832506145157-29124-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision dc741732bb6a40213e11a13869fad06a49a209a3
build type DEBUG
built by None at 15 Dec 2025 20:43:14 UTC on 24a791456cd2
build id 9530
I20251215 21:01:54.466564 30229 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251215 21:01:54.467387 30229 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251215 21:01:54.472622 30237 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251215 21:01:54.472625 30234 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251215 21:01:54.472622 30235 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251215 21:01:54.473150 30229 server_base.cc:1047] running on GCE node
I20251215 21:01:54.473594 30229 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251215 21:01:54.474448 30229 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251215 21:01:54.475593 30229 hybrid_clock.cc:648] HybridClock initialized: now 1765832514475579 us; error 38 us; skew 500 ppm
Dec 15 21:01:54 dist-test-slave-bjsf krb5kdc[30213](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765832514, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.113.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251215 21:01:54.478061 30229 init.cc:377] Logged in from keytab as kudu/127.28.113.62@KRBTEST.COM (short username kudu)
I20251215 21:01:54.479104 30229 webserver.cc:492] Webserver started at http://127.28.113.62:46721/ using document root <none> and password file <none>
I20251215 21:01:54.479616 30229 fs_manager.cc:362] Metadata directory not provided
I20251215 21:01:54.479662 30229 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251215 21:01:54.479826 30229 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251215 21:01:54.481515 30229 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765832506145157-29124-0/minicluster-data/master-0/data/instance:
uuid: "700c44b9a4694ad6993e6d84e5c0475c"
format_stamp: "Formatted at 2025-12-15 21:01:54 on dist-test-slave-bjsf"
server_key: "f5d32700ca2ac6eb5993966e1d2c6bfa"
server_key_iv: "7543a8daf395250ae5a7d92ffb394963"
server_key_version: "encryptionkey@0"
I20251215 21:01:54.481940 30229 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765832506145157-29124-0/minicluster-data/master-0/wal/instance:
uuid: "700c44b9a4694ad6993e6d84e5c0475c"
format_stamp: "Formatted at 2025-12-15 21:01:54 on dist-test-slave-bjsf"
server_key: "f5d32700ca2ac6eb5993966e1d2c6bfa"
server_key_iv: "7543a8daf395250ae5a7d92ffb394963"
server_key_version: "encryptionkey@0"
I20251215 21:01:54.485183 30229 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20251215 21:01:54.487263 30244 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251215 21:01:54.488279 30229 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20251215 21:01:54.488384 30229 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765832506145157-29124-0/minicluster-data/master-0/data,/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765832506145157-29124-0/minicluster-data/master-0/wal
uuid: "700c44b9a4694ad6993e6d84e5c0475c"
format_stamp: "Formatted at 2025-12-15 21:01:54 on dist-test-slave-bjsf"
server_key: "f5d32700ca2ac6eb5993966e1d2c6bfa"
server_key_iv: "7543a8daf395250ae5a7d92ffb394963"
server_key_version: "encryptionkey@0"
I20251215 21:01:54.488477 30229 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765832506145157-29124-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765832506145157-29124-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765832506145157-29124-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251215 21:01:54.505095 30229 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251215 21:01:54.511143 30229 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251215 21:01:54.511305 30229 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251215 21:01:54.518887 30229 rpc_server.cc:307] RPC server started. Bound to: 127.28.113.62:40865
I20251215 21:01:54.519093 30306 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.28.113.62:40865 every 8 connection(s)
I20251215 21:01:54.519919 30229 server_base.cc:1179] Dumped server information to /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765832506145157-29124-0/minicluster-data/master-0/data/info.pb
I20251215 21:01:54.522541 30307 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 21:01:54.527469 30307 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 700c44b9a4694ad6993e6d84e5c0475c: Bootstrap starting.
I20251215 21:01:54.529073 29124 external_mini_cluster.cc:1428] Started /tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu as pid 30229
I20251215 21:01:54.529152 29124 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765832506145157-29124-0/minicluster-data/master-0/wal/instance
I20251215 21:01:54.529340 29124 external_mini_cluster.cc:1468] Setting key dff90d2ae000ecc173b9bc44370641d0
I20251215 21:01:54.529853 30307 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 700c44b9a4694ad6993e6d84e5c0475c: Neither blocks nor log segments found. Creating new log.
I20251215 21:01:54.530634 30307 log.cc:826] T 00000000000000000000000000000000 P 700c44b9a4694ad6993e6d84e5c0475c: Log is configured to *not* fsync() on all Append() calls
I20251215 21:01:54.532374 30307 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 700c44b9a4694ad6993e6d84e5c0475c: No bootstrap required, opened a new log
I20251215 21:01:54.535043 30307 raft_consensus.cc:359] T 00000000000000000000000000000000 P 700c44b9a4694ad6993e6d84e5c0475c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "700c44b9a4694ad6993e6d84e5c0475c" member_type: VOTER last_known_addr { host: "127.28.113.62" port: 40865 } }
I20251215 21:01:54.535230 30307 raft_consensus.cc:385] T 00000000000000000000000000000000 P 700c44b9a4694ad6993e6d84e5c0475c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251215 21:01:54.535288 30307 raft_consensus.cc:740] T 00000000000000000000000000000000 P 700c44b9a4694ad6993e6d84e5c0475c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 700c44b9a4694ad6993e6d84e5c0475c, State: Initialized, Role: FOLLOWER
I20251215 21:01:54.535688 30307 consensus_queue.cc:260] T 00000000000000000000000000000000 P 700c44b9a4694ad6993e6d84e5c0475c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "700c44b9a4694ad6993e6d84e5c0475c" member_type: VOTER last_known_addr { host: "127.28.113.62" port: 40865 } }
I20251215 21:01:54.535811 30307 raft_consensus.cc:399] T 00000000000000000000000000000000 P 700c44b9a4694ad6993e6d84e5c0475c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251215 21:01:54.535878 30307 raft_consensus.cc:493] T 00000000000000000000000000000000 P 700c44b9a4694ad6993e6d84e5c0475c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251215 21:01:54.535988 30307 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 700c44b9a4694ad6993e6d84e5c0475c [term 0 FOLLOWER]: Advancing to term 1
Dec 15 21:01:54 dist-test-slave-bjsf krb5kdc[30213](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765832514, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.113.62@KRBTEST.COM
I20251215 21:01:54.536947 30307 raft_consensus.cc:515] T 00000000000000000000000000000000 P 700c44b9a4694ad6993e6d84e5c0475c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "700c44b9a4694ad6993e6d84e5c0475c" member_type: VOTER last_known_addr { host: "127.28.113.62" port: 40865 } }
I20251215 21:01:54.537308 30307 leader_election.cc:304] T 00000000000000000000000000000000 P 700c44b9a4694ad6993e6d84e5c0475c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 700c44b9a4694ad6993e6d84e5c0475c; no voters:
I20251215 21:01:54.537597 30307 leader_election.cc:290] T 00000000000000000000000000000000 P 700c44b9a4694ad6993e6d84e5c0475c [CANDIDATE]: Term 1 election: Requested vote from peers
I20251215 21:01:54.537686 30312 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 700c44b9a4694ad6993e6d84e5c0475c [term 1 FOLLOWER]: Leader election won for term 1
I20251215 21:01:54.537933 30312 raft_consensus.cc:697] T 00000000000000000000000000000000 P 700c44b9a4694ad6993e6d84e5c0475c [term 1 LEADER]: Becoming Leader. State: Replica: 700c44b9a4694ad6993e6d84e5c0475c, State: Running, Role: LEADER
I20251215 21:01:54.538275 30312 consensus_queue.cc:237] T 00000000000000000000000000000000 P 700c44b9a4694ad6993e6d84e5c0475c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "700c44b9a4694ad6993e6d84e5c0475c" member_type: VOTER last_known_addr { host: "127.28.113.62" port: 40865 } }
I20251215 21:01:54.538709 30307 sys_catalog.cc:565] T 00000000000000000000000000000000 P 700c44b9a4694ad6993e6d84e5c0475c [sys.catalog]: configured and running, proceeding with master startup.
I20251215 21:01:54.539410 30314 sys_catalog.cc:455] T 00000000000000000000000000000000 P 700c44b9a4694ad6993e6d84e5c0475c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 700c44b9a4694ad6993e6d84e5c0475c. Latest consensus state: current_term: 1 leader_uuid: "700c44b9a4694ad6993e6d84e5c0475c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "700c44b9a4694ad6993e6d84e5c0475c" member_type: VOTER last_known_addr { host: "127.28.113.62" port: 40865 } } }
I20251215 21:01:54.540010 30314 sys_catalog.cc:458] T 00000000000000000000000000000000 P 700c44b9a4694ad6993e6d84e5c0475c [sys.catalog]: This master's current role is: LEADER
I20251215 21:01:54.540228 30313 sys_catalog.cc:455] T 00000000000000000000000000000000 P 700c44b9a4694ad6993e6d84e5c0475c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "700c44b9a4694ad6993e6d84e5c0475c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "700c44b9a4694ad6993e6d84e5c0475c" member_type: VOTER last_known_addr { host: "127.28.113.62" port: 40865 } } }
I20251215 21:01:54.540367 30313 sys_catalog.cc:458] T 00000000000000000000000000000000 P 700c44b9a4694ad6993e6d84e5c0475c [sys.catalog]: This master's current role is: LEADER
I20251215 21:01:54.541666 30310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:01:54.530294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40124 (local address 127.28.113.62:40865)
1215 21:01:54.530699 (+ 405us) server_negotiation.cc:207] Beginning negotiation
1215 21:01:54.530709 (+ 10us) server_negotiation.cc:400] Waiting for connection header
1215 21:01:54.530753 (+ 44us) server_negotiation.cc:408] Connection header received
1215 21:01:54.531386 (+ 633us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1215 21:01:54.531406 (+ 20us) server_negotiation.cc:462] Received NEGOTIATE request from client
1215 21:01:54.531688 (+ 282us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1215 21:01:54.532010 (+ 322us) server_negotiation.cc:227] Negotiated authn=SASL
1215 21:01:54.532857 (+ 847us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:01:54.533553 (+ 696us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1215 21:01:54.534164 (+ 611us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:01:54.534430 (+ 266us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:01:54.536680 (+ 2250us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1215 21:01:54.536700 (+ 20us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1215 21:01:54.536710 (+ 10us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1215 21:01:54.536749 (+ 39us) server_negotiation.cc:923] calling sasl_server_start()
1215 21:01:54.538755 (+ 2006us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1215 21:01:54.539258 (+ 503us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1215 21:01:54.539263 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1215 21:01:54.539268 (+ 5us) server_negotiation.cc:968] calling sasl_server_step()
1215 21:01:54.539359 (+ 91us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1215 21:01:54.539626 (+ 267us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1215 21:01:54.539630 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1215 21:01:54.539632 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1215 21:01:54.540073 (+ 441us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1215 21:01:54.540274 (+ 201us) server_negotiation.cc:1036] Waiting for connection context
1215 21:01:54.540685 (+ 411us) server_negotiation.cc:300] Negotiation successful
1215 21:01:54.540957 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":279,"thread_start_us":137,"threads_started":1}
W20251215 21:01:54.544278 30327 catalog_manager.cc:1568] T 00000000000000000000000000000000 P 700c44b9a4694ad6993e6d84e5c0475c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20251215 21:01:54.544342 30327 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20251215 21:01:54.544407 30321 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251215 21:01:54.545089 30321 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251215 21:01:54.550369 30321 catalog_manager.cc:1357] Generated new cluster ID: 8610cf374e7649eab4d7aad9570960f1
I20251215 21:01:54.550436 30321 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251215 21:01:54.564514 30321 catalog_manager.cc:1380] Generated new certificate authority record
I20251215 21:01:54.566017 30321 catalog_manager.cc:1514] Loading token signing keys...
I20251215 21:01:54.586936 30321 catalog_manager.cc:6027] T 00000000000000000000000000000000 P 700c44b9a4694ad6993e6d84e5c0475c: Generated new TSK 0
I20251215 21:01:54.587869 30321 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.113.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.113.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.113.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765832506145157-29124-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.28.113.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765832506145157-29124-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.28.113.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.28.113.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.28.113.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765832506145157-29124-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.28.113.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765832506145157-29124-0/krb5kdc/kudu.keytab.
I20251215 21:01:54.630774 29124 external_mini_cluster.cc:1366] Running /tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu
/tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765832506145157-29124-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765832506145157-29124-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765832506145157-29124-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765832506145157-29124-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.113.1:0
--local_ip_for_outbound_sockets=127.28.113.1
--webserver_interface=127.28.113.1
--webserver_port=0
--tserver_master_addrs=127.28.113.62:40865
--keytab_file=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765832506145157-29124-0/krb5kdc/kudu.keytab
--principal=kudu/127.28.113.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.28.113.20:38907
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765832506145157-29124-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765832506145157-29124-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765832506145157-29124-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251215 21:01:54.732950 30335 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251215 21:01:54.733163 30335 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251215 21:01:54.733225 30335 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251215 21:01:54.736454 30335 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251215 21:01:54.736514 30335 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251215 21:01:54.736568 30335 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20251215 21:01:54.736635 30335 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.113.1
I20251215 21:01:54.740792 30335 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.113.20:38907
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765832506145157-29124-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765832506145157-29124-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765832506145157-29124-0/krb5kdc/kudu.keytab
--principal=kudu/127.28.113.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.113.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765832506145157-29124-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.113.1
--webserver_port=0
--webserver_require_spnego=true
--enable_txn_system_client_init=true
--tserver_master_addrs=127.28.113.62:40865
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30335
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.113.1
--log_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765832506145157-29124-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision dc741732bb6a40213e11a13869fad06a49a209a3
build type DEBUG
built by None at 15 Dec 2025 20:43:14 UTC on 24a791456cd2
build id 9530
I20251215 21:01:54.741820 30335 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251215 21:01:54.742643 30335 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251215 21:01:54.749008 30341 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251215 21:01:54.749017 30340 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251215 21:01:54.749068 30343 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251215 21:01:54.749203 30335 server_base.cc:1047] running on GCE node
I20251215 21:01:54.749562 30335 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251215 21:01:54.750027 30335 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251215 21:01:54.751161 30335 hybrid_clock.cc:648] HybridClock initialized: now 1765832514751140 us; error 35 us; skew 500 ppm
Dec 15 21:01:54 dist-test-slave-bjsf krb5kdc[30213](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765832514, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.113.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251215 21:01:54.753510 30335 init.cc:377] Logged in from keytab as kudu/127.28.113.1@KRBTEST.COM (short username kudu)
I20251215 21:01:54.754462 30335 webserver.cc:492] Webserver started at http://127.28.113.1:33847/ using document root <none> and password file <none>
I20251215 21:01:54.754968 30335 fs_manager.cc:362] Metadata directory not provided
I20251215 21:01:54.755012 30335 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251215 21:01:54.755184 30335 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251215 21:01:54.756805 30335 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765832506145157-29124-0/minicluster-data/ts-0/data/instance:
uuid: "cd74be91011a4300b5edb2f531b0c3c5"
format_stamp: "Formatted at 2025-12-15 21:01:54 on dist-test-slave-bjsf"
server_key: "d7479c90149682cb9a8616e4708e01bb"
server_key_iv: "d8c3f26ea8a020f67e841287630f4b99"
server_key_version: "encryptionkey@0"
I20251215 21:01:54.757228 30335 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765832506145157-29124-0/minicluster-data/ts-0/wal/instance:
uuid: "cd74be91011a4300b5edb2f531b0c3c5"
format_stamp: "Formatted at 2025-12-15 21:01:54 on dist-test-slave-bjsf"
server_key: "d7479c90149682cb9a8616e4708e01bb"
server_key_iv: "d8c3f26ea8a020f67e841287630f4b99"
server_key_version: "encryptionkey@0"
I20251215 21:01:54.760306 30335 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20251215 21:01:54.762297 30350 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251215 21:01:54.763370 30335 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20251215 21:01:54.763456 30335 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765832506145157-29124-0/minicluster-data/ts-0/data,/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765832506145157-29124-0/minicluster-data/ts-0/wal
uuid: "cd74be91011a4300b5edb2f531b0c3c5"
format_stamp: "Formatted at 2025-12-15 21:01:54 on dist-test-slave-bjsf"
server_key: "d7479c90149682cb9a8616e4708e01bb"
server_key_iv: "d8c3f26ea8a020f67e841287630f4b99"
server_key_version: "encryptionkey@0"
I20251215 21:01:54.763541 30335 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765832506145157-29124-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765832506145157-29124-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765832506145157-29124-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251215 21:01:54.775576 30335 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251215 21:01:54.778133 30335 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251215 21:01:54.778282 30335 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251215 21:01:54.779629 30335 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251215 21:01:54.779676 30335 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251215 21:01:54.779716 30335 ts_tablet_manager.cc:616] Registered 0 tablets
I20251215 21:01:54.779745 30335 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Dec 15 21:01:54 dist-test-slave-bjsf krb5kdc[30213](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765832514, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.113.1@KRBTEST.COM for kudu/127.28.113.62@KRBTEST.COM
I20251215 21:01:54.789222 30335 rpc_server.cc:307] RPC server started. Bound to: 127.28.113.1:40521
I20251215 21:01:54.789242 30465 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.28.113.1:40521 every 8 connection(s)
I20251215 21:01:54.790108 30335 server_base.cc:1179] Dumped server information to /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765832506145157-29124-0/minicluster-data/ts-0/data/info.pb
I20251215 21:01:54.791466 30310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:01:54.780365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.113.1:49611 (local address 127.28.113.62:40865)
1215 21:01:54.780511 (+ 146us) server_negotiation.cc:207] Beginning negotiation
1215 21:01:54.780516 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1215 21:01:54.781146 (+ 630us) server_negotiation.cc:408] Connection header received
1215 21:01:54.782030 (+ 884us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1215 21:01:54.782034 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1215 21:01:54.782093 (+ 59us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1215 21:01:54.782189 (+ 96us) server_negotiation.cc:227] Negotiated authn=SASL
1215 21:01:54.783673 (+ 1484us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:01:54.784470 (+ 797us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1215 21:01:54.785290 (+ 820us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:01:54.785501 (+ 211us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:01:54.787901 (+ 2400us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1215 21:01:54.787929 (+ 28us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1215 21:01:54.787933 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1215 21:01:54.787986 (+ 53us) server_negotiation.cc:923] calling sasl_server_start()
1215 21:01:54.789651 (+ 1665us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1215 21:01:54.790085 (+ 434us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1215 21:01:54.790088 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1215 21:01:54.790089 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1215 21:01:54.790131 (+ 42us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1215 21:01:54.790471 (+ 340us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1215 21:01:54.790474 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1215 21:01:54.790476 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1215 21:01:54.790681 (+ 205us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1215 21:01:54.790781 (+ 100us) server_negotiation.cc:1036] Waiting for connection context
1215 21:01:54.791243 (+ 462us) server_negotiation.cc:300] Negotiation successful
1215 21:01:54.791350 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20251215 21:01:54.792044 30359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:01:54.780585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.113.62:40865 (local address 127.28.113.1:49611)
1215 21:01:54.781006 (+ 421us) negotiation.cc:107] Waiting for socket to connect
1215 21:01:54.781044 (+ 38us) client_negotiation.cc:174] Beginning negotiation
1215 21:01:54.781833 (+ 789us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1215 21:01:54.782415 (+ 582us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1215 21:01:54.782424 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
1215 21:01:54.782891 (+ 467us) client_negotiation.cc:189] Negotiated authn=SASL
1215 21:01:54.783514 (+ 623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:01:54.783530 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:01:54.784576 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1215 21:01:54.784580 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1215 21:01:54.785165 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:01:54.785176 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:01:54.785399 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:01:54.786108 (+ 709us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1215 21:01:54.786141 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
1215 21:01:54.787718 (+ 1577us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1215 21:01:54.789775 (+ 2057us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1215 21:01:54.789779 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1215 21:01:54.789787 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
1215 21:01:54.789994 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1215 21:01:54.790268 (+ 274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1215 21:01:54.790272 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1215 21:01:54.790274 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1215 21:01:54.790379 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1215 21:01:54.790777 (+ 398us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1215 21:01:54.790782 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1215 21:01:54.791026 (+ 244us) client_negotiation.cc:769] Sending connection context
1215 21:01:54.791194 (+ 168us) client_negotiation.cc:240] Negotiation successful
1215 21:01:54.791386 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":114,"threads_started":1}
I20251215 21:01:54.793155 30466 heartbeater.cc:344] Connected to a master server at 127.28.113.62:40865
I20251215 21:01:54.793299 30466 heartbeater.cc:461] Registering TS with master...
I20251215 21:01:54.793780 30466 heartbeater.cc:507] Master 127.28.113.62:40865 requested a full tablet report, sending...
I20251215 21:01:54.795137 30260 ts_manager.cc:194] Registered new tserver with Master: cd74be91011a4300b5edb2f531b0c3c5 (127.28.113.1:40521)
I20251215 21:01:54.795822 29124 external_mini_cluster.cc:1428] Started /tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu as pid 30335
I20251215 21:01:54.795905 29124 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765832506145157-29124-0/minicluster-data/ts-0/wal/instance
I20251215 21:01:54.796149 29124 external_mini_cluster.cc:1468] Setting key fd6db6ba3ebca8e1b0ac3cce5aa42b91
I20251215 21:01:54.796386 30260 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.113.1@KRBTEST.COM'} at 127.28.113.1:49611
I20251215 21:01:54.802453 30310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:01:54.796905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.113.1:32935 (local address 127.28.113.62:40865)
1215 21:01:54.797018 (+ 113us) server_negotiation.cc:207] Beginning negotiation
1215 21:01:54.797021 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1215 21:01:54.797031 (+ 10us) server_negotiation.cc:408] Connection header received
1215 21:01:54.797058 (+ 27us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1215 21:01:54.797060 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1215 21:01:54.797093 (+ 33us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1215 21:01:54.797157 (+ 64us) server_negotiation.cc:227] Negotiated authn=SASL
1215 21:01:54.797852 (+ 695us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:01:54.798313 (+ 461us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1215 21:01:54.798986 (+ 673us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:01:54.799159 (+ 173us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:01:54.799933 (+ 774us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1215 21:01:54.799946 (+ 13us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1215 21:01:54.799948 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1215 21:01:54.799988 (+ 40us) server_negotiation.cc:923] calling sasl_server_start()
1215 21:01:54.801226 (+ 1238us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1215 21:01:54.801566 (+ 340us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1215 21:01:54.801569 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1215 21:01:54.801570 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1215 21:01:54.801615 (+ 45us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1215 21:01:54.801799 (+ 184us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1215 21:01:54.801803 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1215 21:01:54.801805 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1215 21:01:54.801928 (+ 123us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1215 21:01:54.802005 (+ 77us) server_negotiation.cc:1036] Waiting for connection context
1215 21:01:54.802216 (+ 211us) server_negotiation.cc:300] Negotiation successful
1215 21:01:54.802311 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20251215 21:01:54.802469 30472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:01:54.794847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.113.62:40865 (local address 127.28.113.1:32935)
1215 21:01:54.795101 (+ 254us) negotiation.cc:107] Waiting for socket to connect
1215 21:01:54.795115 (+ 14us) client_negotiation.cc:174] Beginning negotiation
1215 21:01:54.795198 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1215 21:01:54.797255 (+ 2057us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1215 21:01:54.797258 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1215 21:01:54.797451 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
1215 21:01:54.797720 (+ 269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:01:54.797729 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:01:54.798418 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1215 21:01:54.798422 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1215 21:01:54.798878 (+ 456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:01:54.798887 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:01:54.798985 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:01:54.799541 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1215 21:01:54.799553 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
1215 21:01:54.799812 (+ 259us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1215 21:01:54.801323 (+ 1511us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1215 21:01:54.801327 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1215 21:01:54.801329 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1215 21:01:54.801481 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1215 21:01:54.801686 (+ 205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1215 21:01:54.801689 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1215 21:01:54.801690 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1215 21:01:54.801731 (+ 41us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1215 21:01:54.802005 (+ 274us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1215 21:01:54.802009 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1215 21:01:54.802080 (+ 71us) client_negotiation.cc:769] Sending connection context
1215 21:01:54.802161 (+ 81us) client_negotiation.cc:240] Negotiation successful
1215 21:01:54.802281 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":82,"threads_started":1}
WARNING: no policy specified for kudu/127.28.113.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.113.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.113.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765832506145157-29124-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.28.113.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765832506145157-29124-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.28.113.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.28.113.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.28.113.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765832506145157-29124-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.28.113.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765832506145157-29124-0/krb5kdc/kudu.keytab.
I20251215 21:01:54.836686 29124 external_mini_cluster.cc:1366] Running /tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu
/tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765832506145157-29124-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765832506145157-29124-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765832506145157-29124-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765832506145157-29124-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.113.2:0
--local_ip_for_outbound_sockets=127.28.113.2
--webserver_interface=127.28.113.2
--webserver_port=0
--tserver_master_addrs=127.28.113.62:40865
--keytab_file=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765832506145157-29124-0/krb5kdc/kudu.keytab
--principal=kudu/127.28.113.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.28.113.20:38907
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765832506145157-29124-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765832506145157-29124-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765832506145157-29124-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251215 21:01:54.938241 30477 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251215 21:01:54.938459 30477 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251215 21:01:54.938535 30477 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251215 21:01:54.941826 30477 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251215 21:01:54.941883 30477 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251215 21:01:54.941947 30477 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20251215 21:01:54.942009 30477 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.113.2
I20251215 21:01:54.946187 30477 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.113.20:38907
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765832506145157-29124-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765832506145157-29124-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765832506145157-29124-0/krb5kdc/kudu.keytab
--principal=kudu/127.28.113.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.113.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765832506145157-29124-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.113.2
--webserver_port=0
--webserver_require_spnego=true
--enable_txn_system_client_init=true
--tserver_master_addrs=127.28.113.62:40865
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30477
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.113.2
--log_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765832506145157-29124-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision dc741732bb6a40213e11a13869fad06a49a209a3
build type DEBUG
built by None at 15 Dec 2025 20:43:14 UTC on 24a791456cd2
build id 9530
I20251215 21:01:54.947222 30477 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251215 21:01:54.948153 30477 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251215 21:01:54.954196 30482 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251215 21:01:54.954198 30483 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251215 21:01:54.954208 30485 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251215 21:01:54.954398 30477 server_base.cc:1047] running on GCE node
I20251215 21:01:54.954831 30477 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251215 21:01:54.955353 30477 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251215 21:01:54.956501 30477 hybrid_clock.cc:648] HybridClock initialized: now 1765832514956483 us; error 35 us; skew 500 ppm
Dec 15 21:01:54 dist-test-slave-bjsf krb5kdc[30213](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765832514, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.113.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251215 21:01:54.958978 30477 init.cc:377] Logged in from keytab as kudu/127.28.113.2@KRBTEST.COM (short username kudu)
I20251215 21:01:54.959934 30477 webserver.cc:492] Webserver started at http://127.28.113.2:38431/ using document root <none> and password file <none>
I20251215 21:01:54.960456 30477 fs_manager.cc:362] Metadata directory not provided
I20251215 21:01:54.960503 30477 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251215 21:01:54.960655 30477 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251215 21:01:54.962291 30477 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765832506145157-29124-0/minicluster-data/ts-1/data/instance:
uuid: "10081c0117c2422780822c7590f0b49f"
format_stamp: "Formatted at 2025-12-15 21:01:54 on dist-test-slave-bjsf"
server_key: "eacd2dfed01855561b6478dec4dfbee5"
server_key_iv: "0f206022df6e6c7ceadaa4335c01fab3"
server_key_version: "encryptionkey@0"
I20251215 21:01:54.962719 30477 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765832506145157-29124-0/minicluster-data/ts-1/wal/instance:
uuid: "10081c0117c2422780822c7590f0b49f"
format_stamp: "Formatted at 2025-12-15 21:01:54 on dist-test-slave-bjsf"
server_key: "eacd2dfed01855561b6478dec4dfbee5"
server_key_iv: "0f206022df6e6c7ceadaa4335c01fab3"
server_key_version: "encryptionkey@0"
I20251215 21:01:54.965865 30477 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20251215 21:01:54.967808 30492 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251215 21:01:54.968874 30477 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20251215 21:01:54.968977 30477 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765832506145157-29124-0/minicluster-data/ts-1/data,/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765832506145157-29124-0/minicluster-data/ts-1/wal
uuid: "10081c0117c2422780822c7590f0b49f"
format_stamp: "Formatted at 2025-12-15 21:01:54 on dist-test-slave-bjsf"
server_key: "eacd2dfed01855561b6478dec4dfbee5"
server_key_iv: "0f206022df6e6c7ceadaa4335c01fab3"
server_key_version: "encryptionkey@0"
I20251215 21:01:54.969062 30477 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765832506145157-29124-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765832506145157-29124-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765832506145157-29124-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251215 21:01:54.985401 30477 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251215 21:01:54.987843 30477 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251215 21:01:54.988029 30477 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251215 21:01:54.989425 30477 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251215 21:01:54.989475 30477 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251215 21:01:54.989526 30477 ts_tablet_manager.cc:616] Registered 0 tablets
I20251215 21:01:54.989554 30477 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Dec 15 21:01:54 dist-test-slave-bjsf krb5kdc[30213](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765832514, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.113.2@KRBTEST.COM for kudu/127.28.113.62@KRBTEST.COM
I20251215 21:01:55.000015 30477 rpc_server.cc:307] RPC server started. Bound to: 127.28.113.2:36671
I20251215 21:01:55.000043 30607 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.28.113.2:36671 every 8 connection(s)
I20251215 21:01:55.001061 30310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:01:54.989981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.113.2:55567 (local address 127.28.113.62:40865)
1215 21:01:54.990132 (+ 151us) server_negotiation.cc:207] Beginning negotiation
1215 21:01:54.990138 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1215 21:01:54.990799 (+ 661us) server_negotiation.cc:408] Connection header received
1215 21:01:54.991663 (+ 864us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1215 21:01:54.991667 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1215 21:01:54.991729 (+ 62us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1215 21:01:54.991828 (+ 99us) server_negotiation.cc:227] Negotiated authn=SASL
1215 21:01:54.993366 (+ 1538us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:01:54.994150 (+ 784us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1215 21:01:54.994855 (+ 705us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:01:54.995033 (+ 178us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:01:54.997575 (+ 2542us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1215 21:01:54.997601 (+ 26us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1215 21:01:54.997604 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1215 21:01:54.997630 (+ 26us) server_negotiation.cc:923] calling sasl_server_start()
1215 21:01:54.999124 (+ 1494us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1215 21:01:54.999634 (+ 510us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1215 21:01:54.999638 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1215 21:01:54.999640 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1215 21:01:54.999689 (+ 49us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1215 21:01:55.000018 (+ 329us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1215 21:01:55.000022 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1215 21:01:55.000024 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1215 21:01:55.000218 (+ 194us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1215 21:01:55.000315 (+ 97us) server_negotiation.cc:1036] Waiting for connection context
1215 21:01:55.000785 (+ 470us) server_negotiation.cc:300] Negotiation successful
1215 21:01:55.000907 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20251215 21:01:55.001041 30477 server_base.cc:1179] Dumped server information to /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765832506145157-29124-0/minicluster-data/ts-1/data/info.pb
I20251215 21:01:55.001644 29124 external_mini_cluster.cc:1428] Started /tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu as pid 30477
I20251215 21:01:55.001749 29124 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765832506145157-29124-0/minicluster-data/ts-1/wal/instance
I20251215 21:01:55.001893 30501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:01:54.990239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.113.62:40865 (local address 127.28.113.2:55567)
1215 21:01:54.990656 (+ 417us) negotiation.cc:107] Waiting for socket to connect
1215 21:01:54.990696 (+ 40us) client_negotiation.cc:174] Beginning negotiation
1215 21:01:54.991471 (+ 775us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1215 21:01:54.991987 (+ 516us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1215 21:01:54.991997 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
1215 21:01:54.992545 (+ 548us) client_negotiation.cc:189] Negotiated authn=SASL
1215 21:01:54.993218 (+ 673us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:01:54.993231 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:01:54.994272 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1215 21:01:54.994276 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1215 21:01:54.994744 (+ 468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:01:54.994753 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:01:54.994978 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:01:54.995607 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1215 21:01:54.995644 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
1215 21:01:54.997421 (+ 1777us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1215 21:01:54.999227 (+ 1806us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1215 21:01:54.999233 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1215 21:01:54.999243 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
1215 21:01:54.999531 (+ 288us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1215 21:01:54.999768 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1215 21:01:54.999771 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1215 21:01:54.999774 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
1215 21:01:54.999880 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1215 21:01:55.000317 (+ 437us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1215 21:01:55.000322 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1215 21:01:55.000532 (+ 210us) client_negotiation.cc:769] Sending connection context
1215 21:01:55.000707 (+ 175us) client_negotiation.cc:240] Negotiation successful
1215 21:01:55.000908 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":101,"threads_started":1}
I20251215 21:01:55.001995 29124 external_mini_cluster.cc:1468] Setting key c0e707d4fa327f7c314e52f4eef594cf
I20251215 21:01:55.003032 30608 heartbeater.cc:344] Connected to a master server at 127.28.113.62:40865
I20251215 21:01:55.003183 30608 heartbeater.cc:461] Registering TS with master...
I20251215 21:01:55.003733 30608 heartbeater.cc:507] Master 127.28.113.62:40865 requested a full tablet report, sending...
I20251215 21:01:55.004741 30260 ts_manager.cc:194] Registered new tserver with Master: 10081c0117c2422780822c7590f0b49f (127.28.113.2:36671)
I20251215 21:01:55.005396 30260 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.113.2@KRBTEST.COM'} at 127.28.113.2:55567
WARNING: no policy specified for kudu/127.28.113.3@KRBTEST.COM; defaulting to no policy
I20251215 21:01:55.011659 30310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:01:55.005150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.113.2:40683 (local address 127.28.113.62:40865)
1215 21:01:55.005257 (+ 107us) server_negotiation.cc:207] Beginning negotiation
1215 21:01:55.005261 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1215 21:01:55.005489 (+ 228us) server_negotiation.cc:408] Connection header received
1215 21:01:55.005603 (+ 114us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1215 21:01:55.005606 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1215 21:01:55.005654 (+ 48us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1215 21:01:55.005722 (+ 68us) server_negotiation.cc:227] Negotiated authn=SASL
1215 21:01:55.006379 (+ 657us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:01:55.007050 (+ 671us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1215 21:01:55.007856 (+ 806us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:01:55.008053 (+ 197us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:01:55.008929 (+ 876us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1215 21:01:55.008943 (+ 14us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1215 21:01:55.008945 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1215 21:01:55.008967 (+ 22us) server_negotiation.cc:923] calling sasl_server_start()
1215 21:01:55.010230 (+ 1263us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1215 21:01:55.010675 (+ 445us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1215 21:01:55.010678 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1215 21:01:55.010679 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1215 21:01:55.010715 (+ 36us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1215 21:01:55.010948 (+ 233us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1215 21:01:55.010950 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1215 21:01:55.010951 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1215 21:01:55.011105 (+ 154us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1215 21:01:55.011174 (+ 69us) server_negotiation.cc:1036] Waiting for connection context
1215 21:01:55.011436 (+ 262us) server_negotiation.cc:300] Negotiation successful
1215 21:01:55.011530 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20251215 21:01:55.011726 30615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:01:55.005068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.113.62:40865 (local address 127.28.113.2:40683)
1215 21:01:55.005404 (+ 336us) negotiation.cc:107] Waiting for socket to connect
1215 21:01:55.005419 (+ 15us) client_negotiation.cc:174] Beginning negotiation
1215 21:01:55.005518 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1215 21:01:55.005741 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1215 21:01:55.005745 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
1215 21:01:55.005964 (+ 219us) client_negotiation.cc:189] Negotiated authn=SASL
1215 21:01:55.006244 (+ 280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:01:55.006252 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:01:55.007185 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1215 21:01:55.007189 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1215 21:01:55.007737 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:01:55.007748 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:01:55.007867 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:01:55.008402 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1215 21:01:55.008419 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
1215 21:01:55.008802 (+ 383us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1215 21:01:55.010341 (+ 1539us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1215 21:01:55.010345 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1215 21:01:55.010347 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1215 21:01:55.010555 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1215 21:01:55.010798 (+ 243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1215 21:01:55.010802 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1215 21:01:55.010804 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1215 21:01:55.010862 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1215 21:01:55.011194 (+ 332us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1215 21:01:55.011197 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1215 21:01:55.011296 (+ 99us) client_negotiation.cc:769] Sending connection context
1215 21:01:55.011404 (+ 108us) client_negotiation.cc:240] Negotiation successful
1215 21:01:55.011542 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":82,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.113.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.113.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765832506145157-29124-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.28.113.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765832506145157-29124-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.28.113.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.28.113.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.28.113.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765832506145157-29124-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.28.113.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765832506145157-29124-0/krb5kdc/kudu.keytab.
I20251215 21:01:55.044471 29124 external_mini_cluster.cc:1366] Running /tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu
/tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765832506145157-29124-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765832506145157-29124-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765832506145157-29124-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765832506145157-29124-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.113.3:0
--local_ip_for_outbound_sockets=127.28.113.3
--webserver_interface=127.28.113.3
--webserver_port=0
--tserver_master_addrs=127.28.113.62:40865
--keytab_file=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765832506145157-29124-0/krb5kdc/kudu.keytab
--principal=kudu/127.28.113.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.28.113.20:38907
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765832506145157-29124-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765832506145157-29124-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765832506145157-29124-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251215 21:01:55.147625 30619 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251215 21:01:55.147826 30619 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251215 21:01:55.147884 30619 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251215 21:01:55.151006 30619 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251215 21:01:55.151058 30619 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251215 21:01:55.151108 30619 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20251215 21:01:55.151149 30619 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.113.3
I20251215 21:01:55.155063 30619 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.113.20:38907
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765832506145157-29124-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765832506145157-29124-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765832506145157-29124-0/krb5kdc/kudu.keytab
--principal=kudu/127.28.113.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.113.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765832506145157-29124-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.113.3
--webserver_port=0
--webserver_require_spnego=true
--enable_txn_system_client_init=true
--tserver_master_addrs=127.28.113.62:40865
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30619
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.113.3
--log_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765832506145157-29124-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision dc741732bb6a40213e11a13869fad06a49a209a3
build type DEBUG
built by None at 15 Dec 2025 20:43:14 UTC on 24a791456cd2
build id 9530
I20251215 21:01:55.156028 30619 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251215 21:01:55.156781 30619 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251215 21:01:55.162955 30624 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251215 21:01:55.162954 30627 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251215 21:01:55.163151 30619 server_base.cc:1047] running on GCE node
W20251215 21:01:55.163161 30625 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251215 21:01:55.163503 30619 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251215 21:01:55.164019 30619 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251215 21:01:55.165150 30619 hybrid_clock.cc:648] HybridClock initialized: now 1765832515165140 us; error 26 us; skew 500 ppm
Dec 15 21:01:55 dist-test-slave-bjsf krb5kdc[30213](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765832515, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.113.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251215 21:01:55.167510 30619 init.cc:377] Logged in from keytab as kudu/127.28.113.3@KRBTEST.COM (short username kudu)
I20251215 21:01:55.168488 30619 webserver.cc:492] Webserver started at http://127.28.113.3:46777/ using document root <none> and password file <none>
I20251215 21:01:55.168980 30619 fs_manager.cc:362] Metadata directory not provided
I20251215 21:01:55.169023 30619 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251215 21:01:55.169165 30619 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251215 21:01:55.170727 30619 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765832506145157-29124-0/minicluster-data/ts-2/data/instance:
uuid: "47a2c6b9946243d4b031a7d3c187f61b"
format_stamp: "Formatted at 2025-12-15 21:01:55 on dist-test-slave-bjsf"
server_key: "97191f3e2c79ab240665be353b041b18"
server_key_iv: "2cdb2d709868ff105fe1ba83df911b1d"
server_key_version: "encryptionkey@0"
I20251215 21:01:55.171119 30619 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765832506145157-29124-0/minicluster-data/ts-2/wal/instance:
uuid: "47a2c6b9946243d4b031a7d3c187f61b"
format_stamp: "Formatted at 2025-12-15 21:01:55 on dist-test-slave-bjsf"
server_key: "97191f3e2c79ab240665be353b041b18"
server_key_iv: "2cdb2d709868ff105fe1ba83df911b1d"
server_key_version: "encryptionkey@0"
I20251215 21:01:55.174288 30619 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20251215 21:01:55.176251 30634 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251215 21:01:55.177204 30619 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s
I20251215 21:01:55.177292 30619 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765832506145157-29124-0/minicluster-data/ts-2/data,/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765832506145157-29124-0/minicluster-data/ts-2/wal
uuid: "47a2c6b9946243d4b031a7d3c187f61b"
format_stamp: "Formatted at 2025-12-15 21:01:55 on dist-test-slave-bjsf"
server_key: "97191f3e2c79ab240665be353b041b18"
server_key_iv: "2cdb2d709868ff105fe1ba83df911b1d"
server_key_version: "encryptionkey@0"
I20251215 21:01:55.177363 30619 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765832506145157-29124-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765832506145157-29124-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765832506145157-29124-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251215 21:01:55.212077 30619 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251215 21:01:55.215592 30619 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251215 21:01:55.215785 30619 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251215 21:01:55.217615 30619 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251215 21:01:55.217677 30619 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251215 21:01:55.217733 30619 ts_tablet_manager.cc:616] Registered 0 tablets
I20251215 21:01:55.217757 30619 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Dec 15 21:01:55 dist-test-slave-bjsf krb5kdc[30213](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765832515, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.113.3@KRBTEST.COM for kudu/127.28.113.62@KRBTEST.COM
I20251215 21:01:55.227428 30619 rpc_server.cc:307] RPC server started. Bound to: 127.28.113.3:35943
I20251215 21:01:55.227483 30749 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.28.113.3:35943 every 8 connection(s)
I20251215 21:01:55.228334 30619 server_base.cc:1179] Dumped server information to /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765832506145157-29124-0/minicluster-data/ts-2/data/info.pb
I20251215 21:01:55.229955 29124 external_mini_cluster.cc:1428] Started /tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu as pid 30619
I20251215 21:01:55.230050 29124 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765832506145157-29124-0/minicluster-data/ts-2/wal/instance
I20251215 21:01:55.230271 29124 external_mini_cluster.cc:1468] Setting key bd3335140653810e2c4f941f112e3132
I20251215 21:01:55.230311 30310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:01:55.218799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.113.3:52175 (local address 127.28.113.62:40865)
1215 21:01:55.218969 (+ 170us) server_negotiation.cc:207] Beginning negotiation
1215 21:01:55.218975 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1215 21:01:55.219748 (+ 773us) server_negotiation.cc:408] Connection header received
1215 21:01:55.220791 (+ 1043us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1215 21:01:55.220795 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1215 21:01:55.220870 (+ 75us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1215 21:01:55.220972 (+ 102us) server_negotiation.cc:227] Negotiated authn=SASL
1215 21:01:55.222508 (+ 1536us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:01:55.223251 (+ 743us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1215 21:01:55.224137 (+ 886us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:01:55.224332 (+ 195us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:01:55.226811 (+ 2479us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1215 21:01:55.226829 (+ 18us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1215 21:01:55.226831 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1215 21:01:55.226856 (+ 25us) server_negotiation.cc:923] calling sasl_server_start()
1215 21:01:55.228278 (+ 1422us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1215 21:01:55.228937 (+ 659us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1215 21:01:55.228940 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1215 21:01:55.228942 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1215 21:01:55.228997 (+ 55us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1215 21:01:55.229303 (+ 306us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1215 21:01:55.229307 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1215 21:01:55.229309 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1215 21:01:55.229494 (+ 185us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1215 21:01:55.229581 (+ 87us) server_negotiation.cc:1036] Waiting for connection context
1215 21:01:55.230044 (+ 463us) server_negotiation.cc:300] Negotiation successful
1215 21:01:55.230160 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20251215 21:01:55.230896 30643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:01:55.219147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.113.62:40865 (local address 127.28.113.3:52175)
1215 21:01:55.219600 (+ 453us) negotiation.cc:107] Waiting for socket to connect
1215 21:01:55.219642 (+ 42us) client_negotiation.cc:174] Beginning negotiation
1215 21:01:55.220536 (+ 894us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1215 21:01:55.221140 (+ 604us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1215 21:01:55.221153 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
1215 21:01:55.221622 (+ 469us) client_negotiation.cc:189] Negotiated authn=SASL
1215 21:01:55.222368 (+ 746us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:01:55.222386 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:01:55.223378 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1215 21:01:55.223383 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1215 21:01:55.224021 (+ 638us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:01:55.224031 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:01:55.224259 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:01:55.224891 (+ 632us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1215 21:01:55.224922 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
1215 21:01:55.226670 (+ 1748us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1215 21:01:55.228470 (+ 1800us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1215 21:01:55.228476 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1215 21:01:55.228487 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
1215 21:01:55.228810 (+ 323us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1215 21:01:55.229084 (+ 274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1215 21:01:55.229107 (+ 23us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1215 21:01:55.229110 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
1215 21:01:55.229210 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1215 21:01:55.229600 (+ 390us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1215 21:01:55.229606 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1215 21:01:55.229814 (+ 208us) client_negotiation.cc:769] Sending connection context
1215 21:01:55.229989 (+ 175us) client_negotiation.cc:240] Negotiation successful
1215 21:01:55.230225 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":111,"threads_started":1}
I20251215 21:01:55.232051 30750 heartbeater.cc:344] Connected to a master server at 127.28.113.62:40865
I20251215 21:01:55.232201 30750 heartbeater.cc:461] Registering TS with master...
I20251215 21:01:55.232714 30750 heartbeater.cc:507] Master 127.28.113.62:40865 requested a full tablet report, sending...
I20251215 21:01:55.233564 30260 ts_manager.cc:194] Registered new tserver with Master: 47a2c6b9946243d4b031a7d3c187f61b (127.28.113.3:35943)
I20251215 21:01:55.234172 30260 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.113.3@KRBTEST.COM'} at 127.28.113.3:52175
I20251215 21:01:55.238962 30756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:01:55.233753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.113.62:40865 (local address 127.28.113.3:59571)
1215 21:01:55.234014 (+ 261us) negotiation.cc:107] Waiting for socket to connect
1215 21:01:55.234025 (+ 11us) client_negotiation.cc:174] Beginning negotiation
1215 21:01:55.234097 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1215 21:01:55.234292 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1215 21:01:55.234295 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1215 21:01:55.234445 (+ 150us) client_negotiation.cc:189] Negotiated authn=SASL
1215 21:01:55.234700 (+ 255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:01:55.234708 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:01:55.235584 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1215 21:01:55.235586 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1215 21:01:55.235992 (+ 406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:01:55.236001 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:01:55.236086 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:01:55.236520 (+ 434us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1215 21:01:55.236532 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
1215 21:01:55.236787 (+ 255us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1215 21:01:55.238060 (+ 1273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1215 21:01:55.238062 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1215 21:01:55.238064 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1215 21:01:55.238205 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1215 21:01:55.238361 (+ 156us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1215 21:01:55.238363 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1215 21:01:55.238364 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1215 21:01:55.238398 (+ 34us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1215 21:01:55.238609 (+ 211us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1215 21:01:55.238611 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1215 21:01:55.238677 (+ 66us) client_negotiation.cc:769] Sending connection context
1215 21:01:55.238761 (+ 84us) client_negotiation.cc:240] Negotiation successful
1215 21:01:55.238846 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":81,"threads_started":1}
I20251215 21:01:55.238968 30310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:01:55.233811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.113.3:59571 (local address 127.28.113.62:40865)
1215 21:01:55.233924 (+ 113us) server_negotiation.cc:207] Beginning negotiation
1215 21:01:55.233928 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1215 21:01:55.234078 (+ 150us) server_negotiation.cc:408] Connection header received
1215 21:01:55.234155 (+ 77us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1215 21:01:55.234158 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1215 21:01:55.234201 (+ 43us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1215 21:01:55.234277 (+ 76us) server_negotiation.cc:227] Negotiated authn=SASL
1215 21:01:55.234812 (+ 535us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:01:55.235493 (+ 681us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1215 21:01:55.236089 (+ 596us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:01:55.236215 (+ 126us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:01:55.236873 (+ 658us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1215 21:01:55.236884 (+ 11us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1215 21:01:55.236886 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1215 21:01:55.236901 (+ 15us) server_negotiation.cc:923] calling sasl_server_start()
1215 21:01:55.237976 (+ 1075us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1215 21:01:55.238276 (+ 300us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1215 21:01:55.238278 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1215 21:01:55.238279 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1215 21:01:55.238310 (+ 31us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1215 21:01:55.238443 (+ 133us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1215 21:01:55.238445 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1215 21:01:55.238446 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1215 21:01:55.238551 (+ 105us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1215 21:01:55.238610 (+ 59us) server_negotiation.cc:1036] Waiting for connection context
1215 21:01:55.238790 (+ 180us) server_negotiation.cc:300] Negotiation successful
1215 21:01:55.238872 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20251215 21:01:55.244076 29124 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
Dec 15 21:01:55 dist-test-slave-bjsf krb5kdc[30213](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765832515, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Dec 15 21:01:55 dist-test-slave-bjsf krb5kdc[30213](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765832515, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.28.113.62@KRBTEST.COM
I20251215 21:01:55.264868 30310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:01:55.258377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40128 (local address 127.28.113.62:40865)
1215 21:01:55.258501 (+ 124us) server_negotiation.cc:207] Beginning negotiation
1215 21:01:55.258505 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1215 21:01:55.258516 (+ 11us) server_negotiation.cc:408] Connection header received
1215 21:01:55.258553 (+ 37us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1215 21:01:55.258556 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1215 21:01:55.258606 (+ 50us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1215 21:01:55.258671 (+ 65us) server_negotiation.cc:227] Negotiated authn=SASL
1215 21:01:55.259398 (+ 727us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:01:55.259883 (+ 485us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1215 21:01:55.260532 (+ 649us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:01:55.260715 (+ 183us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:01:55.262569 (+ 1854us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1215 21:01:55.262592 (+ 23us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1215 21:01:55.262594 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1215 21:01:55.262619 (+ 25us) server_negotiation.cc:923] calling sasl_server_start()
1215 21:01:55.263785 (+ 1166us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1215 21:01:55.264146 (+ 361us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1215 21:01:55.264149 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1215 21:01:55.264150 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1215 21:01:55.264187 (+ 37us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1215 21:01:55.264329 (+ 142us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1215 21:01:55.264331 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1215 21:01:55.264332 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1215 21:01:55.264456 (+ 124us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1215 21:01:55.264545 (+ 89us) server_negotiation.cc:1036] Waiting for connection context
1215 21:01:55.264672 (+ 127us) server_negotiation.cc:300] Negotiation successful
1215 21:01:55.264754 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20251215 21:01:55.267452 30260 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40128:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251215 21:01:55.269759 30260 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251215 21:01:55.281337 30771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:01:55.277676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.113.2:36671 (local address 127.0.0.1:44928)
1215 21:01:55.278423 (+ 747us) negotiation.cc:107] Waiting for socket to connect
1215 21:01:55.278434 (+ 11us) client_negotiation.cc:174] Beginning negotiation
1215 21:01:55.278495 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1215 21:01:55.279002 (+ 507us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1215 21:01:55.279005 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1215 21:01:55.279021 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1215 21:01:55.279213 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:01:55.279220 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:01:55.280174 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1215 21:01:55.280177 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1215 21:01:55.280881 (+ 704us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:01:55.280890 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:01:55.280987 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:01:55.281042 (+ 55us) client_negotiation.cc:769] Sending connection context
1215 21:01:55.281126 (+ 84us) client_negotiation.cc:240] Negotiation successful
1215 21:01:55.281209 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":688,"thread_start_us":87,"threads_started":1}
I20251215 21:01:55.282068 30767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:01:55.277045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.113.3:35943 (local address 127.0.0.1:58698)
1215 21:01:55.278056 (+ 1011us) negotiation.cc:107] Waiting for socket to connect
1215 21:01:55.278095 (+ 39us) client_negotiation.cc:174] Beginning negotiation
1215 21:01:55.278231 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1215 21:01:55.278853 (+ 622us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1215 21:01:55.278856 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1215 21:01:55.278870 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1215 21:01:55.279095 (+ 225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:01:55.279102 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:01:55.280094 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1215 21:01:55.280098 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1215 21:01:55.281009 (+ 911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:01:55.281019 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:01:55.281822 (+ 803us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:01:55.281835 (+ 13us) client_negotiation.cc:769] Sending connection context
1215 21:01:55.281900 (+ 65us) client_negotiation.cc:240] Negotiation successful
1215 21:01:55.281956 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":855,"thread_start_us":70,"threads_started":1}
I20251215 21:01:55.282166 30768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:01:55.277203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58698 (local address 127.28.113.3:35943)
1215 21:01:55.277538 (+ 335us) server_negotiation.cc:207] Beginning negotiation
1215 21:01:55.277545 (+ 7us) server_negotiation.cc:400] Waiting for connection header
1215 21:01:55.278174 (+ 629us) server_negotiation.cc:408] Connection header received
1215 21:01:55.278391 (+ 217us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1215 21:01:55.278397 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
1215 21:01:55.278579 (+ 182us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1215 21:01:55.278667 (+ 88us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1215 21:01:55.279213 (+ 546us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:01:55.279946 (+ 733us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1215 21:01:55.281165 (+ 1219us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:01:55.281638 (+ 473us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:01:55.281746 (+ 108us) server_negotiation.cc:1036] Waiting for connection context
1215 21:01:55.281959 (+ 213us) server_negotiation.cc:300] Negotiation successful
1215 21:01:55.282043 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":246,"thread_start_us":81,"threads_started":1}
I20251215 21:01:55.282514 30769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:01:55.277473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.113.1:40521 (local address 127.0.0.1:37470)
1215 21:01:55.278056 (+ 583us) negotiation.cc:107] Waiting for socket to connect
1215 21:01:55.278095 (+ 39us) client_negotiation.cc:174] Beginning negotiation
1215 21:01:55.278231 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1215 21:01:55.278773 (+ 542us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1215 21:01:55.278779 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
1215 21:01:55.278801 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1215 21:01:55.279095 (+ 294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:01:55.279102 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:01:55.281297 (+ 2195us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1215 21:01:55.281300 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1215 21:01:55.282167 (+ 867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:01:55.282177 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:01:55.282282 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:01:55.282297 (+ 15us) client_negotiation.cc:769] Sending connection context
1215 21:01:55.282343 (+ 46us) client_negotiation.cc:240] Negotiation successful
1215 21:01:55.282409 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":492,"thread_start_us":102,"threads_started":1}
I20251215 21:01:55.282683 30772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:01:55.277836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44928 (local address 127.28.113.2:36671)
1215 21:01:55.278298 (+ 462us) server_negotiation.cc:207] Beginning negotiation
1215 21:01:55.278514 (+ 216us) server_negotiation.cc:400] Waiting for connection header
1215 21:01:55.278544 (+ 30us) server_negotiation.cc:408] Connection header received
1215 21:01:55.278615 (+ 71us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1215 21:01:55.278619 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1215 21:01:55.278751 (+ 132us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1215 21:01:55.278861 (+ 110us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1215 21:01:55.279324 (+ 463us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:01:55.280081 (+ 757us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1215 21:01:55.281628 (+ 1547us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:01:55.282129 (+ 501us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:01:55.282419 (+ 290us) server_negotiation.cc:1036] Waiting for connection context
1215 21:01:55.282477 (+ 58us) server_negotiation.cc:300] Negotiation successful
1215 21:01:55.282558 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":372,"thread_start_us":89,"threads_started":1}
I20251215 21:01:55.283120 30770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:01:55.277636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37470 (local address 127.28.113.1:40521)
1215 21:01:55.277958 (+ 322us) server_negotiation.cc:207] Beginning negotiation
1215 21:01:55.277965 (+ 7us) server_negotiation.cc:400] Waiting for connection header
1215 21:01:55.278350 (+ 385us) server_negotiation.cc:408] Connection header received
1215 21:01:55.278443 (+ 93us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1215 21:01:55.278448 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1215 21:01:55.278626 (+ 178us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1215 21:01:55.278765 (+ 139us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1215 21:01:55.280105 (+ 1340us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:01:55.281198 (+ 1093us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1215 21:01:55.282313 (+ 1115us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:01:55.282829 (+ 516us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:01:55.282918 (+ 89us) server_negotiation.cc:1036] Waiting for connection context
1215 21:01:55.282968 (+ 50us) server_negotiation.cc:300] Negotiation successful
1215 21:01:55.283029 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":237,"thread_start_us":81,"threads_started":1}
I20251215 21:01:55.285192 30683 tablet_service.cc:1505] Processing CreateTablet for tablet 2c138804eabc4370b35ba28052a5f187 (DEFAULT_TABLE table=test-table [id=fbdc85fe372540dbb6aca1c724eddbdd]), partition=RANGE (key) PARTITION UNBOUNDED
I20251215 21:01:55.285678 30542 tablet_service.cc:1505] Processing CreateTablet for tablet 2c138804eabc4370b35ba28052a5f187 (DEFAULT_TABLE table=test-table [id=fbdc85fe372540dbb6aca1c724eddbdd]), partition=RANGE (key) PARTITION UNBOUNDED
I20251215 21:01:55.285678 30400 tablet_service.cc:1505] Processing CreateTablet for tablet 2c138804eabc4370b35ba28052a5f187 (DEFAULT_TABLE table=test-table [id=fbdc85fe372540dbb6aca1c724eddbdd]), partition=RANGE (key) PARTITION UNBOUNDED
I20251215 21:01:55.286020 30683 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2c138804eabc4370b35ba28052a5f187. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 21:01:55.286545 30542 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2c138804eabc4370b35ba28052a5f187. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 21:01:55.286545 30400 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2c138804eabc4370b35ba28052a5f187. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 21:01:55.291127 30773 tablet_bootstrap.cc:492] T 2c138804eabc4370b35ba28052a5f187 P 47a2c6b9946243d4b031a7d3c187f61b: Bootstrap starting.
I20251215 21:01:55.292872 30774 tablet_bootstrap.cc:492] T 2c138804eabc4370b35ba28052a5f187 P cd74be91011a4300b5edb2f531b0c3c5: Bootstrap starting.
I20251215 21:01:55.293275 30773 tablet_bootstrap.cc:654] T 2c138804eabc4370b35ba28052a5f187 P 47a2c6b9946243d4b031a7d3c187f61b: Neither blocks nor log segments found. Creating new log.
I20251215 21:01:55.293634 30775 tablet_bootstrap.cc:492] T 2c138804eabc4370b35ba28052a5f187 P 10081c0117c2422780822c7590f0b49f: Bootstrap starting.
I20251215 21:01:55.294577 30773 log.cc:826] T 2c138804eabc4370b35ba28052a5f187 P 47a2c6b9946243d4b031a7d3c187f61b: Log is configured to *not* fsync() on all Append() calls
I20251215 21:01:55.294782 30774 tablet_bootstrap.cc:654] T 2c138804eabc4370b35ba28052a5f187 P cd74be91011a4300b5edb2f531b0c3c5: Neither blocks nor log segments found. Creating new log.
I20251215 21:01:55.295514 30774 log.cc:826] T 2c138804eabc4370b35ba28052a5f187 P cd74be91011a4300b5edb2f531b0c3c5: Log is configured to *not* fsync() on all Append() calls
I20251215 21:01:55.295601 30775 tablet_bootstrap.cc:654] T 2c138804eabc4370b35ba28052a5f187 P 10081c0117c2422780822c7590f0b49f: Neither blocks nor log segments found. Creating new log.
I20251215 21:01:55.296634 30773 tablet_bootstrap.cc:492] T 2c138804eabc4370b35ba28052a5f187 P 47a2c6b9946243d4b031a7d3c187f61b: No bootstrap required, opened a new log
I20251215 21:01:55.296756 30775 log.cc:826] T 2c138804eabc4370b35ba28052a5f187 P 10081c0117c2422780822c7590f0b49f: Log is configured to *not* fsync() on all Append() calls
I20251215 21:01:55.296849 30773 ts_tablet_manager.cc:1403] T 2c138804eabc4370b35ba28052a5f187 P 47a2c6b9946243d4b031a7d3c187f61b: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.001s
I20251215 21:01:55.297062 30774 tablet_bootstrap.cc:492] T 2c138804eabc4370b35ba28052a5f187 P cd74be91011a4300b5edb2f531b0c3c5: No bootstrap required, opened a new log
I20251215 21:01:55.297221 30774 ts_tablet_manager.cc:1403] T 2c138804eabc4370b35ba28052a5f187 P cd74be91011a4300b5edb2f531b0c3c5: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20251215 21:01:55.299278 30773 raft_consensus.cc:359] T 2c138804eabc4370b35ba28052a5f187 P 47a2c6b9946243d4b031a7d3c187f61b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47a2c6b9946243d4b031a7d3c187f61b" member_type: VOTER last_known_addr { host: "127.28.113.3" port: 35943 } } peers { permanent_uuid: "10081c0117c2422780822c7590f0b49f" member_type: VOTER last_known_addr { host: "127.28.113.2" port: 36671 } } peers { permanent_uuid: "cd74be91011a4300b5edb2f531b0c3c5" member_type: VOTER last_known_addr { host: "127.28.113.1" port: 40521 } }
I20251215 21:01:55.299455 30773 raft_consensus.cc:385] T 2c138804eabc4370b35ba28052a5f187 P 47a2c6b9946243d4b031a7d3c187f61b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251215 21:01:55.299489 30773 raft_consensus.cc:740] T 2c138804eabc4370b35ba28052a5f187 P 47a2c6b9946243d4b031a7d3c187f61b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47a2c6b9946243d4b031a7d3c187f61b, State: Initialized, Role: FOLLOWER
I20251215 21:01:55.299788 30774 raft_consensus.cc:359] T 2c138804eabc4370b35ba28052a5f187 P cd74be91011a4300b5edb2f531b0c3c5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47a2c6b9946243d4b031a7d3c187f61b" member_type: VOTER last_known_addr { host: "127.28.113.3" port: 35943 } } peers { permanent_uuid: "10081c0117c2422780822c7590f0b49f" member_type: VOTER last_known_addr { host: "127.28.113.2" port: 36671 } } peers { permanent_uuid: "cd74be91011a4300b5edb2f531b0c3c5" member_type: VOTER last_known_addr { host: "127.28.113.1" port: 40521 } }
I20251215 21:01:55.299886 30773 consensus_queue.cc:260] T 2c138804eabc4370b35ba28052a5f187 P 47a2c6b9946243d4b031a7d3c187f61b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47a2c6b9946243d4b031a7d3c187f61b" member_type: VOTER last_known_addr { host: "127.28.113.3" port: 35943 } } peers { permanent_uuid: "10081c0117c2422780822c7590f0b49f" member_type: VOTER last_known_addr { host: "127.28.113.2" port: 36671 } } peers { permanent_uuid: "cd74be91011a4300b5edb2f531b0c3c5" member_type: VOTER last_known_addr { host: "127.28.113.1" port: 40521 } }
I20251215 21:01:55.299952 30774 raft_consensus.cc:385] T 2c138804eabc4370b35ba28052a5f187 P cd74be91011a4300b5edb2f531b0c3c5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251215 21:01:55.300026 30774 raft_consensus.cc:740] T 2c138804eabc4370b35ba28052a5f187 P cd74be91011a4300b5edb2f531b0c3c5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd74be91011a4300b5edb2f531b0c3c5, State: Initialized, Role: FOLLOWER
I20251215 21:01:55.300374 30774 consensus_queue.cc:260] T 2c138804eabc4370b35ba28052a5f187 P cd74be91011a4300b5edb2f531b0c3c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47a2c6b9946243d4b031a7d3c187f61b" member_type: VOTER last_known_addr { host: "127.28.113.3" port: 35943 } } peers { permanent_uuid: "10081c0117c2422780822c7590f0b49f" member_type: VOTER last_known_addr { host: "127.28.113.2" port: 36671 } } peers { permanent_uuid: "cd74be91011a4300b5edb2f531b0c3c5" member_type: VOTER last_known_addr { host: "127.28.113.1" port: 40521 } }
I20251215 21:01:55.300613 30750 heartbeater.cc:499] Master 127.28.113.62:40865 was elected leader, sending a full tablet report...
I20251215 21:01:55.300685 30773 ts_tablet_manager.cc:1434] T 2c138804eabc4370b35ba28052a5f187 P 47a2c6b9946243d4b031a7d3c187f61b: Time spent starting tablet: real 0.004s user 0.002s sys 0.002s
I20251215 21:01:55.301023 30466 heartbeater.cc:499] Master 127.28.113.62:40865 was elected leader, sending a full tablet report...
I20251215 21:01:55.301926 30774 ts_tablet_manager.cc:1434] T 2c138804eabc4370b35ba28052a5f187 P cd74be91011a4300b5edb2f531b0c3c5: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20251215 21:01:55.302287 30775 tablet_bootstrap.cc:492] T 2c138804eabc4370b35ba28052a5f187 P 10081c0117c2422780822c7590f0b49f: No bootstrap required, opened a new log
I20251215 21:01:55.302498 30775 ts_tablet_manager.cc:1403] T 2c138804eabc4370b35ba28052a5f187 P 10081c0117c2422780822c7590f0b49f: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.000s
I20251215 21:01:55.305598 30775 raft_consensus.cc:359] T 2c138804eabc4370b35ba28052a5f187 P 10081c0117c2422780822c7590f0b49f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47a2c6b9946243d4b031a7d3c187f61b" member_type: VOTER last_known_addr { host: "127.28.113.3" port: 35943 } } peers { permanent_uuid: "10081c0117c2422780822c7590f0b49f" member_type: VOTER last_known_addr { host: "127.28.113.2" port: 36671 } } peers { permanent_uuid: "cd74be91011a4300b5edb2f531b0c3c5" member_type: VOTER last_known_addr { host: "127.28.113.1" port: 40521 } }
I20251215 21:01:55.305836 30775 raft_consensus.cc:385] T 2c138804eabc4370b35ba28052a5f187 P 10081c0117c2422780822c7590f0b49f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251215 21:01:55.305908 30775 raft_consensus.cc:740] T 2c138804eabc4370b35ba28052a5f187 P 10081c0117c2422780822c7590f0b49f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10081c0117c2422780822c7590f0b49f, State: Initialized, Role: FOLLOWER
I20251215 21:01:55.306280 30775 consensus_queue.cc:260] T 2c138804eabc4370b35ba28052a5f187 P 10081c0117c2422780822c7590f0b49f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47a2c6b9946243d4b031a7d3c187f61b" member_type: VOTER last_known_addr { host: "127.28.113.3" port: 35943 } } peers { permanent_uuid: "10081c0117c2422780822c7590f0b49f" member_type: VOTER last_known_addr { host: "127.28.113.2" port: 36671 } } peers { permanent_uuid: "cd74be91011a4300b5edb2f531b0c3c5" member_type: VOTER last_known_addr { host: "127.28.113.1" port: 40521 } }
I20251215 21:01:55.306836 30608 heartbeater.cc:499] Master 127.28.113.62:40865 was elected leader, sending a full tablet report...
I20251215 21:01:55.306960 30775 ts_tablet_manager.cc:1434] T 2c138804eabc4370b35ba28052a5f187 P 10081c0117c2422780822c7590f0b49f: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
W20251215 21:01:55.479239 30751 tablet.cc:2378] T 2c138804eabc4370b35ba28052a5f187 P 47a2c6b9946243d4b031a7d3c187f61b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20251215 21:01:55.502614 30609 tablet.cc:2378] T 2c138804eabc4370b35ba28052a5f187 P 10081c0117c2422780822c7590f0b49f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20251215 21:01:55.541633 30467 tablet.cc:2378] T 2c138804eabc4370b35ba28052a5f187 P cd74be91011a4300b5edb2f531b0c3c5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20251215 21:01:55.542317 30781 raft_consensus.cc:493] T 2c138804eabc4370b35ba28052a5f187 P 10081c0117c2422780822c7590f0b49f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 21:01:55.542486 30781 raft_consensus.cc:515] T 2c138804eabc4370b35ba28052a5f187 P 10081c0117c2422780822c7590f0b49f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47a2c6b9946243d4b031a7d3c187f61b" member_type: VOTER last_known_addr { host: "127.28.113.3" port: 35943 } } peers { permanent_uuid: "10081c0117c2422780822c7590f0b49f" member_type: VOTER last_known_addr { host: "127.28.113.2" port: 36671 } } peers { permanent_uuid: "cd74be91011a4300b5edb2f531b0c3c5" member_type: VOTER last_known_addr { host: "127.28.113.1" port: 40521 } }
I20251215 21:01:55.543540 30781 leader_election.cc:290] T 2c138804eabc4370b35ba28052a5f187 P 10081c0117c2422780822c7590f0b49f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 47a2c6b9946243d4b031a7d3c187f61b (127.28.113.3:35943), cd74be91011a4300b5edb2f531b0c3c5 (127.28.113.1:40521)
I20251215 21:01:55.547232 30783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:01:55.543810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.113.3:35943 (local address 127.28.113.2:46577)
1215 21:01:55.544094 (+ 284us) negotiation.cc:107] Waiting for socket to connect
1215 21:01:55.544109 (+ 15us) client_negotiation.cc:174] Beginning negotiation
1215 21:01:55.544234 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1215 21:01:55.544492 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1215 21:01:55.544495 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1215 21:01:55.544520 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1215 21:01:55.544798 (+ 278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:01:55.544807 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:01:55.545847 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1215 21:01:55.545850 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1215 21:01:55.546797 (+ 947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:01:55.546807 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:01:55.546924 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:01:55.546946 (+ 22us) client_negotiation.cc:769] Sending connection context
1215 21:01:55.546995 (+ 49us) client_negotiation.cc:240] Negotiation successful
1215 21:01:55.547058 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":208,"spinlock_wait_cycles":3712,"thread_start_us":109,"threads_started":1}
I20251215 21:01:55.547235 30782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:01:55.543810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.113.1:40521 (local address 127.28.113.2:41675)
1215 21:01:55.544158 (+ 348us) negotiation.cc:107] Waiting for socket to connect
1215 21:01:55.544169 (+ 11us) client_negotiation.cc:174] Beginning negotiation
1215 21:01:55.544240 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1215 21:01:55.544492 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1215 21:01:55.544495 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1215 21:01:55.544526 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1215 21:01:55.544800 (+ 274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:01:55.544806 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:01:55.545844 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1215 21:01:55.545847 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1215 21:01:55.546804 (+ 957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:01:55.546810 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:01:55.546924 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:01:55.546946 (+ 22us) client_negotiation.cc:769] Sending connection context
1215 21:01:55.546995 (+ 49us) client_negotiation.cc:240] Negotiation successful
1215 21:01:55.547058 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":294,"mutex_wait_us":11,"spinlock_wait_cycles":12544,"thread_start_us":85,"threads_started":1}
I20251215 21:01:55.547691 30770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:01:55.543892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.113.2:41675 (local address 127.28.113.1:40521)
1215 21:01:55.544146 (+ 254us) server_negotiation.cc:207] Beginning negotiation
1215 21:01:55.544149 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1215 21:01:55.544229 (+ 80us) server_negotiation.cc:408] Connection header received
1215 21:01:55.544335 (+ 106us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1215 21:01:55.544339 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1215 21:01:55.544391 (+ 52us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1215 21:01:55.544474 (+ 83us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1215 21:01:55.544967 (+ 493us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:01:55.545731 (+ 764us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1215 21:01:55.546945 (+ 1214us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:01:55.547430 (+ 485us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:01:55.547469 (+ 39us) server_negotiation.cc:1036] Waiting for connection context
1215 21:01:55.547519 (+ 50us) server_negotiation.cc:300] Negotiation successful
1215 21:01:55.547578 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":164}
I20251215 21:01:55.547691 30768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:01:55.543904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.113.2:46577 (local address 127.28.113.3:35943)
1215 21:01:55.544075 (+ 171us) server_negotiation.cc:207] Beginning negotiation
1215 21:01:55.544079 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1215 21:01:55.544191 (+ 112us) server_negotiation.cc:408] Connection header received
1215 21:01:55.544335 (+ 144us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1215 21:01:55.544339 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1215 21:01:55.544391 (+ 52us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1215 21:01:55.544474 (+ 83us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1215 21:01:55.544967 (+ 493us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:01:55.545731 (+ 764us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1215 21:01:55.546939 (+ 1208us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:01:55.547440 (+ 501us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:01:55.547469 (+ 29us) server_negotiation.cc:1036] Waiting for connection context
1215 21:01:55.547536 (+ 67us) server_negotiation.cc:300] Negotiation successful
1215 21:01:55.547588 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20251215 21:01:55.548264 30704 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2c138804eabc4370b35ba28052a5f187" candidate_uuid: "10081c0117c2422780822c7590f0b49f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "47a2c6b9946243d4b031a7d3c187f61b" is_pre_election: true
I20251215 21:01:55.548264 30420 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2c138804eabc4370b35ba28052a5f187" candidate_uuid: "10081c0117c2422780822c7590f0b49f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd74be91011a4300b5edb2f531b0c3c5" is_pre_election: true
I20251215 21:01:55.548501 30704 raft_consensus.cc:2468] T 2c138804eabc4370b35ba28052a5f187 P 47a2c6b9946243d4b031a7d3c187f61b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10081c0117c2422780822c7590f0b49f in term 0.
I20251215 21:01:55.548499 30420 raft_consensus.cc:2468] T 2c138804eabc4370b35ba28052a5f187 P cd74be91011a4300b5edb2f531b0c3c5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10081c0117c2422780822c7590f0b49f in term 0.
I20251215 21:01:55.548919 30495 leader_election.cc:304] T 2c138804eabc4370b35ba28052a5f187 P 10081c0117c2422780822c7590f0b49f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10081c0117c2422780822c7590f0b49f, cd74be91011a4300b5edb2f531b0c3c5; no voters:
I20251215 21:01:55.549151 30781 raft_consensus.cc:2804] T 2c138804eabc4370b35ba28052a5f187 P 10081c0117c2422780822c7590f0b49f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251215 21:01:55.549217 30781 raft_consensus.cc:493] T 2c138804eabc4370b35ba28052a5f187 P 10081c0117c2422780822c7590f0b49f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 21:01:55.549247 30781 raft_consensus.cc:3060] T 2c138804eabc4370b35ba28052a5f187 P 10081c0117c2422780822c7590f0b49f [term 0 FOLLOWER]: Advancing to term 1
I20251215 21:01:55.550129 30781 raft_consensus.cc:515] T 2c138804eabc4370b35ba28052a5f187 P 10081c0117c2422780822c7590f0b49f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47a2c6b9946243d4b031a7d3c187f61b" member_type: VOTER last_known_addr { host: "127.28.113.3" port: 35943 } } peers { permanent_uuid: "10081c0117c2422780822c7590f0b49f" member_type: VOTER last_known_addr { host: "127.28.113.2" port: 36671 } } peers { permanent_uuid: "cd74be91011a4300b5edb2f531b0c3c5" member_type: VOTER last_known_addr { host: "127.28.113.1" port: 40521 } }
I20251215 21:01:55.550457 30781 leader_election.cc:290] T 2c138804eabc4370b35ba28052a5f187 P 10081c0117c2422780822c7590f0b49f [CANDIDATE]: Term 1 election: Requested vote from peers 47a2c6b9946243d4b031a7d3c187f61b (127.28.113.3:35943), cd74be91011a4300b5edb2f531b0c3c5 (127.28.113.1:40521)
I20251215 21:01:55.550809 30704 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2c138804eabc4370b35ba28052a5f187" candidate_uuid: "10081c0117c2422780822c7590f0b49f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "47a2c6b9946243d4b031a7d3c187f61b"
I20251215 21:01:55.550810 30420 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2c138804eabc4370b35ba28052a5f187" candidate_uuid: "10081c0117c2422780822c7590f0b49f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd74be91011a4300b5edb2f531b0c3c5"
I20251215 21:01:55.550926 30704 raft_consensus.cc:3060] T 2c138804eabc4370b35ba28052a5f187 P 47a2c6b9946243d4b031a7d3c187f61b [term 0 FOLLOWER]: Advancing to term 1
I20251215 21:01:55.550926 30420 raft_consensus.cc:3060] T 2c138804eabc4370b35ba28052a5f187 P cd74be91011a4300b5edb2f531b0c3c5 [term 0 FOLLOWER]: Advancing to term 1
I20251215 21:01:55.552047 30704 raft_consensus.cc:2468] T 2c138804eabc4370b35ba28052a5f187 P 47a2c6b9946243d4b031a7d3c187f61b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10081c0117c2422780822c7590f0b49f in term 1.
I20251215 21:01:55.552047 30420 raft_consensus.cc:2468] T 2c138804eabc4370b35ba28052a5f187 P cd74be91011a4300b5edb2f531b0c3c5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10081c0117c2422780822c7590f0b49f in term 1.
I20251215 21:01:55.552325 30495 leader_election.cc:304] T 2c138804eabc4370b35ba28052a5f187 P 10081c0117c2422780822c7590f0b49f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10081c0117c2422780822c7590f0b49f, cd74be91011a4300b5edb2f531b0c3c5; no voters:
I20251215 21:01:55.552479 30781 raft_consensus.cc:2804] T 2c138804eabc4370b35ba28052a5f187 P 10081c0117c2422780822c7590f0b49f [term 1 FOLLOWER]: Leader election won for term 1
I20251215 21:01:55.552637 30781 raft_consensus.cc:697] T 2c138804eabc4370b35ba28052a5f187 P 10081c0117c2422780822c7590f0b49f [term 1 LEADER]: Becoming Leader. State: Replica: 10081c0117c2422780822c7590f0b49f, State: Running, Role: LEADER
I20251215 21:01:55.552951 30781 consensus_queue.cc:237] T 2c138804eabc4370b35ba28052a5f187 P 10081c0117c2422780822c7590f0b49f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47a2c6b9946243d4b031a7d3c187f61b" member_type: VOTER last_known_addr { host: "127.28.113.3" port: 35943 } } peers { permanent_uuid: "10081c0117c2422780822c7590f0b49f" member_type: VOTER last_known_addr { host: "127.28.113.2" port: 36671 } } peers { permanent_uuid: "cd74be91011a4300b5edb2f531b0c3c5" member_type: VOTER last_known_addr { host: "127.28.113.1" port: 40521 } }
I20251215 21:01:55.555946 30259 catalog_manager.cc:5654] T 2c138804eabc4370b35ba28052a5f187 P 10081c0117c2422780822c7590f0b49f reported cstate change: term changed from 0 to 1, leader changed from <none> to 10081c0117c2422780822c7590f0b49f (127.28.113.2). New cstate: current_term: 1 leader_uuid: "10081c0117c2422780822c7590f0b49f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47a2c6b9946243d4b031a7d3c187f61b" member_type: VOTER last_known_addr { host: "127.28.113.3" port: 35943 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "10081c0117c2422780822c7590f0b49f" member_type: VOTER last_known_addr { host: "127.28.113.2" port: 36671 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cd74be91011a4300b5edb2f531b0c3c5" member_type: VOTER last_known_addr { host: "127.28.113.1" port: 40521 } health_report { overall_health: UNKNOWN } } }
I20251215 21:01:55.578605 30772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:01:55.575992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44944 (local address 127.28.113.2:36671)
1215 21:01:55.576112 (+ 120us) server_negotiation.cc:207] Beginning negotiation
1215 21:01:55.576115 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1215 21:01:55.576129 (+ 14us) server_negotiation.cc:408] Connection header received
1215 21:01:55.576259 (+ 130us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1215 21:01:55.576262 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1215 21:01:55.576316 (+ 54us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1215 21:01:55.576392 (+ 76us) server_negotiation.cc:227] Negotiated authn=TOKEN
1215 21:01:55.576679 (+ 287us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:01:55.577164 (+ 485us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1215 21:01:55.577721 (+ 557us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:01:55.577865 (+ 144us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:01:55.577911 (+ 46us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1215 21:01:55.578333 (+ 422us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1215 21:01:55.578416 (+ 83us) server_negotiation.cc:1036] Waiting for connection context
1215 21:01:55.578459 (+ 43us) server_negotiation.cc:300] Negotiation successful
1215 21:01:55.578502 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20251215 21:01:55.583820 30704 raft_consensus.cc:1275] T 2c138804eabc4370b35ba28052a5f187 P 47a2c6b9946243d4b031a7d3c187f61b [term 1 FOLLOWER]: Refusing update from remote peer 10081c0117c2422780822c7590f0b49f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251215 21:01:55.583820 30420 raft_consensus.cc:1275] T 2c138804eabc4370b35ba28052a5f187 P cd74be91011a4300b5edb2f531b0c3c5 [term 1 FOLLOWER]: Refusing update from remote peer 10081c0117c2422780822c7590f0b49f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251215 21:01:55.584452 30781 consensus_queue.cc:1048] T 2c138804eabc4370b35ba28052a5f187 P 10081c0117c2422780822c7590f0b49f [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd74be91011a4300b5edb2f531b0c3c5" member_type: VOTER last_known_addr { host: "127.28.113.1" port: 40521 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251215 21:01:55.584628 30784 consensus_queue.cc:1048] T 2c138804eabc4370b35ba28052a5f187 P 10081c0117c2422780822c7590f0b49f [LEADER]: Connected to new peer: Peer: permanent_uuid: "47a2c6b9946243d4b031a7d3c187f61b" member_type: VOTER last_known_addr { host: "127.28.113.3" port: 35943 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251215 21:01:55.592231 30789 mvcc.cc:204] Tried to move back new op lower bound from 7232849983824216064 to 7232849983707357184. Current Snapshot: MvccSnapshot[applied={T|T < 7232849983824216064}]
I20251215 21:01:55.592742 30788 mvcc.cc:204] Tried to move back new op lower bound from 7232849983824216064 to 7232849983707357184. Current Snapshot: MvccSnapshot[applied={T|T < 7232849983824216064}]
I20251215 21:01:55.601796 30259 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40128:
table { table_name: "test-table" } modify_external_catalogs: true
I20251215 21:01:55.601963 30259 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40128:
table { table_name: "test-table" } modify_external_catalogs: true
I20251215 21:01:55.604259 30259 catalog_manager.cc:5941] T 00000000000000000000000000000000 P 700c44b9a4694ad6993e6d84e5c0475c: Sending DeleteTablet for 3 replicas of tablet 2c138804eabc4370b35ba28052a5f187
I20251215 21:01:55.605131 30400 tablet_service.cc:1552] Processing DeleteTablet for tablet 2c138804eabc4370b35ba28052a5f187 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-15 21:01:55 UTC) from {username='kudu', principal='kudu/127.28.113.62@KRBTEST.COM'} at 127.0.0.1:37470
I20251215 21:01:55.605131 30683 tablet_service.cc:1552] Processing DeleteTablet for tablet 2c138804eabc4370b35ba28052a5f187 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-15 21:01:55 UTC) from {username='kudu', principal='kudu/127.28.113.62@KRBTEST.COM'} at 127.0.0.1:58698
I20251215 21:01:55.605535 30259 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40128:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251215 21:01:55.605593 30542 tablet_service.cc:1552] Processing DeleteTablet for tablet 2c138804eabc4370b35ba28052a5f187 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-15 21:01:55 UTC) from {username='kudu', principal='kudu/127.28.113.62@KRBTEST.COM'} at 127.0.0.1:44928
W20251215 21:01:55.605913 30259 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251215 21:01:55.605981 30798 tablet_replica.cc:333] T 2c138804eabc4370b35ba28052a5f187 P 10081c0117c2422780822c7590f0b49f: stopping tablet replica
I20251215 21:01:55.606232 30796 tablet_replica.cc:333] T 2c138804eabc4370b35ba28052a5f187 P 47a2c6b9946243d4b031a7d3c187f61b: stopping tablet replica
I20251215 21:01:55.606215 30798 raft_consensus.cc:2243] T 2c138804eabc4370b35ba28052a5f187 P 10081c0117c2422780822c7590f0b49f [term 1 LEADER]: Raft consensus shutting down.
I20251215 21:01:55.606495 30796 raft_consensus.cc:2243] T 2c138804eabc4370b35ba28052a5f187 P 47a2c6b9946243d4b031a7d3c187f61b [term 1 FOLLOWER]: Raft consensus shutting down.
I20251215 21:01:55.606600 30798 raft_consensus.cc:2272] T 2c138804eabc4370b35ba28052a5f187 P 10081c0117c2422780822c7590f0b49f [term 1 FOLLOWER]: Raft consensus is shut down!
I20251215 21:01:55.606885 30796 raft_consensus.cc:2272] T 2c138804eabc4370b35ba28052a5f187 P 47a2c6b9946243d4b031a7d3c187f61b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251215 21:01:55.607331 30797 tablet_replica.cc:333] T 2c138804eabc4370b35ba28052a5f187 P cd74be91011a4300b5edb2f531b0c3c5: stopping tablet replica
I20251215 21:01:55.607556 30797 raft_consensus.cc:2243] T 2c138804eabc4370b35ba28052a5f187 P cd74be91011a4300b5edb2f531b0c3c5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251215 21:01:55.607831 30797 raft_consensus.cc:2272] T 2c138804eabc4370b35ba28052a5f187 P cd74be91011a4300b5edb2f531b0c3c5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251215 21:01:55.609088 30797 ts_tablet_manager.cc:1916] T 2c138804eabc4370b35ba28052a5f187 P cd74be91011a4300b5edb2f531b0c3c5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251215 21:01:55.609088 30796 ts_tablet_manager.cc:1916] T 2c138804eabc4370b35ba28052a5f187 P 47a2c6b9946243d4b031a7d3c187f61b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251215 21:01:55.610709 30542 tablet_service.cc:1505] Processing CreateTablet for tablet 063a48df2fef4446be204530a8df2ae5 (DEFAULT_TABLE table=test-table [id=ec582cc4325f465d8a5cda6c6f83e584]), partition=RANGE (key) PARTITION UNBOUNDED
I20251215 21:01:55.610908 30683 tablet_service.cc:1505] Processing CreateTablet for tablet 063a48df2fef4446be204530a8df2ae5 (DEFAULT_TABLE table=test-table [id=ec582cc4325f465d8a5cda6c6f83e584]), partition=RANGE (key) PARTITION UNBOUNDED
I20251215 21:01:55.610968 30542 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 063a48df2fef4446be204530a8df2ae5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 21:01:55.611128 30683 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 063a48df2fef4446be204530a8df2ae5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 21:01:55.611459 30797 ts_tablet_manager.cc:1929] T 2c138804eabc4370b35ba28052a5f187 P cd74be91011a4300b5edb2f531b0c3c5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251215 21:01:55.611539 30797 log.cc:1199] T 2c138804eabc4370b35ba28052a5f187 P cd74be91011a4300b5edb2f531b0c3c5: Deleting WAL directory at /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765832506145157-29124-0/minicluster-data/ts-0/wal/wals/2c138804eabc4370b35ba28052a5f187
I20251215 21:01:55.611785 30797 ts_tablet_manager.cc:1950] T 2c138804eabc4370b35ba28052a5f187 P cd74be91011a4300b5edb2f531b0c3c5: Deleting consensus metadata
I20251215 21:01:55.613173 30775 tablet_bootstrap.cc:492] T 063a48df2fef4446be204530a8df2ae5 P 10081c0117c2422780822c7590f0b49f: Bootstrap starting.
I20251215 21:01:55.613198 30773 tablet_bootstrap.cc:492] T 063a48df2fef4446be204530a8df2ae5 P 47a2c6b9946243d4b031a7d3c187f61b: Bootstrap starting.
I20251215 21:01:55.614120 30773 tablet_bootstrap.cc:654] T 063a48df2fef4446be204530a8df2ae5 P 47a2c6b9946243d4b031a7d3c187f61b: Neither blocks nor log segments found. Creating new log.
I20251215 21:01:55.614120 30775 tablet_bootstrap.cc:654] T 063a48df2fef4446be204530a8df2ae5 P 10081c0117c2422780822c7590f0b49f: Neither blocks nor log segments found. Creating new log.
I20251215 21:01:55.615320 30775 tablet_bootstrap.cc:492] T 063a48df2fef4446be204530a8df2ae5 P 10081c0117c2422780822c7590f0b49f: No bootstrap required, opened a new log
I20251215 21:01:55.615401 30775 ts_tablet_manager.cc:1403] T 063a48df2fef4446be204530a8df2ae5 P 10081c0117c2422780822c7590f0b49f: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20251215 21:01:55.615365 30247 catalog_manager.cc:4985] TS cd74be91011a4300b5edb2f531b0c3c5 (127.28.113.1:40521): tablet 2c138804eabc4370b35ba28052a5f187 (table test-table [id=fbdc85fe372540dbb6aca1c724eddbdd]) successfully deleted
I20251215 21:01:55.615561 30400 tablet_service.cc:1505] Processing CreateTablet for tablet 063a48df2fef4446be204530a8df2ae5 (DEFAULT_TABLE table=test-table [id=ec582cc4325f465d8a5cda6c6f83e584]), partition=RANGE (key) PARTITION UNBOUNDED
I20251215 21:01:55.615782 30400 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 063a48df2fef4446be204530a8df2ae5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 21:01:55.615819 30775 raft_consensus.cc:359] T 063a48df2fef4446be204530a8df2ae5 P 10081c0117c2422780822c7590f0b49f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd74be91011a4300b5edb2f531b0c3c5" member_type: VOTER last_known_addr { host: "127.28.113.1" port: 40521 } } peers { permanent_uuid: "10081c0117c2422780822c7590f0b49f" member_type: VOTER last_known_addr { host: "127.28.113.2" port: 36671 } } peers { permanent_uuid: "47a2c6b9946243d4b031a7d3c187f61b" member_type: VOTER last_known_addr { host: "127.28.113.3" port: 35943 } }
I20251215 21:01:55.615924 30775 raft_consensus.cc:385] T 063a48df2fef4446be204530a8df2ae5 P 10081c0117c2422780822c7590f0b49f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251215 21:01:55.615978 30775 raft_consensus.cc:740] T 063a48df2fef4446be204530a8df2ae5 P 10081c0117c2422780822c7590f0b49f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10081c0117c2422780822c7590f0b49f, State: Initialized, Role: FOLLOWER
I20251215 21:01:55.616096 30775 consensus_queue.cc:260] T 063a48df2fef4446be204530a8df2ae5 P 10081c0117c2422780822c7590f0b49f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd74be91011a4300b5edb2f531b0c3c5" member_type: VOTER last_known_addr { host: "127.28.113.1" port: 40521 } } peers { permanent_uuid: "10081c0117c2422780822c7590f0b49f" member_type: VOTER last_known_addr { host: "127.28.113.2" port: 36671 } } peers { permanent_uuid: "47a2c6b9946243d4b031a7d3c187f61b" member_type: VOTER last_known_addr { host: "127.28.113.3" port: 35943 } }
I20251215 21:01:55.616333 30775 ts_tablet_manager.cc:1434] T 063a48df2fef4446be204530a8df2ae5 P 10081c0117c2422780822c7590f0b49f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20251215 21:01:55.616496 30773 tablet_bootstrap.cc:492] T 063a48df2fef4446be204530a8df2ae5 P 47a2c6b9946243d4b031a7d3c187f61b: No bootstrap required, opened a new log
I20251215 21:01:55.616566 30773 ts_tablet_manager.cc:1403] T 063a48df2fef4446be204530a8df2ae5 P 47a2c6b9946243d4b031a7d3c187f61b: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20251215 21:01:55.617014 30773 raft_consensus.cc:359] T 063a48df2fef4446be204530a8df2ae5 P 47a2c6b9946243d4b031a7d3c187f61b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd74be91011a4300b5edb2f531b0c3c5" member_type: VOTER last_known_addr { host: "127.28.113.1" port: 40521 } } peers { permanent_uuid: "10081c0117c2422780822c7590f0b49f" member_type: VOTER last_known_addr { host: "127.28.113.2" port: 36671 } } peers { permanent_uuid: "47a2c6b9946243d4b031a7d3c187f61b" member_type: VOTER last_known_addr { host: "127.28.113.3" port: 35943 } }
I20251215 21:01:55.617102 30773 raft_consensus.cc:385] T 063a48df2fef4446be204530a8df2ae5 P 47a2c6b9946243d4b031a7d3c187f61b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251215 21:01:55.617129 30773 raft_consensus.cc:740] T 063a48df2fef4446be204530a8df2ae5 P 47a2c6b9946243d4b031a7d3c187f61b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47a2c6b9946243d4b031a7d3c187f61b, State: Initialized, Role: FOLLOWER
I20251215 21:01:55.617244 30773 consensus_queue.cc:260] T 063a48df2fef4446be204530a8df2ae5 P 47a2c6b9946243d4b031a7d3c187f61b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd74be91011a4300b5edb2f531b0c3c5" member_type: VOTER last_known_addr { host: "127.28.113.1" port: 40521 } } peers { permanent_uuid: "10081c0117c2422780822c7590f0b49f" member_type: VOTER last_known_addr { host: "127.28.113.2" port: 36671 } } peers { permanent_uuid: "47a2c6b9946243d4b031a7d3c187f61b" member_type: VOTER last_known_addr { host: "127.28.113.3" port: 35943 } }
I20251215 21:01:55.617462 30773 ts_tablet_manager.cc:1434] T 063a48df2fef4446be204530a8df2ae5 P 47a2c6b9946243d4b031a7d3c187f61b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20251215 21:01:55.617785 30774 tablet_bootstrap.cc:492] T 063a48df2fef4446be204530a8df2ae5 P cd74be91011a4300b5edb2f531b0c3c5: Bootstrap starting.
I20251215 21:01:55.618750 30774 tablet_bootstrap.cc:654] T 063a48df2fef4446be204530a8df2ae5 P cd74be91011a4300b5edb2f531b0c3c5: Neither blocks nor log segments found. Creating new log.
I20251215 21:01:55.620656 30774 tablet_bootstrap.cc:492] T 063a48df2fef4446be204530a8df2ae5 P cd74be91011a4300b5edb2f531b0c3c5: No bootstrap required, opened a new log
I20251215 21:01:55.620748 30774 ts_tablet_manager.cc:1403] T 063a48df2fef4446be204530a8df2ae5 P cd74be91011a4300b5edb2f531b0c3c5: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20251215 21:01:55.620834 30796 ts_tablet_manager.cc:1929] T 2c138804eabc4370b35ba28052a5f187 P 47a2c6b9946243d4b031a7d3c187f61b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251215 21:01:55.620882 30798 ts_tablet_manager.cc:1916] T 2c138804eabc4370b35ba28052a5f187 P 10081c0117c2422780822c7590f0b49f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251215 21:01:55.620942 30796 log.cc:1199] T 2c138804eabc4370b35ba28052a5f187 P 47a2c6b9946243d4b031a7d3c187f61b: Deleting WAL directory at /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765832506145157-29124-0/minicluster-data/ts-2/wal/wals/2c138804eabc4370b35ba28052a5f187
I20251215 21:01:55.621201 30774 raft_consensus.cc:359] T 063a48df2fef4446be204530a8df2ae5 P cd74be91011a4300b5edb2f531b0c3c5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd74be91011a4300b5edb2f531b0c3c5" member_type: VOTER last_known_addr { host: "127.28.113.1" port: 40521 } } peers { permanent_uuid: "10081c0117c2422780822c7590f0b49f" member_type: VOTER last_known_addr { host: "127.28.113.2" port: 36671 } } peers { permanent_uuid: "47a2c6b9946243d4b031a7d3c187f61b" member_type: VOTER last_known_addr { host: "127.28.113.3" port: 35943 } }
I20251215 21:01:55.621312 30774 raft_consensus.cc:385] T 063a48df2fef4446be204530a8df2ae5 P cd74be91011a4300b5edb2f531b0c3c5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251215 21:01:55.621354 30774 raft_consensus.cc:740] T 063a48df2fef4446be204530a8df2ae5 P cd74be91011a4300b5edb2f531b0c3c5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd74be91011a4300b5edb2f531b0c3c5, State: Initialized, Role: FOLLOWER
I20251215 21:01:55.621470 30774 consensus_queue.cc:260] T 063a48df2fef4446be204530a8df2ae5 P cd74be91011a4300b5edb2f531b0c3c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd74be91011a4300b5edb2f531b0c3c5" member_type: VOTER last_known_addr { host: "127.28.113.1" port: 40521 } } peers { permanent_uuid: "10081c0117c2422780822c7590f0b49f" member_type: VOTER last_known_addr { host: "127.28.113.2" port: 36671 } } peers { permanent_uuid: "47a2c6b9946243d4b031a7d3c187f61b" member_type: VOTER last_known_addr { host: "127.28.113.3" port: 35943 } }
I20251215 21:01:55.621726 30774 ts_tablet_manager.cc:1434] T 063a48df2fef4446be204530a8df2ae5 P cd74be91011a4300b5edb2f531b0c3c5: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20251215 21:01:55.622078 30796 ts_tablet_manager.cc:1950] T 2c138804eabc4370b35ba28052a5f187 P 47a2c6b9946243d4b031a7d3c187f61b: Deleting consensus metadata
I20251215 21:01:55.623368 30798 ts_tablet_manager.cc:1929] T 2c138804eabc4370b35ba28052a5f187 P 10081c0117c2422780822c7590f0b49f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251215 21:01:55.623452 30798 log.cc:1199] T 2c138804eabc4370b35ba28052a5f187 P 10081c0117c2422780822c7590f0b49f: Deleting WAL directory at /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765832506145157-29124-0/minicluster-data/ts-1/wal/wals/2c138804eabc4370b35ba28052a5f187
I20251215 21:01:55.623720 30798 ts_tablet_manager.cc:1950] T 2c138804eabc4370b35ba28052a5f187 P 10081c0117c2422780822c7590f0b49f: Deleting consensus metadata
I20251215 21:01:55.623764 30246 catalog_manager.cc:4985] TS 47a2c6b9946243d4b031a7d3c187f61b (127.28.113.3:35943): tablet 2c138804eabc4370b35ba28052a5f187 (table test-table [id=fbdc85fe372540dbb6aca1c724eddbdd]) successfully deleted
I20251215 21:01:55.624680 30246 catalog_manager.cc:4985] TS 10081c0117c2422780822c7590f0b49f (127.28.113.2:36671): tablet 2c138804eabc4370b35ba28052a5f187 (table test-table [id=fbdc85fe372540dbb6aca1c724eddbdd]) successfully deleted
I20251215 21:01:55.712167 30784 raft_consensus.cc:493] T 063a48df2fef4446be204530a8df2ae5 P 10081c0117c2422780822c7590f0b49f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 21:01:55.712298 30784 raft_consensus.cc:515] T 063a48df2fef4446be204530a8df2ae5 P 10081c0117c2422780822c7590f0b49f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd74be91011a4300b5edb2f531b0c3c5" member_type: VOTER last_known_addr { host: "127.28.113.1" port: 40521 } } peers { permanent_uuid: "10081c0117c2422780822c7590f0b49f" member_type: VOTER last_known_addr { host: "127.28.113.2" port: 36671 } } peers { permanent_uuid: "47a2c6b9946243d4b031a7d3c187f61b" member_type: VOTER last_known_addr { host: "127.28.113.3" port: 35943 } }
I20251215 21:01:55.712708 30784 leader_election.cc:290] T 063a48df2fef4446be204530a8df2ae5 P 10081c0117c2422780822c7590f0b49f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cd74be91011a4300b5edb2f531b0c3c5 (127.28.113.1:40521), 47a2c6b9946243d4b031a7d3c187f61b (127.28.113.3:35943)
I20251215 21:01:55.713133 30704 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "063a48df2fef4446be204530a8df2ae5" candidate_uuid: "10081c0117c2422780822c7590f0b49f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "47a2c6b9946243d4b031a7d3c187f61b" is_pre_election: true
I20251215 21:01:55.713150 30420 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "063a48df2fef4446be204530a8df2ae5" candidate_uuid: "10081c0117c2422780822c7590f0b49f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd74be91011a4300b5edb2f531b0c3c5" is_pre_election: true
I20251215 21:01:55.713275 30704 raft_consensus.cc:2468] T 063a48df2fef4446be204530a8df2ae5 P 47a2c6b9946243d4b031a7d3c187f61b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10081c0117c2422780822c7590f0b49f in term 0.
I20251215 21:01:55.713307 30420 raft_consensus.cc:2468] T 063a48df2fef4446be204530a8df2ae5 P cd74be91011a4300b5edb2f531b0c3c5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10081c0117c2422780822c7590f0b49f in term 0.
I20251215 21:01:55.713551 30494 leader_election.cc:304] T 063a48df2fef4446be204530a8df2ae5 P 10081c0117c2422780822c7590f0b49f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10081c0117c2422780822c7590f0b49f, 47a2c6b9946243d4b031a7d3c187f61b; no voters:
I20251215 21:01:55.713728 30784 raft_consensus.cc:2804] T 063a48df2fef4446be204530a8df2ae5 P 10081c0117c2422780822c7590f0b49f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251215 21:01:55.713792 30784 raft_consensus.cc:493] T 063a48df2fef4446be204530a8df2ae5 P 10081c0117c2422780822c7590f0b49f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 21:01:55.713830 30784 raft_consensus.cc:3060] T 063a48df2fef4446be204530a8df2ae5 P 10081c0117c2422780822c7590f0b49f [term 0 FOLLOWER]: Advancing to term 1
I20251215 21:01:55.714675 30784 raft_consensus.cc:515] T 063a48df2fef4446be204530a8df2ae5 P 10081c0117c2422780822c7590f0b49f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd74be91011a4300b5edb2f531b0c3c5" member_type: VOTER last_known_addr { host: "127.28.113.1" port: 40521 } } peers { permanent_uuid: "10081c0117c2422780822c7590f0b49f" member_type: VOTER last_known_addr { host: "127.28.113.2" port: 36671 } } peers { permanent_uuid: "47a2c6b9946243d4b031a7d3c187f61b" member_type: VOTER last_known_addr { host: "127.28.113.3" port: 35943 } }
I20251215 21:01:55.715006 30784 leader_election.cc:290] T 063a48df2fef4446be204530a8df2ae5 P 10081c0117c2422780822c7590f0b49f [CANDIDATE]: Term 1 election: Requested vote from peers cd74be91011a4300b5edb2f531b0c3c5 (127.28.113.1:40521), 47a2c6b9946243d4b031a7d3c187f61b (127.28.113.3:35943)
I20251215 21:01:55.715330 30704 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "063a48df2fef4446be204530a8df2ae5" candidate_uuid: "10081c0117c2422780822c7590f0b49f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "47a2c6b9946243d4b031a7d3c187f61b"
I20251215 21:01:55.715343 30420 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "063a48df2fef4446be204530a8df2ae5" candidate_uuid: "10081c0117c2422780822c7590f0b49f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd74be91011a4300b5edb2f531b0c3c5"
I20251215 21:01:55.715430 30704 raft_consensus.cc:3060] T 063a48df2fef4446be204530a8df2ae5 P 47a2c6b9946243d4b031a7d3c187f61b [term 0 FOLLOWER]: Advancing to term 1
I20251215 21:01:55.715456 30420 raft_consensus.cc:3060] T 063a48df2fef4446be204530a8df2ae5 P cd74be91011a4300b5edb2f531b0c3c5 [term 0 FOLLOWER]: Advancing to term 1
I20251215 21:01:55.716264 30704 raft_consensus.cc:2468] T 063a48df2fef4446be204530a8df2ae5 P 47a2c6b9946243d4b031a7d3c187f61b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10081c0117c2422780822c7590f0b49f in term 1.
I20251215 21:01:55.716534 30494 leader_election.cc:304] T 063a48df2fef4446be204530a8df2ae5 P 10081c0117c2422780822c7590f0b49f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10081c0117c2422780822c7590f0b49f, 47a2c6b9946243d4b031a7d3c187f61b; no voters:
I20251215 21:01:55.716566 30420 raft_consensus.cc:2468] T 063a48df2fef4446be204530a8df2ae5 P cd74be91011a4300b5edb2f531b0c3c5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10081c0117c2422780822c7590f0b49f in term 1.
I20251215 21:01:55.716686 30784 raft_consensus.cc:2804] T 063a48df2fef4446be204530a8df2ae5 P 10081c0117c2422780822c7590f0b49f [term 1 FOLLOWER]: Leader election won for term 1
I20251215 21:01:55.716761 30784 raft_consensus.cc:697] T 063a48df2fef4446be204530a8df2ae5 P 10081c0117c2422780822c7590f0b49f [term 1 LEADER]: Becoming Leader. State: Replica: 10081c0117c2422780822c7590f0b49f, State: Running, Role: LEADER
I20251215 21:01:55.716890 30784 consensus_queue.cc:237] T 063a48df2fef4446be204530a8df2ae5 P 10081c0117c2422780822c7590f0b49f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd74be91011a4300b5edb2f531b0c3c5" member_type: VOTER last_known_addr { host: "127.28.113.1" port: 40521 } } peers { permanent_uuid: "10081c0117c2422780822c7590f0b49f" member_type: VOTER last_known_addr { host: "127.28.113.2" port: 36671 } } peers { permanent_uuid: "47a2c6b9946243d4b031a7d3c187f61b" member_type: VOTER last_known_addr { host: "127.28.113.3" port: 35943 } }
I20251215 21:01:55.718474 30260 catalog_manager.cc:5654] T 063a48df2fef4446be204530a8df2ae5 P 10081c0117c2422780822c7590f0b49f reported cstate change: term changed from 0 to 1, leader changed from <none> to 10081c0117c2422780822c7590f0b49f (127.28.113.2). New cstate: current_term: 1 leader_uuid: "10081c0117c2422780822c7590f0b49f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd74be91011a4300b5edb2f531b0c3c5" member_type: VOTER last_known_addr { host: "127.28.113.1" port: 40521 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "10081c0117c2422780822c7590f0b49f" member_type: VOTER last_known_addr { host: "127.28.113.2" port: 36671 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "47a2c6b9946243d4b031a7d3c187f61b" member_type: VOTER last_known_addr { host: "127.28.113.3" port: 35943 } health_report { overall_health: UNKNOWN } } }
Dec 15 21:01:55 dist-test-slave-bjsf krb5kdc[30213](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765832514, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.113.62@KRBTEST.COM for kudu/127.28.113.62@KRBTEST.COM
I20251215 21:01:55.773564 30808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:01:55.765813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.113.62:40865 (local address 127.0.0.1:40142)
1215 21:01:55.766137 (+ 324us) negotiation.cc:107] Waiting for socket to connect
1215 21:01:55.766151 (+ 14us) client_negotiation.cc:174] Beginning negotiation
1215 21:01:55.766261 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1215 21:01:55.766474 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1215 21:01:55.766477 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1215 21:01:55.766888 (+ 411us) client_negotiation.cc:189] Negotiated authn=SASL
1215 21:01:55.767120 (+ 232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:01:55.767129 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:01:55.767878 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1215 21:01:55.767881 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1215 21:01:55.768283 (+ 402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:01:55.768291 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:01:55.768393 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:01:55.769025 (+ 632us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1215 21:01:55.769038 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
1215 21:01:55.770571 (+ 1533us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1215 21:01:55.772118 (+ 1547us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1215 21:01:55.772127 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1215 21:01:55.772133 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
1215 21:01:55.772394 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1215 21:01:55.772635 (+ 241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1215 21:01:55.772638 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1215 21:01:55.772640 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1215 21:01:55.772699 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1215 21:01:55.773051 (+ 352us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1215 21:01:55.773058 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1215 21:01:55.773172 (+ 114us) client_negotiation.cc:769] Sending connection context
1215 21:01:55.773278 (+ 106us) client_negotiation.cc:240] Negotiation successful
1215 21:01:55.773401 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":63,"threads_started":1}
I20251215 21:01:55.773610 30809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:01:55.765896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40142 (local address 127.28.113.62:40865)
1215 21:01:55.766132 (+ 236us) server_negotiation.cc:207] Beginning negotiation
1215 21:01:55.766136 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1215 21:01:55.766217 (+ 81us) server_negotiation.cc:408] Connection header received
1215 21:01:55.766340 (+ 123us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1215 21:01:55.766343 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1215 21:01:55.766379 (+ 36us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1215 21:01:55.766446 (+ 67us) server_negotiation.cc:227] Negotiated authn=SASL
1215 21:01:55.767226 (+ 780us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:01:55.767782 (+ 556us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1215 21:01:55.768381 (+ 599us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:01:55.768559 (+ 178us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:01:55.770678 (+ 2119us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1215 21:01:55.770694 (+ 16us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1215 21:01:55.770696 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1215 21:01:55.770716 (+ 20us) server_negotiation.cc:923] calling sasl_server_start()
1215 21:01:55.772008 (+ 1292us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1215 21:01:55.772497 (+ 489us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1215 21:01:55.772500 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1215 21:01:55.772502 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1215 21:01:55.772553 (+ 51us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1215 21:01:55.772773 (+ 220us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1215 21:01:55.772775 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1215 21:01:55.772777 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1215 21:01:55.772954 (+ 177us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1215 21:01:55.773049 (+ 95us) server_negotiation.cc:1036] Waiting for connection context
1215 21:01:55.773372 (+ 323us) server_negotiation.cc:300] Negotiation successful
1215 21:01:55.773479 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155,"thread_start_us":81,"threads_started":1}
I20251215 21:01:55.777397 30260 catalog_manager.cc:2257] Servicing CreateTable request from {username='kudu', principal='kudu/127.28.113.62@KRBTEST.COM'} at 127.0.0.1:40142:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20251215 21:01:55.778044 30260 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251215 21:01:55.782577 30542 tablet_service.cc:1505] Processing CreateTablet for tablet 26b3b23078cf4145808ec4741cb18d7d (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=6b0cc9affdf545a8a4e3657e8b78079b]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20251215 21:01:55.782536 30400 tablet_service.cc:1505] Processing CreateTablet for tablet 26b3b23078cf4145808ec4741cb18d7d (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=6b0cc9affdf545a8a4e3657e8b78079b]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20251215 21:01:55.782915 30542 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 26b3b23078cf4145808ec4741cb18d7d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 21:01:55.782915 30400 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 26b3b23078cf4145808ec4741cb18d7d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 21:01:55.782867 30683 tablet_service.cc:1505] Processing CreateTablet for tablet 26b3b23078cf4145808ec4741cb18d7d (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=6b0cc9affdf545a8a4e3657e8b78079b]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20251215 21:01:55.783212 30683 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 26b3b23078cf4145808ec4741cb18d7d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 21:01:55.785423 30773 tablet_bootstrap.cc:492] T 26b3b23078cf4145808ec4741cb18d7d P 47a2c6b9946243d4b031a7d3c187f61b: Bootstrap starting.
I20251215 21:01:55.785478 30774 tablet_bootstrap.cc:492] T 26b3b23078cf4145808ec4741cb18d7d P cd74be91011a4300b5edb2f531b0c3c5: Bootstrap starting.
I20251215 21:01:55.785602 30775 tablet_bootstrap.cc:492] T 26b3b23078cf4145808ec4741cb18d7d P 10081c0117c2422780822c7590f0b49f: Bootstrap starting.
I20251215 21:01:55.786286 30775 tablet_bootstrap.cc:654] T 26b3b23078cf4145808ec4741cb18d7d P 10081c0117c2422780822c7590f0b49f: Neither blocks nor log segments found. Creating new log.
I20251215 21:01:55.786415 30773 tablet_bootstrap.cc:654] T 26b3b23078cf4145808ec4741cb18d7d P 47a2c6b9946243d4b031a7d3c187f61b: Neither blocks nor log segments found. Creating new log.
I20251215 21:01:55.786676 30774 tablet_bootstrap.cc:654] T 26b3b23078cf4145808ec4741cb18d7d P cd74be91011a4300b5edb2f531b0c3c5: Neither blocks nor log segments found. Creating new log.
I20251215 21:01:55.787504 30775 tablet_bootstrap.cc:492] T 26b3b23078cf4145808ec4741cb18d7d P 10081c0117c2422780822c7590f0b49f: No bootstrap required, opened a new log
I20251215 21:01:55.787602 30775 ts_tablet_manager.cc:1403] T 26b3b23078cf4145808ec4741cb18d7d P 10081c0117c2422780822c7590f0b49f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20251215 21:01:55.787806 30773 tablet_bootstrap.cc:492] T 26b3b23078cf4145808ec4741cb18d7d P 47a2c6b9946243d4b031a7d3c187f61b: No bootstrap required, opened a new log
I20251215 21:01:55.787894 30773 ts_tablet_manager.cc:1403] T 26b3b23078cf4145808ec4741cb18d7d P 47a2c6b9946243d4b031a7d3c187f61b: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20251215 21:01:55.788046 30775 raft_consensus.cc:359] T 26b3b23078cf4145808ec4741cb18d7d P 10081c0117c2422780822c7590f0b49f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd74be91011a4300b5edb2f531b0c3c5" member_type: VOTER last_known_addr { host: "127.28.113.1" port: 40521 } } peers { permanent_uuid: "10081c0117c2422780822c7590f0b49f" member_type: VOTER last_known_addr { host: "127.28.113.2" port: 36671 } } peers { permanent_uuid: "47a2c6b9946243d4b031a7d3c187f61b" member_type: VOTER last_known_addr { host: "127.28.113.3" port: 35943 } }
I20251215 21:01:55.788161 30775 raft_consensus.cc:385] T 26b3b23078cf4145808ec4741cb18d7d P 10081c0117c2422780822c7590f0b49f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251215 21:01:55.788198 30775 raft_consensus.cc:740] T 26b3b23078cf4145808ec4741cb18d7d P 10081c0117c2422780822c7590f0b49f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10081c0117c2422780822c7590f0b49f, State: Initialized, Role: FOLLOWER
I20251215 21:01:55.788314 30775 consensus_queue.cc:260] T 26b3b23078cf4145808ec4741cb18d7d P 10081c0117c2422780822c7590f0b49f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd74be91011a4300b5edb2f531b0c3c5" member_type: VOTER last_known_addr { host: "127.28.113.1" port: 40521 } } peers { permanent_uuid: "10081c0117c2422780822c7590f0b49f" member_type: VOTER last_known_addr { host: "127.28.113.2" port: 36671 } } peers { permanent_uuid: "47a2c6b9946243d4b031a7d3c187f61b" member_type: VOTER last_known_addr { host: "127.28.113.3" port: 35943 } }
I20251215 21:01:55.788363 30774 tablet_bootstrap.cc:492] T 26b3b23078cf4145808ec4741cb18d7d P cd74be91011a4300b5edb2f531b0c3c5: No bootstrap required, opened a new log
I20251215 21:01:55.788435 30774 ts_tablet_manager.cc:1403] T 26b3b23078cf4145808ec4741cb18d7d P cd74be91011a4300b5edb2f531b0c3c5: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20251215 21:01:55.788579 30775 ts_tablet_manager.cc:1434] T 26b3b23078cf4145808ec4741cb18d7d P 10081c0117c2422780822c7590f0b49f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20251215 21:01:55.788913 30774 raft_consensus.cc:359] T 26b3b23078cf4145808ec4741cb18d7d P cd74be91011a4300b5edb2f531b0c3c5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd74be91011a4300b5edb2f531b0c3c5" member_type: VOTER last_known_addr { host: "127.28.113.1" port: 40521 } } peers { permanent_uuid: "10081c0117c2422780822c7590f0b49f" member_type: VOTER last_known_addr { host: "127.28.113.2" port: 36671 } } peers { permanent_uuid: "47a2c6b9946243d4b031a7d3c187f61b" member_type: VOTER last_known_addr { host: "127.28.113.3" port: 35943 } }
I20251215 21:01:55.789016 30774 raft_consensus.cc:385] T 26b3b23078cf4145808ec4741cb18d7d P cd74be91011a4300b5edb2f531b0c3c5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251215 21:01:55.789048 30774 raft_consensus.cc:740] T 26b3b23078cf4145808ec4741cb18d7d P cd74be91011a4300b5edb2f531b0c3c5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd74be91011a4300b5edb2f531b0c3c5, State: Initialized, Role: FOLLOWER
I20251215 21:01:55.789124 30773 raft_consensus.cc:359] T 26b3b23078cf4145808ec4741cb18d7d P 47a2c6b9946243d4b031a7d3c187f61b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd74be91011a4300b5edb2f531b0c3c5" member_type: VOTER last_known_addr { host: "127.28.113.1" port: 40521 } } peers { permanent_uuid: "10081c0117c2422780822c7590f0b49f" member_type: VOTER last_known_addr { host: "127.28.113.2" port: 36671 } } peers { permanent_uuid: "47a2c6b9946243d4b031a7d3c187f61b" member_type: VOTER last_known_addr { host: "127.28.113.3" port: 35943 } }
I20251215 21:01:55.789157 30774 consensus_queue.cc:260] T 26b3b23078cf4145808ec4741cb18d7d P cd74be91011a4300b5edb2f531b0c3c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd74be91011a4300b5edb2f531b0c3c5" member_type: VOTER last_known_addr { host: "127.28.113.1" port: 40521 } } peers { permanent_uuid: "10081c0117c2422780822c7590f0b49f" member_type: VOTER last_known_addr { host: "127.28.113.2" port: 36671 } } peers { permanent_uuid: "47a2c6b9946243d4b031a7d3c187f61b" member_type: VOTER last_known_addr { host: "127.28.113.3" port: 35943 } }
I20251215 21:01:55.788556 30784 tablet_replica.cc:442] T 26b3b23078cf4145808ec4741cb18d7d P 10081c0117c2422780822c7590f0b49f: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd74be91011a4300b5edb2f531b0c3c5" member_type: VOTER last_known_addr { host: "127.28.113.1" port: 40521 } } peers { permanent_uuid: "10081c0117c2422780822c7590f0b49f" member_type: VOTER last_known_addr { host: "127.28.113.2" port: 36671 } } peers { permanent_uuid: "47a2c6b9946243d4b031a7d3c187f61b" member_type: VOTER last_known_addr { host: "127.28.113.3" port: 35943 } } }
I20251215 21:01:55.789208 30773 raft_consensus.cc:385] T 26b3b23078cf4145808ec4741cb18d7d P 47a2c6b9946243d4b031a7d3c187f61b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251215 21:01:55.789237 30773 raft_consensus.cc:740] T 26b3b23078cf4145808ec4741cb18d7d P 47a2c6b9946243d4b031a7d3c187f61b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47a2c6b9946243d4b031a7d3c187f61b, State: Initialized, Role: FOLLOWER
I20251215 21:01:55.789256 30784 tablet_replica.cc:445] T 26b3b23078cf4145808ec4741cb18d7d P 10081c0117c2422780822c7590f0b49f: This TxnStatusTablet replica's current role is: FOLLOWER
I20251215 21:01:55.789341 30773 consensus_queue.cc:260] T 26b3b23078cf4145808ec4741cb18d7d P 47a2c6b9946243d4b031a7d3c187f61b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd74be91011a4300b5edb2f531b0c3c5" member_type: VOTER last_known_addr { host: "127.28.113.1" port: 40521 } } peers { permanent_uuid: "10081c0117c2422780822c7590f0b49f" member_type: VOTER last_known_addr { host: "127.28.113.2" port: 36671 } } peers { permanent_uuid: "47a2c6b9946243d4b031a7d3c187f61b" member_type: VOTER last_known_addr { host: "127.28.113.3" port: 35943 } }
I20251215 21:01:55.789453 30780 tablet_replica.cc:442] T 26b3b23078cf4145808ec4741cb18d7d P cd74be91011a4300b5edb2f531b0c3c5: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd74be91011a4300b5edb2f531b0c3c5" member_type: VOTER last_known_addr { host: "127.28.113.1" port: 40521 } } peers { permanent_uuid: "10081c0117c2422780822c7590f0b49f" member_type: VOTER last_known_addr { host: "127.28.113.2" port: 36671 } } peers { permanent_uuid: "47a2c6b9946243d4b031a7d3c187f61b" member_type: VOTER last_known_addr { host: "127.28.113.3" port: 35943 } } }
I20251215 21:01:55.789552 30780 tablet_replica.cc:445] T 26b3b23078cf4145808ec4741cb18d7d P cd74be91011a4300b5edb2f531b0c3c5: This TxnStatusTablet replica's current role is: FOLLOWER
I20251215 21:01:55.789564 30773 ts_tablet_manager.cc:1434] T 26b3b23078cf4145808ec4741cb18d7d P 47a2c6b9946243d4b031a7d3c187f61b: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251215 21:01:55.789558 30774 ts_tablet_manager.cc:1434] T 26b3b23078cf4145808ec4741cb18d7d P cd74be91011a4300b5edb2f531b0c3c5: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20251215 21:01:55.789768 30779 tablet_replica.cc:442] T 26b3b23078cf4145808ec4741cb18d7d P 47a2c6b9946243d4b031a7d3c187f61b: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd74be91011a4300b5edb2f531b0c3c5" member_type: VOTER last_known_addr { host: "127.28.113.1" port: 40521 } } peers { permanent_uuid: "10081c0117c2422780822c7590f0b49f" member_type: VOTER last_known_addr { host: "127.28.113.2" port: 36671 } } peers { permanent_uuid: "47a2c6b9946243d4b031a7d3c187f61b" member_type: VOTER last_known_addr { host: "127.28.113.3" port: 35943 } } }
I20251215 21:01:55.789853 30779 tablet_replica.cc:445] T 26b3b23078cf4145808ec4741cb18d7d P 47a2c6b9946243d4b031a7d3c187f61b: This TxnStatusTablet replica's current role is: FOLLOWER
I20251215 21:01:55.912861 30780 raft_consensus.cc:493] T 26b3b23078cf4145808ec4741cb18d7d P cd74be91011a4300b5edb2f531b0c3c5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 21:01:55.913020 30780 raft_consensus.cc:515] T 26b3b23078cf4145808ec4741cb18d7d P cd74be91011a4300b5edb2f531b0c3c5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd74be91011a4300b5edb2f531b0c3c5" member_type: VOTER last_known_addr { host: "127.28.113.1" port: 40521 } } peers { permanent_uuid: "10081c0117c2422780822c7590f0b49f" member_type: VOTER last_known_addr { host: "127.28.113.2" port: 36671 } } peers { permanent_uuid: "47a2c6b9946243d4b031a7d3c187f61b" member_type: VOTER last_known_addr { host: "127.28.113.3" port: 35943 } }
I20251215 21:01:55.914068 30780 leader_election.cc:290] T 26b3b23078cf4145808ec4741cb18d7d P cd74be91011a4300b5edb2f531b0c3c5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10081c0117c2422780822c7590f0b49f (127.28.113.2:36671), 47a2c6b9946243d4b031a7d3c187f61b (127.28.113.3:35943)
I20251215 21:01:55.917057 30813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:01:55.914259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.113.2:36671 (local address 127.28.113.1:47567)
1215 21:01:55.914537 (+ 278us) negotiation.cc:107] Waiting for socket to connect
1215 21:01:55.914551 (+ 14us) client_negotiation.cc:174] Beginning negotiation
1215 21:01:55.914672 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1215 21:01:55.914981 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1215 21:01:55.914984 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1215 21:01:55.915006 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1215 21:01:55.915229 (+ 223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:01:55.915236 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:01:55.915991 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1215 21:01:55.915994 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1215 21:01:55.916590 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:01:55.916598 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:01:55.916711 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:01:55.916737 (+ 26us) client_negotiation.cc:769] Sending connection context
1215 21:01:55.916792 (+ 55us) client_negotiation.cc:240] Negotiation successful
1215 21:01:55.916850 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":107,"threads_started":1}
I20251215 21:01:55.917057 30814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:01:55.914631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.113.3:35943 (local address 127.28.113.1:42639)
1215 21:01:55.914875 (+ 244us) negotiation.cc:107] Waiting for socket to connect
1215 21:01:55.914886 (+ 11us) client_negotiation.cc:174] Beginning negotiation
1215 21:01:55.914957 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1215 21:01:55.915178 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1215 21:01:55.915181 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1215 21:01:55.915197 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1215 21:01:55.915400 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:01:55.915406 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:01:55.916092 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1215 21:01:55.916095 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1215 21:01:55.916673 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:01:55.916680 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:01:55.916777 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:01:55.916796 (+ 19us) client_negotiation.cc:769] Sending connection context
1215 21:01:55.916844 (+ 48us) client_negotiation.cc:240] Negotiation successful
1215 21:01:55.916893 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":76,"threads_started":1}
I20251215 21:01:55.917440 30768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:01:55.914760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.113.1:42639 (local address 127.28.113.3:35943)
1215 21:01:55.914922 (+ 162us) server_negotiation.cc:207] Beginning negotiation
1215 21:01:55.914926 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1215 21:01:55.914939 (+ 13us) server_negotiation.cc:408] Connection header received
1215 21:01:55.915055 (+ 116us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1215 21:01:55.915057 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1215 21:01:55.915099 (+ 42us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1215 21:01:55.915157 (+ 58us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1215 21:01:55.915508 (+ 351us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:01:55.915992 (+ 484us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1215 21:01:55.916776 (+ 784us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:01:55.917208 (+ 432us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:01:55.917243 (+ 35us) server_negotiation.cc:1036] Waiting for connection context
1215 21:01:55.917291 (+ 48us) server_negotiation.cc:300] Negotiation successful
1215 21:01:55.917338 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20251215 21:01:55.917447 30772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:01:55.914343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.113.1:47567 (local address 127.28.113.2:36671)
1215 21:01:55.914479 (+ 136us) server_negotiation.cc:207] Beginning negotiation
1215 21:01:55.914485 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1215 21:01:55.914637 (+ 152us) server_negotiation.cc:408] Connection header received
1215 21:01:55.914796 (+ 159us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1215 21:01:55.914800 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1215 21:01:55.914859 (+ 59us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1215 21:01:55.914949 (+ 90us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1215 21:01:55.915358 (+ 409us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:01:55.915855 (+ 497us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1215 21:01:55.916722 (+ 867us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:01:55.917188 (+ 466us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:01:55.917232 (+ 44us) server_negotiation.cc:1036] Waiting for connection context
1215 21:01:55.917278 (+ 46us) server_negotiation.cc:300] Negotiation successful
1215 21:01:55.917331 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20251215 21:01:55.917816 30704 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "26b3b23078cf4145808ec4741cb18d7d" candidate_uuid: "cd74be91011a4300b5edb2f531b0c3c5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "47a2c6b9946243d4b031a7d3c187f61b" is_pre_election: true
I20251215 21:01:55.917948 30704 raft_consensus.cc:2468] T 26b3b23078cf4145808ec4741cb18d7d P 47a2c6b9946243d4b031a7d3c187f61b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd74be91011a4300b5edb2f531b0c3c5 in term 0.
I20251215 21:01:55.917958 30562 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "26b3b23078cf4145808ec4741cb18d7d" candidate_uuid: "cd74be91011a4300b5edb2f531b0c3c5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10081c0117c2422780822c7590f0b49f" is_pre_election: true
I20251215 21:01:55.918154 30562 raft_consensus.cc:2468] T 26b3b23078cf4145808ec4741cb18d7d P 10081c0117c2422780822c7590f0b49f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd74be91011a4300b5edb2f531b0c3c5 in term 0.
I20251215 21:01:55.918293 30352 leader_election.cc:304] T 26b3b23078cf4145808ec4741cb18d7d P cd74be91011a4300b5edb2f531b0c3c5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 47a2c6b9946243d4b031a7d3c187f61b, cd74be91011a4300b5edb2f531b0c3c5; no voters:
I20251215 21:01:55.918521 30780 raft_consensus.cc:2804] T 26b3b23078cf4145808ec4741cb18d7d P cd74be91011a4300b5edb2f531b0c3c5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251215 21:01:55.918584 30780 raft_consensus.cc:493] T 26b3b23078cf4145808ec4741cb18d7d P cd74be91011a4300b5edb2f531b0c3c5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 21:01:55.918635 30780 raft_consensus.cc:3060] T 26b3b23078cf4145808ec4741cb18d7d P cd74be91011a4300b5edb2f531b0c3c5 [term 0 FOLLOWER]: Advancing to term 1
I20251215 21:01:55.919540 30780 raft_consensus.cc:515] T 26b3b23078cf4145808ec4741cb18d7d P cd74be91011a4300b5edb2f531b0c3c5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd74be91011a4300b5edb2f531b0c3c5" member_type: VOTER last_known_addr { host: "127.28.113.1" port: 40521 } } peers { permanent_uuid: "10081c0117c2422780822c7590f0b49f" member_type: VOTER last_known_addr { host: "127.28.113.2" port: 36671 } } peers { permanent_uuid: "47a2c6b9946243d4b031a7d3c187f61b" member_type: VOTER last_known_addr { host: "127.28.113.3" port: 35943 } }
I20251215 21:01:55.919883 30780 leader_election.cc:290] T 26b3b23078cf4145808ec4741cb18d7d P cd74be91011a4300b5edb2f531b0c3c5 [CANDIDATE]: Term 1 election: Requested vote from peers 10081c0117c2422780822c7590f0b49f (127.28.113.2:36671), 47a2c6b9946243d4b031a7d3c187f61b (127.28.113.3:35943)
I20251215 21:01:55.920205 30562 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "26b3b23078cf4145808ec4741cb18d7d" candidate_uuid: "cd74be91011a4300b5edb2f531b0c3c5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10081c0117c2422780822c7590f0b49f"
I20251215 21:01:55.920265 30704 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "26b3b23078cf4145808ec4741cb18d7d" candidate_uuid: "cd74be91011a4300b5edb2f531b0c3c5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "47a2c6b9946243d4b031a7d3c187f61b"
I20251215 21:01:55.920308 30562 raft_consensus.cc:3060] T 26b3b23078cf4145808ec4741cb18d7d P 10081c0117c2422780822c7590f0b49f [term 0 FOLLOWER]: Advancing to term 1
I20251215 21:01:55.920387 30704 raft_consensus.cc:3060] T 26b3b23078cf4145808ec4741cb18d7d P 47a2c6b9946243d4b031a7d3c187f61b [term 0 FOLLOWER]: Advancing to term 1
I20251215 21:01:55.921142 30562 raft_consensus.cc:2468] T 26b3b23078cf4145808ec4741cb18d7d P 10081c0117c2422780822c7590f0b49f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd74be91011a4300b5edb2f531b0c3c5 in term 1.
I20251215 21:01:55.921222 30704 raft_consensus.cc:2468] T 26b3b23078cf4145808ec4741cb18d7d P 47a2c6b9946243d4b031a7d3c187f61b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd74be91011a4300b5edb2f531b0c3c5 in term 1.
I20251215 21:01:55.921396 30352 leader_election.cc:304] T 26b3b23078cf4145808ec4741cb18d7d P cd74be91011a4300b5edb2f531b0c3c5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10081c0117c2422780822c7590f0b49f, cd74be91011a4300b5edb2f531b0c3c5; no voters:
I20251215 21:01:55.921557 30780 raft_consensus.cc:2804] T 26b3b23078cf4145808ec4741cb18d7d P cd74be91011a4300b5edb2f531b0c3c5 [term 1 FOLLOWER]: Leader election won for term 1
I20251215 21:01:55.921757 30780 raft_consensus.cc:697] T 26b3b23078cf4145808ec4741cb18d7d P cd74be91011a4300b5edb2f531b0c3c5 [term 1 LEADER]: Becoming Leader. State: Replica: cd74be91011a4300b5edb2f531b0c3c5, State: Running, Role: LEADER
I20251215 21:01:55.921988 30780 consensus_queue.cc:237] T 26b3b23078cf4145808ec4741cb18d7d P cd74be91011a4300b5edb2f531b0c3c5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd74be91011a4300b5edb2f531b0c3c5" member_type: VOTER last_known_addr { host: "127.28.113.1" port: 40521 } } peers { permanent_uuid: "10081c0117c2422780822c7590f0b49f" member_type: VOTER last_known_addr { host: "127.28.113.2" port: 36671 } } peers { permanent_uuid: "47a2c6b9946243d4b031a7d3c187f61b" member_type: VOTER last_known_addr { host: "127.28.113.3" port: 35943 } }
I20251215 21:01:55.923120 30815 tablet_replica.cc:442] T 26b3b23078cf4145808ec4741cb18d7d P cd74be91011a4300b5edb2f531b0c3c5: TxnStatusTablet state changed. Reason: New leader cd74be91011a4300b5edb2f531b0c3c5. Latest consensus state: current_term: 1 leader_uuid: "cd74be91011a4300b5edb2f531b0c3c5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd74be91011a4300b5edb2f531b0c3c5" member_type: VOTER last_known_addr { host: "127.28.113.1" port: 40521 } } peers { permanent_uuid: "10081c0117c2422780822c7590f0b49f" member_type: VOTER last_known_addr { host: "127.28.113.2" port: 36671 } } peers { permanent_uuid: "47a2c6b9946243d4b031a7d3c187f61b" member_type: VOTER last_known_addr { host: "127.28.113.3" port: 35943 } } }
I20251215 21:01:55.923224 30815 tablet_replica.cc:445] T 26b3b23078cf4145808ec4741cb18d7d P cd74be91011a4300b5edb2f531b0c3c5: This TxnStatusTablet replica's current role is: LEADER
I20251215 21:01:55.923468 30817 txn_status_manager.cc:874] Waiting until node catch up with all replicated operations in previous term...
I20251215 21:01:55.923609 30817 txn_status_manager.cc:930] Loading transaction status metadata into memory...
I20251215 21:01:55.924356 30259 catalog_manager.cc:5654] T 26b3b23078cf4145808ec4741cb18d7d P cd74be91011a4300b5edb2f531b0c3c5 reported cstate change: term changed from 0 to 1, leader changed from <none> to cd74be91011a4300b5edb2f531b0c3c5 (127.28.113.1). New cstate: current_term: 1 leader_uuid: "cd74be91011a4300b5edb2f531b0c3c5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd74be91011a4300b5edb2f531b0c3c5" member_type: VOTER last_known_addr { host: "127.28.113.1" port: 40521 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "10081c0117c2422780822c7590f0b49f" member_type: VOTER last_known_addr { host: "127.28.113.2" port: 36671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "47a2c6b9946243d4b031a7d3c187f61b" member_type: VOTER last_known_addr { host: "127.28.113.3" port: 35943 } health_report { overall_health: UNKNOWN } } }
I20251215 21:01:55.942972 30808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:01:55.939637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.113.1:40521 (local address 127.0.0.1:37476)
1215 21:01:55.939778 (+ 141us) negotiation.cc:107] Waiting for socket to connect
1215 21:01:55.939794 (+ 16us) client_negotiation.cc:174] Beginning negotiation
1215 21:01:55.939891 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1215 21:01:55.940222 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1215 21:01:55.940224 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1215 21:01:55.940231 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
1215 21:01:55.940412 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:01:55.940418 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:01:55.941368 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1215 21:01:55.941370 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1215 21:01:55.941791 (+ 421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:01:55.941798 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:01:55.941879 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:01:55.941919 (+ 40us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
1215 21:01:55.942741 (+ 822us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
1215 21:01:55.942749 (+ 8us) client_negotiation.cc:769] Sending connection context
1215 21:01:55.942803 (+ 54us) client_negotiation.cc:240] Negotiation successful
1215 21:01:55.942861 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20251215 21:01:55.943019 30770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:01:55.939740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37476 (local address 127.28.113.1:40521)
1215 21:01:55.939866 (+ 126us) server_negotiation.cc:207] Beginning negotiation
1215 21:01:55.939870 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1215 21:01:55.939885 (+ 15us) server_negotiation.cc:408] Connection header received
1215 21:01:55.940038 (+ 153us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1215 21:01:55.940042 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1215 21:01:55.940119 (+ 77us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1215 21:01:55.940209 (+ 90us) server_negotiation.cc:227] Negotiated authn=TOKEN
1215 21:01:55.940522 (+ 313us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:01:55.941273 (+ 751us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1215 21:01:55.941888 (+ 615us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:01:55.942077 (+ 189us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:01:55.942130 (+ 53us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1215 21:01:55.942652 (+ 522us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1215 21:01:55.942755 (+ 103us) server_negotiation.cc:1036] Waiting for connection context
1215 21:01:55.942827 (+ 72us) server_negotiation.cc:300] Negotiation successful
1215 21:01:55.942894 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20251215 21:01:55.945780 30704 raft_consensus.cc:1275] T 26b3b23078cf4145808ec4741cb18d7d P 47a2c6b9946243d4b031a7d3c187f61b [term 1 FOLLOWER]: Refusing update from remote peer cd74be91011a4300b5edb2f531b0c3c5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251215 21:01:55.945952 30562 raft_consensus.cc:1275] T 26b3b23078cf4145808ec4741cb18d7d P 10081c0117c2422780822c7590f0b49f [term 1 FOLLOWER]: Refusing update from remote peer cd74be91011a4300b5edb2f531b0c3c5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251215 21:01:55.946455 30815 consensus_queue.cc:1048] T 26b3b23078cf4145808ec4741cb18d7d P cd74be91011a4300b5edb2f531b0c3c5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10081c0117c2422780822c7590f0b49f" member_type: VOTER last_known_addr { host: "127.28.113.2" port: 36671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251215 21:01:55.946669 30780 consensus_queue.cc:1048] T 26b3b23078cf4145808ec4741cb18d7d P cd74be91011a4300b5edb2f531b0c3c5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "47a2c6b9946243d4b031a7d3c187f61b" member_type: VOTER last_known_addr { host: "127.28.113.3" port: 35943 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251215 21:01:55.948266 30784 tablet_replica.cc:442] T 26b3b23078cf4145808ec4741cb18d7d P 10081c0117c2422780822c7590f0b49f: TxnStatusTablet state changed. Reason: New leader cd74be91011a4300b5edb2f531b0c3c5. Latest consensus state: current_term: 1 leader_uuid: "cd74be91011a4300b5edb2f531b0c3c5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd74be91011a4300b5edb2f531b0c3c5" member_type: VOTER last_known_addr { host: "127.28.113.1" port: 40521 } } peers { permanent_uuid: "10081c0117c2422780822c7590f0b49f" member_type: VOTER last_known_addr { host: "127.28.113.2" port: 36671 } } peers { permanent_uuid: "47a2c6b9946243d4b031a7d3c187f61b" member_type: VOTER last_known_addr { host: "127.28.113.3" port: 35943 } } }
I20251215 21:01:55.948361 30784 tablet_replica.cc:445] T 26b3b23078cf4145808ec4741cb18d7d P 10081c0117c2422780822c7590f0b49f: This TxnStatusTablet replica's current role is: FOLLOWER
I20251215 21:01:55.948702 30779 tablet_replica.cc:442] T 26b3b23078cf4145808ec4741cb18d7d P 47a2c6b9946243d4b031a7d3c187f61b: TxnStatusTablet state changed. Reason: New leader cd74be91011a4300b5edb2f531b0c3c5. Latest consensus state: current_term: 1 leader_uuid: "cd74be91011a4300b5edb2f531b0c3c5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd74be91011a4300b5edb2f531b0c3c5" member_type: VOTER last_known_addr { host: "127.28.113.1" port: 40521 } } peers { permanent_uuid: "10081c0117c2422780822c7590f0b49f" member_type: VOTER last_known_addr { host: "127.28.113.2" port: 36671 } } peers { permanent_uuid: "47a2c6b9946243d4b031a7d3c187f61b" member_type: VOTER last_known_addr { host: "127.28.113.3" port: 35943 } } }
I20251215 21:01:55.948801 30779 tablet_replica.cc:445] T 26b3b23078cf4145808ec4741cb18d7d P 47a2c6b9946243d4b031a7d3c187f61b: This TxnStatusTablet replica's current role is: FOLLOWER
I20251215 21:01:55.950328 30815 tablet_replica.cc:442] T 26b3b23078cf4145808ec4741cb18d7d P cd74be91011a4300b5edb2f531b0c3c5: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "cd74be91011a4300b5edb2f531b0c3c5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd74be91011a4300b5edb2f531b0c3c5" member_type: VOTER last_known_addr { host: "127.28.113.1" port: 40521 } } peers { permanent_uuid: "10081c0117c2422780822c7590f0b49f" member_type: VOTER last_known_addr { host: "127.28.113.2" port: 36671 } } peers { permanent_uuid: "47a2c6b9946243d4b031a7d3c187f61b" member_type: VOTER last_known_addr { host: "127.28.113.3" port: 35943 } } }
I20251215 21:01:55.950414 30815 tablet_replica.cc:445] T 26b3b23078cf4145808ec4741cb18d7d P cd74be91011a4300b5edb2f531b0c3c5: This TxnStatusTablet replica's current role is: LEADER
I20251215 21:01:55.951846 30784 tablet_replica.cc:442] T 26b3b23078cf4145808ec4741cb18d7d P 10081c0117c2422780822c7590f0b49f: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "cd74be91011a4300b5edb2f531b0c3c5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd74be91011a4300b5edb2f531b0c3c5" member_type: VOTER last_known_addr { host: "127.28.113.1" port: 40521 } } peers { permanent_uuid: "10081c0117c2422780822c7590f0b49f" member_type: VOTER last_known_addr { host: "127.28.113.2" port: 36671 } } peers { permanent_uuid: "47a2c6b9946243d4b031a7d3c187f61b" member_type: VOTER last_known_addr { host: "127.28.113.3" port: 35943 } } }
I20251215 21:01:55.951942 30784 tablet_replica.cc:445] T 26b3b23078cf4145808ec4741cb18d7d P 10081c0117c2422780822c7590f0b49f: This TxnStatusTablet replica's current role is: FOLLOWER
I20251215 21:01:55.952260 30787 mvcc.cc:204] Tried to move back new op lower bound from 7232849985309413376 to 7232849985218949120. Current Snapshot: MvccSnapshot[applied={T|T < 7232849985309413376 or (T in {7232849985309413376})}]
I20251215 21:01:55.953389 30780 tablet_replica.cc:442] T 26b3b23078cf4145808ec4741cb18d7d P cd74be91011a4300b5edb2f531b0c3c5: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "cd74be91011a4300b5edb2f531b0c3c5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd74be91011a4300b5edb2f531b0c3c5" member_type: VOTER last_known_addr { host: "127.28.113.1" port: 40521 } } peers { permanent_uuid: "10081c0117c2422780822c7590f0b49f" member_type: VOTER last_known_addr { host: "127.28.113.2" port: 36671 } } peers { permanent_uuid: "47a2c6b9946243d4b031a7d3c187f61b" member_type: VOTER last_known_addr { host: "127.28.113.3" port: 35943 } } }
I20251215 21:01:55.953486 30780 tablet_replica.cc:445] T 26b3b23078cf4145808ec4741cb18d7d P cd74be91011a4300b5edb2f531b0c3c5: This TxnStatusTablet replica's current role is: LEADER
I20251215 21:01:55.954885 30779 tablet_replica.cc:442] T 26b3b23078cf4145808ec4741cb18d7d P 47a2c6b9946243d4b031a7d3c187f61b: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "cd74be91011a4300b5edb2f531b0c3c5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd74be91011a4300b5edb2f531b0c3c5" member_type: VOTER last_known_addr { host: "127.28.113.1" port: 40521 } } peers { permanent_uuid: "10081c0117c2422780822c7590f0b49f" member_type: VOTER last_known_addr { host: "127.28.113.2" port: 36671 } } peers { permanent_uuid: "47a2c6b9946243d4b031a7d3c187f61b" member_type: VOTER last_known_addr { host: "127.28.113.3" port: 35943 } } }
I20251215 21:01:55.954980 30779 tablet_replica.cc:445] T 26b3b23078cf4145808ec4741cb18d7d P 47a2c6b9946243d4b031a7d3c187f61b: This TxnStatusTablet replica's current role is: FOLLOWER
I20251215 21:01:55.963617 30770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:01:55.960741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.113.2:56843 (local address 127.28.113.1:40521)
1215 21:01:55.960860 (+ 119us) server_negotiation.cc:207] Beginning negotiation
1215 21:01:55.960865 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1215 21:01:55.961088 (+ 223us) server_negotiation.cc:408] Connection header received
1215 21:01:55.961205 (+ 117us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1215 21:01:55.961208 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1215 21:01:55.961252 (+ 44us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1215 21:01:55.961320 (+ 68us) server_negotiation.cc:227] Negotiated authn=TOKEN
1215 21:01:55.961759 (+ 439us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:01:55.962223 (+ 464us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1215 21:01:55.963092 (+ 869us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:01:55.963222 (+ 130us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:01:55.963258 (+ 36us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1215 21:01:55.963317 (+ 59us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1215 21:01:55.963377 (+ 60us) server_negotiation.cc:1036] Waiting for connection context
1215 21:01:55.963472 (+ 95us) server_negotiation.cc:300] Negotiation successful
1215 21:01:55.963520 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20251215 21:01:55.963665 30825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:01:55.960615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.113.1:40521 (local address 127.28.113.2:56843)
1215 21:01:55.961009 (+ 394us) negotiation.cc:107] Waiting for socket to connect
1215 21:01:55.961023 (+ 14us) client_negotiation.cc:174] Beginning negotiation
1215 21:01:55.961125 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1215 21:01:55.961334 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1215 21:01:55.961337 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1215 21:01:55.961345 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
1215 21:01:55.961617 (+ 272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:01:55.961624 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:01:55.962312 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1215 21:01:55.962315 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1215 21:01:55.962980 (+ 665us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:01:55.962989 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:01:55.963093 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:01:55.963137 (+ 44us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
1215 21:01:55.963395 (+ 258us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
1215 21:01:55.963400 (+ 5us) client_negotiation.cc:769] Sending connection context
1215 21:01:55.963457 (+ 57us) client_negotiation.cc:240] Negotiation successful
1215 21:01:55.963531 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":294,"thread_start_us":102,"threads_started":1}
I20251215 21:01:55.969611 30704 raft_consensus.cc:1275] T 063a48df2fef4446be204530a8df2ae5 P 47a2c6b9946243d4b031a7d3c187f61b [term 1 FOLLOWER]: Refusing update from remote peer 10081c0117c2422780822c7590f0b49f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251215 21:01:55.969661 30420 raft_consensus.cc:1275] T 063a48df2fef4446be204530a8df2ae5 P cd74be91011a4300b5edb2f531b0c3c5 [term 1 FOLLOWER]: Refusing update from remote peer 10081c0117c2422780822c7590f0b49f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251215 21:01:55.970005 30781 consensus_queue.cc:1048] T 063a48df2fef4446be204530a8df2ae5 P 10081c0117c2422780822c7590f0b49f [LEADER]: Connected to new peer: Peer: permanent_uuid: "47a2c6b9946243d4b031a7d3c187f61b" member_type: VOTER last_known_addr { host: "127.28.113.3" port: 35943 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251215 21:01:55.970146 30784 consensus_queue.cc:1048] T 063a48df2fef4446be204530a8df2ae5 P 10081c0117c2422780822c7590f0b49f [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd74be91011a4300b5edb2f531b0c3c5" member_type: VOTER last_known_addr { host: "127.28.113.1" port: 40521 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251215 21:01:55.991079 30772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:01:55.988099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.113.1:43863 (local address 127.28.113.2:36671)
1215 21:01:55.988203 (+ 104us) server_negotiation.cc:207] Beginning negotiation
1215 21:01:55.988208 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1215 21:01:55.988431 (+ 223us) server_negotiation.cc:408] Connection header received
1215 21:01:55.988579 (+ 148us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1215 21:01:55.988582 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1215 21:01:55.988623 (+ 41us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1215 21:01:55.988681 (+ 58us) server_negotiation.cc:227] Negotiated authn=TOKEN
1215 21:01:55.989131 (+ 450us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:01:55.989771 (+ 640us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1215 21:01:55.990427 (+ 656us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:01:55.990638 (+ 211us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:01:55.990681 (+ 43us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1215 21:01:55.990748 (+ 67us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1215 21:01:55.990805 (+ 57us) server_negotiation.cc:1036] Waiting for connection context
1215 21:01:55.990903 (+ 98us) server_negotiation.cc:300] Negotiation successful
1215 21:01:55.990946 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20251215 21:01:55.991094 30831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:01:55.988065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.113.2:36671 (local address 127.28.113.1:43863)
1215 21:01:55.988356 (+ 291us) negotiation.cc:107] Waiting for socket to connect
1215 21:01:55.988369 (+ 13us) client_negotiation.cc:174] Beginning negotiation
1215 21:01:55.988465 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1215 21:01:55.988698 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1215 21:01:55.988701 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1215 21:01:55.988709 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
1215 21:01:55.989030 (+ 321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:01:55.989039 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:01:55.989880 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1215 21:01:55.989884 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1215 21:01:55.990291 (+ 407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:01:55.990300 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:01:55.990432 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:01:55.990477 (+ 45us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
1215 21:01:55.990830 (+ 353us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
1215 21:01:55.990835 (+ 5us) client_negotiation.cc:769] Sending connection context
1215 21:01:55.990889 (+ 54us) client_negotiation.cc:240] Negotiation successful
1215 21:01:55.990953 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":108,"threads_started":1}
I20251215 21:01:56.016515 30260 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40128:
table { table_name: "test-table" } modify_external_catalogs: true
I20251215 21:01:56.016672 30260 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40128:
table { table_name: "test-table" } modify_external_catalogs: true
I20251215 21:01:56.018859 30260 catalog_manager.cc:5941] T 00000000000000000000000000000000 P 700c44b9a4694ad6993e6d84e5c0475c: Sending DeleteTablet for 3 replicas of tablet 063a48df2fef4446be204530a8df2ae5
I20251215 21:01:56.019512 30400 tablet_service.cc:1552] Processing DeleteTablet for tablet 063a48df2fef4446be204530a8df2ae5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-15 21:01:56 UTC) from {username='kudu', principal='kudu/127.28.113.62@KRBTEST.COM'} at 127.0.0.1:37470
I20251215 21:01:56.019624 30683 tablet_service.cc:1552] Processing DeleteTablet for tablet 063a48df2fef4446be204530a8df2ae5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-15 21:01:56 UTC) from {username='kudu', principal='kudu/127.28.113.62@KRBTEST.COM'} at 127.0.0.1:58698
I20251215 21:01:56.019819 30797 tablet_replica.cc:333] T 063a48df2fef4446be204530a8df2ae5 P cd74be91011a4300b5edb2f531b0c3c5: stopping tablet replica
I20251215 21:01:56.019815 30796 tablet_replica.cc:333] T 063a48df2fef4446be204530a8df2ae5 P 47a2c6b9946243d4b031a7d3c187f61b: stopping tablet replica
I20251215 21:01:56.019929 30796 raft_consensus.cc:2243] T 063a48df2fef4446be204530a8df2ae5 P 47a2c6b9946243d4b031a7d3c187f61b [term 1 FOLLOWER]: Raft consensus shutting down.
I20251215 21:01:56.019929 30797 raft_consensus.cc:2243] T 063a48df2fef4446be204530a8df2ae5 P cd74be91011a4300b5edb2f531b0c3c5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251215 21:01:56.020057 30797 raft_consensus.cc:2272] T 063a48df2fef4446be204530a8df2ae5 P cd74be91011a4300b5edb2f531b0c3c5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251215 21:01:56.020058 30796 raft_consensus.cc:2272] T 063a48df2fef4446be204530a8df2ae5 P 47a2c6b9946243d4b031a7d3c187f61b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251215 21:01:56.020565 30542 tablet_service.cc:1552] Processing DeleteTablet for tablet 063a48df2fef4446be204530a8df2ae5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-15 21:01:56 UTC) from {username='kudu', principal='kudu/127.28.113.62@KRBTEST.COM'} at 127.0.0.1:44928
I20251215 21:01:56.020740 30798 tablet_replica.cc:333] T 063a48df2fef4446be204530a8df2ae5 P 10081c0117c2422780822c7590f0b49f: stopping tablet replica
I20251215 21:01:56.020850 30798 raft_consensus.cc:2243] T 063a48df2fef4446be204530a8df2ae5 P 10081c0117c2422780822c7590f0b49f [term 1 LEADER]: Raft consensus shutting down.
I20251215 21:01:56.020897 30796 ts_tablet_manager.cc:1916] T 063a48df2fef4446be204530a8df2ae5 P 47a2c6b9946243d4b031a7d3c187f61b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251215 21:01:56.021031 30798 raft_consensus.cc:2272] T 063a48df2fef4446be204530a8df2ae5 P 10081c0117c2422780822c7590f0b49f [term 1 FOLLOWER]: Raft consensus is shut down!
I20251215 21:01:56.021050 30797 ts_tablet_manager.cc:1916] T 063a48df2fef4446be204530a8df2ae5 P cd74be91011a4300b5edb2f531b0c3c5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251215 21:01:56.021636 30798 ts_tablet_manager.cc:1916] T 063a48df2fef4446be204530a8df2ae5 P 10081c0117c2422780822c7590f0b49f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251215 21:01:56.023429 30796 ts_tablet_manager.cc:1929] T 063a48df2fef4446be204530a8df2ae5 P 47a2c6b9946243d4b031a7d3c187f61b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20251215 21:01:56.023428 30797 ts_tablet_manager.cc:1929] T 063a48df2fef4446be204530a8df2ae5 P cd74be91011a4300b5edb2f531b0c3c5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20251215 21:01:56.023481 30798 ts_tablet_manager.cc:1929] T 063a48df2fef4446be204530a8df2ae5 P 10081c0117c2422780822c7590f0b49f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20251215 21:01:56.023504 30796 log.cc:1199] T 063a48df2fef4446be204530a8df2ae5 P 47a2c6b9946243d4b031a7d3c187f61b: Deleting WAL directory at /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765832506145157-29124-0/minicluster-data/ts-2/wal/wals/063a48df2fef4446be204530a8df2ae5
I20251215 21:01:56.023511 30797 log.cc:1199] T 063a48df2fef4446be204530a8df2ae5 P cd74be91011a4300b5edb2f531b0c3c5: Deleting WAL directory at /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765832506145157-29124-0/minicluster-data/ts-0/wal/wals/063a48df2fef4446be204530a8df2ae5
I20251215 21:01:56.023545 30798 log.cc:1199] T 063a48df2fef4446be204530a8df2ae5 P 10081c0117c2422780822c7590f0b49f: Deleting WAL directory at /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765832506145157-29124-0/minicluster-data/ts-1/wal/wals/063a48df2fef4446be204530a8df2ae5
I20251215 21:01:56.023737 30797 ts_tablet_manager.cc:1950] T 063a48df2fef4446be204530a8df2ae5 P cd74be91011a4300b5edb2f531b0c3c5: Deleting consensus metadata
I20251215 21:01:56.023754 30798 ts_tablet_manager.cc:1950] T 063a48df2fef4446be204530a8df2ae5 P 10081c0117c2422780822c7590f0b49f: Deleting consensus metadata
I20251215 21:01:56.023737 30796 ts_tablet_manager.cc:1950] T 063a48df2fef4446be204530a8df2ae5 P 47a2c6b9946243d4b031a7d3c187f61b: Deleting consensus metadata
I20251215 21:01:56.024482 30246 catalog_manager.cc:4985] TS 10081c0117c2422780822c7590f0b49f (127.28.113.2:36671): tablet 063a48df2fef4446be204530a8df2ae5 (table test-table [id=ec582cc4325f465d8a5cda6c6f83e584]) successfully deleted
I20251215 21:01:56.024653 30246 catalog_manager.cc:4985] TS 47a2c6b9946243d4b031a7d3c187f61b (127.28.113.3:35943): tablet 063a48df2fef4446be204530a8df2ae5 (table test-table [id=ec582cc4325f465d8a5cda6c6f83e584]) successfully deleted
I20251215 21:01:56.025811 30247 catalog_manager.cc:4985] TS cd74be91011a4300b5edb2f531b0c3c5 (127.28.113.1:40521): tablet 063a48df2fef4446be204530a8df2ae5 (table test-table [id=ec582cc4325f465d8a5cda6c6f83e584]) successfully deleted
Dec 15 21:01:56 dist-test-slave-bjsf krb5kdc[30213](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765832515, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.28.113.1@KRBTEST.COM
I20251215 21:01:56.037497 30770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:01:56.028340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37490 (local address 127.28.113.1:40521)
1215 21:01:56.028464 (+ 124us) server_negotiation.cc:207] Beginning negotiation
1215 21:01:56.028469 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1215 21:01:56.028749 (+ 280us) server_negotiation.cc:408] Connection header received
1215 21:01:56.028936 (+ 187us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1215 21:01:56.028940 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1215 21:01:56.028988 (+ 48us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1215 21:01:56.029034 (+ 46us) server_negotiation.cc:227] Negotiated authn=SASL
1215 21:01:56.030115 (+ 1081us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:01:56.030608 (+ 493us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1215 21:01:56.031468 (+ 860us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:01:56.031606 (+ 138us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:01:56.033913 (+ 2307us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1215 21:01:56.033936 (+ 23us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1215 21:01:56.033949 (+ 13us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1215 21:01:56.033974 (+ 25us) server_negotiation.cc:923] calling sasl_server_start()
1215 21:01:56.036081 (+ 2107us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1215 21:01:56.036444 (+ 363us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1215 21:01:56.036448 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1215 21:01:56.036450 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1215 21:01:56.036503 (+ 53us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1215 21:01:56.036693 (+ 190us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1215 21:01:56.036696 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1215 21:01:56.036699 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
1215 21:01:56.036882 (+ 183us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1215 21:01:56.037058 (+ 176us) server_negotiation.cc:1036] Waiting for connection context
1215 21:01:56.037231 (+ 173us) server_negotiation.cc:300] Negotiation successful
1215 21:01:56.037346 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
W20251215 21:01:56.037889 30460 server_base.cc:1129] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37490
I20251215 21:01:56.046149 30809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:01:56.039378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40156 (local address 127.28.113.62:40865)
1215 21:01:56.039515 (+ 137us) server_negotiation.cc:207] Beginning negotiation
1215 21:01:56.039520 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1215 21:01:56.039666 (+ 146us) server_negotiation.cc:408] Connection header received
1215 21:01:56.039771 (+ 105us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1215 21:01:56.039774 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1215 21:01:56.039823 (+ 49us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1215 21:01:56.039886 (+ 63us) server_negotiation.cc:227] Negotiated authn=SASL
1215 21:01:56.040759 (+ 873us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:01:56.041244 (+ 485us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1215 21:01:56.042031 (+ 787us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:01:56.042163 (+ 132us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:01:56.043301 (+ 1138us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1215 21:01:56.043320 (+ 19us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1215 21:01:56.043322 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1215 21:01:56.043345 (+ 23us) server_negotiation.cc:923] calling sasl_server_start()
1215 21:01:56.044735 (+ 1390us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1215 21:01:56.045143 (+ 408us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1215 21:01:56.045147 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1215 21:01:56.045148 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1215 21:01:56.045202 (+ 54us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1215 21:01:56.045406 (+ 204us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1215 21:01:56.045408 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1215 21:01:56.045409 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1215 21:01:56.045558 (+ 149us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1215 21:01:56.045660 (+ 102us) server_negotiation.cc:1036] Waiting for connection context
1215 21:01:56.045858 (+ 198us) server_negotiation.cc:300] Negotiation successful
1215 21:01:56.045976 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
W20251215 21:01:56.046547 30260 server_base.cc:1129] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40156
I20251215 21:01:56.047734 29124 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu with pid 30335
I20251215 21:01:56.054243 29124 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu with pid 30477
I20251215 21:01:56.060132 29124 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu with pid 30619
I20251215 21:01:56.065320 29124 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu with pid 30229
2025-12-15T21:01:56Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (3657 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765832506145157-29124-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Dec 15 21:01:56 dist-test-slave-bjsf krb5kdc[30839](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Dec 15 21:01:56 dist-test-slave-bjsf krb5kdc[30839](info): set up 2 sockets
Dec 15 21:01:56 dist-test-slave-bjsf krb5kdc[30839](info): commencing operation
krb5kdc: starting...
W20251215 21:01:57.883782 29124 mini_kdc.cc:121] Time spent starting KDC: real 1.798s user 0.004s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765832506145157-29124-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765832506145157-29124-0/krb5kdc/test-user.keytab.
Dec 15 21:01:57 dist-test-slave-bjsf krb5kdc[30839](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765832517, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-12-15T21:01:57Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-12-15T21:01:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.113.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.113.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.113.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765832506145157-29124-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.28.113.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765832506145157-29124-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.28.113.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.28.113.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.28.113.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765832506145157-29124-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.28.113.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765832506145157-29124-0/krb5kdc/kudu.keytab.
I20251215 21:01:58.000412 29124 external_mini_cluster.cc:1366] Running /tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu
/tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765832506145157-29124-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765832506145157-29124-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765832506145157-29124-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765832506145157-29124-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.113.62:37805
--webserver_interface=127.28.113.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765832506145157-29124-0/krb5kdc/kudu.keytab
--principal=kudu/127.28.113.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.28.113.20:37295
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.113.62:37805
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765832506145157-29124-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765832506145157-29124-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765832506145157-29124-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251215 21:01:58.101107 30855 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251215 21:01:58.101328 30855 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251215 21:01:58.101389 30855 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251215 21:01:58.104605 30855 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251215 21:01:58.104657 30855 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20251215 21:01:58.104677 30855 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251215 21:01:58.104714 30855 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251215 21:01:58.104744 30855 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251215 21:01:58.104776 30855 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251215 21:01:58.108866 30855 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.113.20:37295
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765832506145157-29124-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765832506145157-29124-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.113.62:37805
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765832506145157-29124-0/krb5kdc/kudu.keytab
--principal=kudu/127.28.113.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.113.62:37805
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765832506145157-29124-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.113.62
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30855
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765832506145157-29124-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision dc741732bb6a40213e11a13869fad06a49a209a3
build type DEBUG
built by None at 15 Dec 2025 20:43:14 UTC on 24a791456cd2
build id 9530
I20251215 21:01:58.109833 30855 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251215 21:01:58.110620 30855 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251215 21:01:58.115738 30863 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251215 21:01:58.115847 30855 server_base.cc:1047] running on GCE node
W20251215 21:01:58.115789 30860 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251215 21:01:58.115767 30861 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251215 21:01:58.116392 30855 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251215 21:01:58.117221 30855 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251215 21:01:58.118356 30855 hybrid_clock.cc:648] HybridClock initialized: now 1765832518118346 us; error 26 us; skew 500 ppm
Dec 15 21:01:58 dist-test-slave-bjsf krb5kdc[30839](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765832518, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.113.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251215 21:01:58.120743 30855 init.cc:377] Logged in from keytab as kudu/127.28.113.62@KRBTEST.COM (short username kudu)
I20251215 21:01:58.121650 30855 webserver.cc:492] Webserver started at http://127.28.113.62:33515/ using document root <none> and password file <none>
I20251215 21:01:58.122118 30855 fs_manager.cc:362] Metadata directory not provided
I20251215 21:01:58.122160 30855 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251215 21:01:58.122313 30855 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251215 21:01:58.123893 30855 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765832506145157-29124-0/minicluster-data/master-0/data/instance:
uuid: "8d6aede0d59c4dbd8c8bca7ade69344b"
format_stamp: "Formatted at 2025-12-15 21:01:58 on dist-test-slave-bjsf"
server_key: "8070bc2706e6d57fa510bab3c48f0869"
server_key_iv: "dd2b4c5af7f02b04b3684a08a86acbd0"
server_key_version: "encryptionkey@0"
I20251215 21:01:58.124339 30855 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765832506145157-29124-0/minicluster-data/master-0/wal/instance:
uuid: "8d6aede0d59c4dbd8c8bca7ade69344b"
format_stamp: "Formatted at 2025-12-15 21:01:58 on dist-test-slave-bjsf"
server_key: "8070bc2706e6d57fa510bab3c48f0869"
server_key_iv: "dd2b4c5af7f02b04b3684a08a86acbd0"
server_key_version: "encryptionkey@0"
I20251215 21:01:58.127313 30855 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20251215 21:01:58.129335 30870 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251215 21:01:58.130244 30855 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20251215 21:01:58.130332 30855 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765832506145157-29124-0/minicluster-data/master-0/data,/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765832506145157-29124-0/minicluster-data/master-0/wal
uuid: "8d6aede0d59c4dbd8c8bca7ade69344b"
format_stamp: "Formatted at 2025-12-15 21:01:58 on dist-test-slave-bjsf"
server_key: "8070bc2706e6d57fa510bab3c48f0869"
server_key_iv: "dd2b4c5af7f02b04b3684a08a86acbd0"
server_key_version: "encryptionkey@0"
I20251215 21:01:58.130401 30855 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765832506145157-29124-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765832506145157-29124-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765832506145157-29124-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251215 21:01:58.157871 30855 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251215 21:01:58.160322 30855 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251215 21:01:58.160458 30855 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251215 21:01:58.167580 30855 rpc_server.cc:307] RPC server started. Bound to: 127.28.113.62:37805
I20251215 21:01:58.167804 30932 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.28.113.62:37805 every 8 connection(s)
I20251215 21:01:58.168493 30855 server_base.cc:1179] Dumped server information to /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765832506145157-29124-0/minicluster-data/master-0/data/info.pb
I20251215 21:01:58.171021 30933 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 21:01:58.174804 29124 external_mini_cluster.cc:1428] Started /tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu as pid 30855
I20251215 21:01:58.174912 29124 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765832506145157-29124-0/minicluster-data/master-0/wal/instance
I20251215 21:01:58.175107 29124 external_mini_cluster.cc:1468] Setting key aa5a960d2cccff558f3a9099eea52243
I20251215 21:01:58.176115 30933 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8d6aede0d59c4dbd8c8bca7ade69344b: Bootstrap starting.
I20251215 21:01:58.178303 30933 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8d6aede0d59c4dbd8c8bca7ade69344b: Neither blocks nor log segments found. Creating new log.
I20251215 21:01:58.178995 30933 log.cc:826] T 00000000000000000000000000000000 P 8d6aede0d59c4dbd8c8bca7ade69344b: Log is configured to *not* fsync() on all Append() calls
I20251215 21:01:58.180693 30933 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8d6aede0d59c4dbd8c8bca7ade69344b: No bootstrap required, opened a new log
Dec 15 21:01:58 dist-test-slave-bjsf krb5kdc[30839](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765832517, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.113.62@KRBTEST.COM
I20251215 21:01:58.183074 30933 raft_consensus.cc:359] T 00000000000000000000000000000000 P 8d6aede0d59c4dbd8c8bca7ade69344b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d6aede0d59c4dbd8c8bca7ade69344b" member_type: VOTER last_known_addr { host: "127.28.113.62" port: 37805 } }
I20251215 21:01:58.183243 30933 raft_consensus.cc:385] T 00000000000000000000000000000000 P 8d6aede0d59c4dbd8c8bca7ade69344b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251215 21:01:58.183308 30933 raft_consensus.cc:740] T 00000000000000000000000000000000 P 8d6aede0d59c4dbd8c8bca7ade69344b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d6aede0d59c4dbd8c8bca7ade69344b, State: Initialized, Role: FOLLOWER
I20251215 21:01:58.183740 30933 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8d6aede0d59c4dbd8c8bca7ade69344b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d6aede0d59c4dbd8c8bca7ade69344b" member_type: VOTER last_known_addr { host: "127.28.113.62" port: 37805 } }
I20251215 21:01:58.183858 30933 raft_consensus.cc:399] T 00000000000000000000000000000000 P 8d6aede0d59c4dbd8c8bca7ade69344b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251215 21:01:58.183914 30933 raft_consensus.cc:493] T 00000000000000000000000000000000 P 8d6aede0d59c4dbd8c8bca7ade69344b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251215 21:01:58.184017 30933 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 8d6aede0d59c4dbd8c8bca7ade69344b [term 0 FOLLOWER]: Advancing to term 1
I20251215 21:01:58.184720 30933 raft_consensus.cc:515] T 00000000000000000000000000000000 P 8d6aede0d59c4dbd8c8bca7ade69344b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d6aede0d59c4dbd8c8bca7ade69344b" member_type: VOTER last_known_addr { host: "127.28.113.62" port: 37805 } }
I20251215 21:01:58.185024 30933 leader_election.cc:304] T 00000000000000000000000000000000 P 8d6aede0d59c4dbd8c8bca7ade69344b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8d6aede0d59c4dbd8c8bca7ade69344b; no voters:
I20251215 21:01:58.185278 30933 leader_election.cc:290] T 00000000000000000000000000000000 P 8d6aede0d59c4dbd8c8bca7ade69344b [CANDIDATE]: Term 1 election: Requested vote from peers
I20251215 21:01:58.185361 30938 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 8d6aede0d59c4dbd8c8bca7ade69344b [term 1 FOLLOWER]: Leader election won for term 1
I20251215 21:01:58.185544 30938 raft_consensus.cc:697] T 00000000000000000000000000000000 P 8d6aede0d59c4dbd8c8bca7ade69344b [term 1 LEADER]: Becoming Leader. State: Replica: 8d6aede0d59c4dbd8c8bca7ade69344b, State: Running, Role: LEADER
I20251215 21:01:58.185884 30938 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8d6aede0d59c4dbd8c8bca7ade69344b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d6aede0d59c4dbd8c8bca7ade69344b" member_type: VOTER last_known_addr { host: "127.28.113.62" port: 37805 } }
I20251215 21:01:58.186369 30933 sys_catalog.cc:565] T 00000000000000000000000000000000 P 8d6aede0d59c4dbd8c8bca7ade69344b [sys.catalog]: configured and running, proceeding with master startup.
I20251215 21:01:58.187022 30938 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d6aede0d59c4dbd8c8bca7ade69344b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8d6aede0d59c4dbd8c8bca7ade69344b. Latest consensus state: current_term: 1 leader_uuid: "8d6aede0d59c4dbd8c8bca7ade69344b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d6aede0d59c4dbd8c8bca7ade69344b" member_type: VOTER last_known_addr { host: "127.28.113.62" port: 37805 } } }
I20251215 21:01:58.187124 30938 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d6aede0d59c4dbd8c8bca7ade69344b [sys.catalog]: This master's current role is: LEADER
I20251215 21:01:58.187129 30936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:01:58.176213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56852 (local address 127.28.113.62:37805)
1215 21:01:58.176554 (+ 341us) server_negotiation.cc:207] Beginning negotiation
1215 21:01:58.176563 (+ 9us) server_negotiation.cc:400] Waiting for connection header
1215 21:01:58.176602 (+ 39us) server_negotiation.cc:408] Connection header received
1215 21:01:58.177097 (+ 495us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1215 21:01:58.177113 (+ 16us) server_negotiation.cc:462] Received NEGOTIATE request from client
1215 21:01:58.177373 (+ 260us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1215 21:01:58.177621 (+ 248us) server_negotiation.cc:227] Negotiated authn=SASL
1215 21:01:58.178604 (+ 983us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:01:58.179333 (+ 729us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1215 21:01:58.179946 (+ 613us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:01:58.180220 (+ 274us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:01:58.182136 (+ 1916us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1215 21:01:58.182152 (+ 16us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1215 21:01:58.182161 (+ 9us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1215 21:01:58.182189 (+ 28us) server_negotiation.cc:923] calling sasl_server_start()
1215 21:01:58.184383 (+ 2194us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1215 21:01:58.184808 (+ 425us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1215 21:01:58.184812 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1215 21:01:58.184816 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
1215 21:01:58.184913 (+ 97us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1215 21:01:58.185124 (+ 211us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1215 21:01:58.185127 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1215 21:01:58.185129 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1215 21:01:58.185478 (+ 349us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1215 21:01:58.186161 (+ 683us) server_negotiation.cc:1036] Waiting for connection context
1215 21:01:58.186284 (+ 123us) server_negotiation.cc:300] Negotiation successful
1215 21:01:58.186486 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":222,"thread_start_us":100,"threads_started":1}
I20251215 21:01:58.187500 30939 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d6aede0d59c4dbd8c8bca7ade69344b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8d6aede0d59c4dbd8c8bca7ade69344b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d6aede0d59c4dbd8c8bca7ade69344b" member_type: VOTER last_known_addr { host: "127.28.113.62" port: 37805 } } }
I20251215 21:01:58.187650 30939 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d6aede0d59c4dbd8c8bca7ade69344b [sys.catalog]: This master's current role is: LEADER
I20251215 21:01:58.187776 30944 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251215 21:01:58.190446 30944 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251215 21:01:58.195076 30944 catalog_manager.cc:1357] Generated new cluster ID: 4b6c45f87b7e4a14a4c1caa41e8645fc
I20251215 21:01:58.195137 30944 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251215 21:01:58.210964 30944 catalog_manager.cc:1380] Generated new certificate authority record
I20251215 21:01:58.212212 30944 catalog_manager.cc:1514] Loading token signing keys...
I20251215 21:01:58.219991 30944 catalog_manager.cc:6027] T 00000000000000000000000000000000 P 8d6aede0d59c4dbd8c8bca7ade69344b: Generated new TSK 0
I20251215 21:01:58.220741 30944 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.113.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.113.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.113.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765832506145157-29124-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.28.113.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765832506145157-29124-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.28.113.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.28.113.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.28.113.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765832506145157-29124-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.28.113.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765832506145157-29124-0/krb5kdc/kudu.keytab.
I20251215 21:01:58.267902 29124 external_mini_cluster.cc:1366] Running /tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu
/tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765832506145157-29124-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765832506145157-29124-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765832506145157-29124-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765832506145157-29124-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.113.1:0
--local_ip_for_outbound_sockets=127.28.113.1
--webserver_interface=127.28.113.1
--webserver_port=0
--tserver_master_addrs=127.28.113.62:37805
--keytab_file=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765832506145157-29124-0/krb5kdc/kudu.keytab
--principal=kudu/127.28.113.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.28.113.20:37295
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765832506145157-29124-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765832506145157-29124-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765832506145157-29124-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251215 21:01:58.367771 30961 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251215 21:01:58.367986 30961 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251215 21:01:58.368048 30961 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251215 21:01:58.371119 30961 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251215 21:01:58.371168 30961 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251215 21:01:58.371219 30961 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20251215 21:01:58.371239 30961 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20251215 21:01:58.371276 30961 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.113.1
I20251215 21:01:58.375254 30961 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.113.20:37295
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765832506145157-29124-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765832506145157-29124-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765832506145157-29124-0/krb5kdc/kudu.keytab
--principal=kudu/127.28.113.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.113.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765832506145157-29124-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.113.1
--webserver_port=0
--webserver_require_spnego=true
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.28.113.62:37805
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30961
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.113.1
--log_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765832506145157-29124-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision dc741732bb6a40213e11a13869fad06a49a209a3
build type DEBUG
built by None at 15 Dec 2025 20:43:14 UTC on 24a791456cd2
build id 9530
I20251215 21:01:58.376241 30961 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251215 21:01:58.376992 30961 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251215 21:01:58.383056 30966 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251215 21:01:58.383059 30969 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251215 21:01:58.383209 30961 server_base.cc:1047] running on GCE node
W20251215 21:01:58.383080 30967 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251215 21:01:58.383688 30961 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251215 21:01:58.384210 30961 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251215 21:01:58.385349 30961 hybrid_clock.cc:648] HybridClock initialized: now 1765832518385341 us; error 29 us; skew 500 ppm
Dec 15 21:01:58 dist-test-slave-bjsf krb5kdc[30839](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765832518, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.113.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251215 21:01:58.387744 30961 init.cc:377] Logged in from keytab as kudu/127.28.113.1@KRBTEST.COM (short username kudu)
I20251215 21:01:58.388782 30961 webserver.cc:492] Webserver started at http://127.28.113.1:33439/ using document root <none> and password file <none>
I20251215 21:01:58.389281 30961 fs_manager.cc:362] Metadata directory not provided
I20251215 21:01:58.389324 30961 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251215 21:01:58.389473 30961 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251215 21:01:58.391048 30961 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765832506145157-29124-0/minicluster-data/ts-0/data/instance:
uuid: "6e18dca41b204cf0a3ebedd00ec07906"
format_stamp: "Formatted at 2025-12-15 21:01:58 on dist-test-slave-bjsf"
server_key: "5a570ca58baa83c543c9dc57e96b12c5"
server_key_iv: "3bced447b4d5453d88ce250a85f12855"
server_key_version: "encryptionkey@0"
I20251215 21:01:58.391444 30961 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765832506145157-29124-0/minicluster-data/ts-0/wal/instance:
uuid: "6e18dca41b204cf0a3ebedd00ec07906"
format_stamp: "Formatted at 2025-12-15 21:01:58 on dist-test-slave-bjsf"
server_key: "5a570ca58baa83c543c9dc57e96b12c5"
server_key_iv: "3bced447b4d5453d88ce250a85f12855"
server_key_version: "encryptionkey@0"
I20251215 21:01:58.394609 30961 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20251215 21:01:58.396754 30976 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251215 21:01:58.397743 30961 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20251215 21:01:58.397853 30961 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765832506145157-29124-0/minicluster-data/ts-0/data,/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765832506145157-29124-0/minicluster-data/ts-0/wal
uuid: "6e18dca41b204cf0a3ebedd00ec07906"
format_stamp: "Formatted at 2025-12-15 21:01:58 on dist-test-slave-bjsf"
server_key: "5a570ca58baa83c543c9dc57e96b12c5"
server_key_iv: "3bced447b4d5453d88ce250a85f12855"
server_key_version: "encryptionkey@0"
I20251215 21:01:58.397938 30961 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765832506145157-29124-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765832506145157-29124-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765832506145157-29124-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251215 21:01:58.406713 30961 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251215 21:01:58.409350 30961 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251215 21:01:58.409518 30961 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251215 21:01:58.410873 30961 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251215 21:01:58.410920 30961 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251215 21:01:58.410969 30961 ts_tablet_manager.cc:616] Registered 0 tablets
I20251215 21:01:58.411015 30961 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Dec 15 21:01:58 dist-test-slave-bjsf krb5kdc[30839](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765832518, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.113.1@KRBTEST.COM for kudu/127.28.113.62@KRBTEST.COM
I20251215 21:01:58.421378 30961 rpc_server.cc:307] RPC server started. Bound to: 127.28.113.1:33435
I20251215 21:01:58.421408 31091 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.28.113.1:33435 every 8 connection(s)
I20251215 21:01:58.422235 30936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:01:58.411441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.113.1:48813 (local address 127.28.113.62:37805)
1215 21:01:58.411585 (+ 144us) server_negotiation.cc:207] Beginning negotiation
1215 21:01:58.411590 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1215 21:01:58.412387 (+ 797us) server_negotiation.cc:408] Connection header received
1215 21:01:58.413230 (+ 843us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1215 21:01:58.413234 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1215 21:01:58.413291 (+ 57us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1215 21:01:58.413362 (+ 71us) server_negotiation.cc:227] Negotiated authn=SASL
1215 21:01:58.414769 (+ 1407us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:01:58.415343 (+ 574us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1215 21:01:58.415988 (+ 645us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:01:58.416169 (+ 181us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:01:58.418622 (+ 2453us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1215 21:01:58.418646 (+ 24us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1215 21:01:58.418649 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1215 21:01:58.418676 (+ 27us) server_negotiation.cc:923] calling sasl_server_start()
1215 21:01:58.420241 (+ 1565us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1215 21:01:58.420754 (+ 513us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1215 21:01:58.420757 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1215 21:01:58.420758 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1215 21:01:58.420802 (+ 44us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1215 21:01:58.421096 (+ 294us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1215 21:01:58.421100 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1215 21:01:58.421102 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1215 21:01:58.421301 (+ 199us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1215 21:01:58.421397 (+ 96us) server_negotiation.cc:1036] Waiting for connection context
1215 21:01:58.421940 (+ 543us) server_negotiation.cc:300] Negotiation successful
1215 21:01:58.422080 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20251215 21:01:58.422536 30961 server_base.cc:1179] Dumped server information to /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765832506145157-29124-0/minicluster-data/ts-0/data/info.pb
I20251215 21:01:58.422776 30985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:01:58.411770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.113.62:37805 (local address 127.28.113.1:48813)
1215 21:01:58.412267 (+ 497us) negotiation.cc:107] Waiting for socket to connect
1215 21:01:58.412301 (+ 34us) client_negotiation.cc:174] Beginning negotiation
1215 21:01:58.413046 (+ 745us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1215 21:01:58.413520 (+ 474us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1215 21:01:58.413534 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
1215 21:01:58.413995 (+ 461us) client_negotiation.cc:189] Negotiated authn=SASL
1215 21:01:58.414635 (+ 640us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:01:58.414649 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:01:58.415450 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1215 21:01:58.415454 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1215 21:01:58.415861 (+ 407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:01:58.415868 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:01:58.416093 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:01:58.416720 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1215 21:01:58.416755 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
1215 21:01:58.418473 (+ 1718us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1215 21:01:58.420349 (+ 1876us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1215 21:01:58.420354 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1215 21:01:58.420366 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
1215 21:01:58.420659 (+ 293us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1215 21:01:58.420882 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1215 21:01:58.420885 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1215 21:01:58.420887 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1215 21:01:58.420995 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1215 21:01:58.421444 (+ 449us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1215 21:01:58.421450 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1215 21:01:58.421700 (+ 250us) client_negotiation.cc:769] Sending connection context
1215 21:01:58.421872 (+ 172us) client_negotiation.cc:240] Negotiation successful
1215 21:01:58.422080 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":293,"thread_start_us":151,"threads_started":1}
I20251215 21:01:58.424037 31092 heartbeater.cc:344] Connected to a master server at 127.28.113.62:37805
I20251215 21:01:58.424188 31092 heartbeater.cc:461] Registering TS with master...
I20251215 21:01:58.424674 31092 heartbeater.cc:507] Master 127.28.113.62:37805 requested a full tablet report, sending...
I20251215 21:01:58.426155 30887 ts_manager.cc:194] Registered new tserver with Master: 6e18dca41b204cf0a3ebedd00ec07906 (127.28.113.1:33435)
I20251215 21:01:58.427570 30887 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.113.1@KRBTEST.COM'} at 127.28.113.1:48813
I20251215 21:01:58.431943 30936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:01:58.426014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.113.1:53225 (local address 127.28.113.62:37805)
1215 21:01:58.426140 (+ 126us) server_negotiation.cc:207] Beginning negotiation
1215 21:01:58.426144 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1215 21:01:58.426406 (+ 262us) server_negotiation.cc:408] Connection header received
1215 21:01:58.426519 (+ 113us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1215 21:01:58.426522 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1215 21:01:58.426569 (+ 47us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1215 21:01:58.426649 (+ 80us) server_negotiation.cc:227] Negotiated authn=SASL
1215 21:01:58.427295 (+ 646us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:01:58.427836 (+ 541us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1215 21:01:58.428504 (+ 668us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:01:58.428687 (+ 183us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:01:58.429504 (+ 817us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1215 21:01:58.429520 (+ 16us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1215 21:01:58.429522 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1215 21:01:58.429542 (+ 20us) server_negotiation.cc:923] calling sasl_server_start()
1215 21:01:58.430668 (+ 1126us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1215 21:01:58.431021 (+ 353us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1215 21:01:58.431024 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1215 21:01:58.431026 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1215 21:01:58.431068 (+ 42us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1215 21:01:58.431261 (+ 193us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1215 21:01:58.431264 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1215 21:01:58.431266 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1215 21:01:58.431399 (+ 133us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1215 21:01:58.431483 (+ 84us) server_negotiation.cc:1036] Waiting for connection context
1215 21:01:58.431701 (+ 218us) server_negotiation.cc:300] Negotiation successful
1215 21:01:58.431802 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20251215 21:01:58.431983 31098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:01:58.425927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.113.62:37805 (local address 127.28.113.1:53225)
1215 21:01:58.426289 (+ 362us) negotiation.cc:107] Waiting for socket to connect
1215 21:01:58.426303 (+ 14us) client_negotiation.cc:174] Beginning negotiation
1215 21:01:58.426405 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1215 21:01:58.426676 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1215 21:01:58.426679 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1215 21:01:58.426878 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
1215 21:01:58.427160 (+ 282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:01:58.427169 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:01:58.427944 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1215 21:01:58.427948 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1215 21:01:58.428397 (+ 449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:01:58.428405 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:01:58.428509 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:01:58.429108 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1215 21:01:58.429120 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
1215 21:01:58.429392 (+ 272us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1215 21:01:58.430768 (+ 1376us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1215 21:01:58.430772 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1215 21:01:58.430774 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1215 21:01:58.430938 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1215 21:01:58.431144 (+ 206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1215 21:01:58.431147 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1215 21:01:58.431149 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1215 21:01:58.431191 (+ 42us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1215 21:01:58.431482 (+ 291us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1215 21:01:58.431485 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1215 21:01:58.431559 (+ 74us) client_negotiation.cc:769] Sending connection context
1215 21:01:58.431643 (+ 84us) client_negotiation.cc:240] Negotiation successful
1215 21:01:58.431771 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":285,"thread_start_us":96,"threads_started":1}
I20251215 21:01:58.432574 29124 external_mini_cluster.cc:1428] Started /tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu as pid 30961
I20251215 21:01:58.432664 29124 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765832506145157-29124-0/minicluster-data/ts-0/wal/instance
I20251215 21:01:58.432855 29124 external_mini_cluster.cc:1468] Setting key 707d268fa180a9ef69e3f67dc34138ef
WARNING: no policy specified for kudu/127.28.113.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.113.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.113.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765832506145157-29124-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.28.113.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765832506145157-29124-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.28.113.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.28.113.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.28.113.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765832506145157-29124-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.28.113.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765832506145157-29124-0/krb5kdc/kudu.keytab.
I20251215 21:01:58.473063 29124 external_mini_cluster.cc:1366] Running /tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu
/tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765832506145157-29124-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765832506145157-29124-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765832506145157-29124-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765832506145157-29124-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.113.2:0
--local_ip_for_outbound_sockets=127.28.113.2
--webserver_interface=127.28.113.2
--webserver_port=0
--tserver_master_addrs=127.28.113.62:37805
--keytab_file=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765832506145157-29124-0/krb5kdc/kudu.keytab
--principal=kudu/127.28.113.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.28.113.20:37295
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765832506145157-29124-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765832506145157-29124-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765832506145157-29124-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251215 21:01:58.574995 31103 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251215 21:01:58.575209 31103 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251215 21:01:58.575282 31103 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251215 21:01:58.578460 31103 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251215 21:01:58.578524 31103 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251215 21:01:58.578588 31103 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20251215 21:01:58.578617 31103 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20251215 21:01:58.578681 31103 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.113.2
I20251215 21:01:58.583155 31103 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.113.20:37295
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765832506145157-29124-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765832506145157-29124-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765832506145157-29124-0/krb5kdc/kudu.keytab
--principal=kudu/127.28.113.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.113.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765832506145157-29124-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.113.2
--webserver_port=0
--webserver_require_spnego=true
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.28.113.62:37805
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31103
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.113.2
--log_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765832506145157-29124-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision dc741732bb6a40213e11a13869fad06a49a209a3
build type DEBUG
built by None at 15 Dec 2025 20:43:14 UTC on 24a791456cd2
build id 9530
I20251215 21:01:58.584218 31103 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251215 21:01:58.585033 31103 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251215 21:01:58.591248 31108 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251215 21:01:58.591269 31109 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251215 21:01:58.591329 31111 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251215 21:01:58.591419 31103 server_base.cc:1047] running on GCE node
I20251215 21:01:58.591830 31103 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251215 21:01:58.592360 31103 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251215 21:01:58.593495 31103 hybrid_clock.cc:648] HybridClock initialized: now 1765832518593485 us; error 31 us; skew 500 ppm
Dec 15 21:01:58 dist-test-slave-bjsf krb5kdc[30839](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765832518, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.113.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251215 21:01:58.595867 31103 init.cc:377] Logged in from keytab as kudu/127.28.113.2@KRBTEST.COM (short username kudu)
I20251215 21:01:58.596926 31103 webserver.cc:492] Webserver started at http://127.28.113.2:41099/ using document root <none> and password file <none>
I20251215 21:01:58.597514 31103 fs_manager.cc:362] Metadata directory not provided
I20251215 21:01:58.597573 31103 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251215 21:01:58.597762 31103 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251215 21:01:58.599378 31103 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765832506145157-29124-0/minicluster-data/ts-1/data/instance:
uuid: "9899e8262dee40858d8880500e8054b1"
format_stamp: "Formatted at 2025-12-15 21:01:58 on dist-test-slave-bjsf"
server_key: "4f36f05ea8f37c9690afcc5e9dac0041"
server_key_iv: "60fd36cbf1c37327bc9c6019662100cc"
server_key_version: "encryptionkey@0"
I20251215 21:01:58.599829 31103 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765832506145157-29124-0/minicluster-data/ts-1/wal/instance:
uuid: "9899e8262dee40858d8880500e8054b1"
format_stamp: "Formatted at 2025-12-15 21:01:58 on dist-test-slave-bjsf"
server_key: "4f36f05ea8f37c9690afcc5e9dac0041"
server_key_iv: "60fd36cbf1c37327bc9c6019662100cc"
server_key_version: "encryptionkey@0"
I20251215 21:01:58.603772 31103 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20251215 21:01:58.606281 31118 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251215 21:01:58.607306 31103 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20251215 21:01:58.607409 31103 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765832506145157-29124-0/minicluster-data/ts-1/data,/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765832506145157-29124-0/minicluster-data/ts-1/wal
uuid: "9899e8262dee40858d8880500e8054b1"
format_stamp: "Formatted at 2025-12-15 21:01:58 on dist-test-slave-bjsf"
server_key: "4f36f05ea8f37c9690afcc5e9dac0041"
server_key_iv: "60fd36cbf1c37327bc9c6019662100cc"
server_key_version: "encryptionkey@0"
I20251215 21:01:58.607496 31103 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765832506145157-29124-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765832506145157-29124-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765832506145157-29124-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251215 21:01:58.629484 31103 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251215 21:01:58.632105 31103 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251215 21:01:58.632266 31103 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251215 21:01:58.633678 31103 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251215 21:01:58.633729 31103 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251215 21:01:58.633781 31103 ts_tablet_manager.cc:616] Registered 0 tablets
I20251215 21:01:58.633808 31103 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Dec 15 21:01:58 dist-test-slave-bjsf krb5kdc[30839](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765832518, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.113.2@KRBTEST.COM for kudu/127.28.113.62@KRBTEST.COM
I20251215 21:01:58.643590 31103 rpc_server.cc:307] RPC server started. Bound to: 127.28.113.2:35261
I20251215 21:01:58.643807 31233 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.28.113.2:35261 every 8 connection(s)
I20251215 21:01:58.644526 31103 server_base.cc:1179] Dumped server information to /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765832506145157-29124-0/minicluster-data/ts-1/data/info.pb
I20251215 21:01:58.645641 30936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:01:58.634323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.113.2:34593 (local address 127.28.113.62:37805)
1215 21:01:58.634436 (+ 113us) server_negotiation.cc:207] Beginning negotiation
1215 21:01:58.634439 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1215 21:01:58.635305 (+ 866us) server_negotiation.cc:408] Connection header received
1215 21:01:58.636226 (+ 921us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1215 21:01:58.636230 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1215 21:01:58.636291 (+ 61us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1215 21:01:58.636386 (+ 95us) server_negotiation.cc:227] Negotiated authn=SASL
1215 21:01:58.637928 (+ 1542us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:01:58.638698 (+ 770us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1215 21:01:58.639482 (+ 784us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:01:58.639680 (+ 198us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:01:58.642237 (+ 2557us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1215 21:01:58.642259 (+ 22us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1215 21:01:58.642262 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1215 21:01:58.642292 (+ 30us) server_negotiation.cc:923] calling sasl_server_start()
1215 21:01:58.643894 (+ 1602us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1215 21:01:58.644321 (+ 427us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1215 21:01:58.644325 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1215 21:01:58.644327 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1215 21:01:58.644372 (+ 45us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1215 21:01:58.644607 (+ 235us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1215 21:01:58.644611 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1215 21:01:58.644613 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1215 21:01:58.644761 (+ 148us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1215 21:01:58.644841 (+ 80us) server_negotiation.cc:1036] Waiting for connection context
1215 21:01:58.645364 (+ 523us) server_negotiation.cc:300] Negotiation successful
1215 21:01:58.645478 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20251215 21:01:58.646204 31127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:01:58.634697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.113.62:37805 (local address 127.28.113.2:34593)
1215 21:01:58.635176 (+ 479us) negotiation.cc:107] Waiting for socket to connect
1215 21:01:58.635213 (+ 37us) client_negotiation.cc:174] Beginning negotiation
1215 21:01:58.636007 (+ 794us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1215 21:01:58.636541 (+ 534us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1215 21:01:58.636551 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
1215 21:01:58.637077 (+ 526us) client_negotiation.cc:189] Negotiated authn=SASL
1215 21:01:58.637744 (+ 667us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:01:58.637770 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:01:58.638801 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1215 21:01:58.638804 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1215 21:01:58.639362 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:01:58.639372 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:01:58.639593 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:01:58.640250 (+ 657us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1215 21:01:58.640282 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
1215 21:01:58.642026 (+ 1744us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1215 21:01:58.644017 (+ 1991us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1215 21:01:58.644021 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1215 21:01:58.644029 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
1215 21:01:58.644232 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1215 21:01:58.644446 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1215 21:01:58.644449 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1215 21:01:58.644451 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1215 21:01:58.644523 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1215 21:01:58.644869 (+ 346us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1215 21:01:58.644876 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1215 21:01:58.645135 (+ 259us) client_negotiation.cc:769] Sending connection context
1215 21:01:58.645318 (+ 183us) client_negotiation.cc:240] Negotiation successful
1215 21:01:58.645557 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":283,"thread_start_us":148,"threads_started":1}
I20251215 21:01:58.647527 29124 external_mini_cluster.cc:1428] Started /tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu as pid 31103
I20251215 21:01:58.647644 29124 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765832506145157-29124-0/minicluster-data/ts-1/wal/instance
I20251215 21:01:58.647898 29124 external_mini_cluster.cc:1468] Setting key 651cda7482d956bcba85e674b7862a6b
I20251215 21:01:58.648002 31234 heartbeater.cc:344] Connected to a master server at 127.28.113.62:37805
I20251215 21:01:58.648154 31234 heartbeater.cc:461] Registering TS with master...
I20251215 21:01:58.648708 31234 heartbeater.cc:507] Master 127.28.113.62:37805 requested a full tablet report, sending...
I20251215 21:01:58.649590 30886 ts_manager.cc:194] Registered new tserver with Master: 9899e8262dee40858d8880500e8054b1 (127.28.113.2:35261)
I20251215 21:01:58.650163 30886 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.113.2@KRBTEST.COM'} at 127.28.113.2:34593
I20251215 21:01:58.656492 30936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:01:58.650729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.113.2:53195 (local address 127.28.113.62:37805)
1215 21:01:58.650867 (+ 138us) server_negotiation.cc:207] Beginning negotiation
1215 21:01:58.650871 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1215 21:01:58.650886 (+ 15us) server_negotiation.cc:408] Connection header received
1215 21:01:58.650919 (+ 33us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1215 21:01:58.650922 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1215 21:01:58.650963 (+ 41us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1215 21:01:58.651037 (+ 74us) server_negotiation.cc:227] Negotiated authn=SASL
1215 21:01:58.651519 (+ 482us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:01:58.652181 (+ 662us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1215 21:01:58.652905 (+ 724us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:01:58.653032 (+ 127us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:01:58.653905 (+ 873us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1215 21:01:58.653919 (+ 14us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1215 21:01:58.653921 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1215 21:01:58.653942 (+ 21us) server_negotiation.cc:923] calling sasl_server_start()
1215 21:01:58.655164 (+ 1222us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1215 21:01:58.655559 (+ 395us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1215 21:01:58.655561 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1215 21:01:58.655563 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1215 21:01:58.655595 (+ 32us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1215 21:01:58.655801 (+ 206us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1215 21:01:58.655803 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1215 21:01:58.655804 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1215 21:01:58.655981 (+ 177us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1215 21:01:58.656060 (+ 79us) server_negotiation.cc:1036] Waiting for connection context
1215 21:01:58.656304 (+ 244us) server_negotiation.cc:300] Negotiation successful
1215 21:01:58.656387 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20251215 21:01:58.656554 31240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:01:58.649647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.113.62:37805 (local address 127.28.113.2:53195)
1215 21:01:58.649910 (+ 263us) negotiation.cc:107] Waiting for socket to connect
1215 21:01:58.649924 (+ 14us) client_negotiation.cc:174] Beginning negotiation
1215 21:01:58.650008 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1215 21:01:58.651068 (+ 1060us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1215 21:01:58.651070 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1215 21:01:58.651210 (+ 140us) client_negotiation.cc:189] Negotiated authn=SASL
1215 21:01:58.651413 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:01:58.651418 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:01:58.652279 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1215 21:01:58.652283 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1215 21:01:58.652810 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:01:58.652820 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:01:58.652924 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:01:58.653410 (+ 486us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1215 21:01:58.653423 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
1215 21:01:58.653798 (+ 375us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1215 21:01:58.655266 (+ 1468us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1215 21:01:58.655270 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1215 21:01:58.655272 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1215 21:01:58.655474 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1215 21:01:58.655664 (+ 190us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1215 21:01:58.655667 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1215 21:01:58.655668 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1215 21:01:58.655720 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1215 21:01:58.656064 (+ 344us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1215 21:01:58.656068 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1215 21:01:58.656159 (+ 91us) client_negotiation.cc:769] Sending connection context
1215 21:01:58.656263 (+ 104us) client_negotiation.cc:240] Negotiation successful
1215 21:01:58.656383 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":93,"threads_started":1}
WARNING: no policy specified for kudu/127.28.113.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.113.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.113.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765832506145157-29124-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.28.113.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765832506145157-29124-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.28.113.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.28.113.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.28.113.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765832506145157-29124-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.28.113.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765832506145157-29124-0/krb5kdc/kudu.keytab.
I20251215 21:01:58.691041 29124 external_mini_cluster.cc:1366] Running /tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu
/tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765832506145157-29124-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765832506145157-29124-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765832506145157-29124-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765832506145157-29124-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.113.3:0
--local_ip_for_outbound_sockets=127.28.113.3
--webserver_interface=127.28.113.3
--webserver_port=0
--tserver_master_addrs=127.28.113.62:37805
--keytab_file=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765832506145157-29124-0/krb5kdc/kudu.keytab
--principal=kudu/127.28.113.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.28.113.20:37295
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765832506145157-29124-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765832506145157-29124-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765832506145157-29124-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251215 21:01:58.795740 31245 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251215 21:01:58.795980 31245 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251215 21:01:58.796075 31245 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251215 21:01:58.799285 31245 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251215 21:01:58.799353 31245 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251215 21:01:58.799435 31245 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20251215 21:01:58.799464 31245 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20251215 21:01:58.799531 31245 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.113.3
I20251215 21:01:58.803709 31245 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.113.20:37295
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765832506145157-29124-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765832506145157-29124-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765832506145157-29124-0/krb5kdc/kudu.keytab
--principal=kudu/127.28.113.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.113.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765832506145157-29124-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.113.3
--webserver_port=0
--webserver_require_spnego=true
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.28.113.62:37805
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31245
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.113.3
--log_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765832506145157-29124-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision dc741732bb6a40213e11a13869fad06a49a209a3
build type DEBUG
built by None at 15 Dec 2025 20:43:14 UTC on 24a791456cd2
build id 9530
I20251215 21:01:58.804790 31245 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251215 21:01:58.805598 31245 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251215 21:01:58.811753 31251 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251215 21:01:58.811755 31250 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251215 21:01:58.811748 31253 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251215 21:01:58.812127 31245 server_base.cc:1047] running on GCE node
I20251215 21:01:58.812482 31245 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251215 21:01:58.812994 31245 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251215 21:01:58.814136 31245 hybrid_clock.cc:648] HybridClock initialized: now 1765832518814124 us; error 32 us; skew 500 ppm
Dec 15 21:01:58 dist-test-slave-bjsf krb5kdc[30839](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765832518, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.113.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251215 21:01:58.816593 31245 init.cc:377] Logged in from keytab as kudu/127.28.113.3@KRBTEST.COM (short username kudu)
I20251215 21:01:58.817550 31245 webserver.cc:492] Webserver started at http://127.28.113.3:42151/ using document root <none> and password file <none>
I20251215 21:01:58.818073 31245 fs_manager.cc:362] Metadata directory not provided
I20251215 21:01:58.818120 31245 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251215 21:01:58.818289 31245 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251215 21:01:58.819892 31245 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765832506145157-29124-0/minicluster-data/ts-2/data/instance:
uuid: "5755a662553c4a84b76c8b2c48880567"
format_stamp: "Formatted at 2025-12-15 21:01:58 on dist-test-slave-bjsf"
server_key: "6d2a5c1ffde6ffd6aab0fd4f90e11ec3"
server_key_iv: "d70a2a9248b1d3d2c776a77536e27f32"
server_key_version: "encryptionkey@0"
I20251215 21:01:58.820345 31245 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765832506145157-29124-0/minicluster-data/ts-2/wal/instance:
uuid: "5755a662553c4a84b76c8b2c48880567"
format_stamp: "Formatted at 2025-12-15 21:01:58 on dist-test-slave-bjsf"
server_key: "6d2a5c1ffde6ffd6aab0fd4f90e11ec3"
server_key_iv: "d70a2a9248b1d3d2c776a77536e27f32"
server_key_version: "encryptionkey@0"
I20251215 21:01:58.823493 31245 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20251215 21:01:58.825708 31260 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251215 21:01:58.826665 31245 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20251215 21:01:58.826762 31245 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765832506145157-29124-0/minicluster-data/ts-2/data,/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765832506145157-29124-0/minicluster-data/ts-2/wal
uuid: "5755a662553c4a84b76c8b2c48880567"
format_stamp: "Formatted at 2025-12-15 21:01:58 on dist-test-slave-bjsf"
server_key: "6d2a5c1ffde6ffd6aab0fd4f90e11ec3"
server_key_iv: "d70a2a9248b1d3d2c776a77536e27f32"
server_key_version: "encryptionkey@0"
I20251215 21:01:58.826841 31245 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765832506145157-29124-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765832506145157-29124-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765832506145157-29124-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251215 21:01:58.848829 31245 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251215 21:01:58.851630 31245 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251215 21:01:58.851804 31245 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251215 21:01:58.853232 31245 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251215 21:01:58.853282 31245 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251215 21:01:58.853335 31245 ts_tablet_manager.cc:616] Registered 0 tablets
I20251215 21:01:58.853361 31245 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Dec 15 21:01:58 dist-test-slave-bjsf krb5kdc[30839](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765832518, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.113.3@KRBTEST.COM for kudu/127.28.113.62@KRBTEST.COM
I20251215 21:01:58.863124 31245 rpc_server.cc:307] RPC server started. Bound to: 127.28.113.3:42057
I20251215 21:01:58.863190 31375 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.28.113.3:42057 every 8 connection(s)
I20251215 21:01:58.864077 31245 server_base.cc:1179] Dumped server information to /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765832506145157-29124-0/minicluster-data/ts-2/data/info.pb
I20251215 21:01:58.864710 30936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:01:58.853971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.113.3:53053 (local address 127.28.113.62:37805)
1215 21:01:58.854116 (+ 145us) server_negotiation.cc:207] Beginning negotiation
1215 21:01:58.854121 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1215 21:01:58.854806 (+ 685us) server_negotiation.cc:408] Connection header received
1215 21:01:58.855776 (+ 970us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1215 21:01:58.855781 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1215 21:01:58.855841 (+ 60us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1215 21:01:58.855897 (+ 56us) server_negotiation.cc:227] Negotiated authn=SASL
1215 21:01:58.857505 (+ 1608us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:01:58.858291 (+ 786us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1215 21:01:58.859045 (+ 754us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:01:58.859221 (+ 176us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:01:58.861262 (+ 2041us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1215 21:01:58.861287 (+ 25us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1215 21:01:58.861290 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1215 21:01:58.861321 (+ 31us) server_negotiation.cc:923] calling sasl_server_start()
1215 21:01:58.862863 (+ 1542us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1215 21:01:58.863362 (+ 499us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1215 21:01:58.863366 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1215 21:01:58.863367 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1215 21:01:58.863413 (+ 46us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1215 21:01:58.863662 (+ 249us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1215 21:01:58.863665 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1215 21:01:58.863667 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1215 21:01:58.863831 (+ 164us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1215 21:01:58.863913 (+ 82us) server_negotiation.cc:1036] Waiting for connection context
1215 21:01:58.864429 (+ 516us) server_negotiation.cc:300] Negotiation successful
1215 21:01:58.864557 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20251215 21:01:58.865307 31269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:01:58.854235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.113.62:37805 (local address 127.28.113.3:53053)
1215 21:01:58.854660 (+ 425us) negotiation.cc:107] Waiting for socket to connect
1215 21:01:58.854702 (+ 42us) client_negotiation.cc:174] Beginning negotiation
1215 21:01:58.855532 (+ 830us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1215 21:01:58.856286 (+ 754us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1215 21:01:58.856296 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
1215 21:01:58.856776 (+ 480us) client_negotiation.cc:189] Negotiated authn=SASL
1215 21:01:58.857370 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:01:58.857383 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:01:58.858424 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1215 21:01:58.858428 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1215 21:01:58.858928 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:01:58.858936 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:01:58.859153 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:01:58.859730 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1215 21:01:58.859756 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
1215 21:01:58.861115 (+ 1359us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1215 21:01:58.862964 (+ 1849us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1215 21:01:58.862969 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1215 21:01:58.862981 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
1215 21:01:58.863272 (+ 291us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1215 21:01:58.863489 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1215 21:01:58.863492 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1215 21:01:58.863494 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1215 21:01:58.863582 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1215 21:01:58.863926 (+ 344us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1215 21:01:58.863933 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1215 21:01:58.864186 (+ 253us) client_negotiation.cc:769] Sending connection context
1215 21:01:58.864381 (+ 195us) client_negotiation.cc:240] Negotiation successful
1215 21:01:58.864628 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":109,"threads_started":1}
I20251215 21:01:58.865658 29124 external_mini_cluster.cc:1428] Started /tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu as pid 31245
I20251215 21:01:58.865767 29124 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765832506145157-29124-0/minicluster-data/ts-2/wal/instance
I20251215 21:01:58.865998 29124 external_mini_cluster.cc:1468] Setting key 47007635d7ccd5fc809ad765bacb34e9
I20251215 21:01:58.866524 31376 heartbeater.cc:344] Connected to a master server at 127.28.113.62:37805
I20251215 21:01:58.866670 31376 heartbeater.cc:461] Registering TS with master...
I20251215 21:01:58.867142 31376 heartbeater.cc:507] Master 127.28.113.62:37805 requested a full tablet report, sending...
I20251215 21:01:58.868060 30887 ts_manager.cc:194] Registered new tserver with Master: 5755a662553c4a84b76c8b2c48880567 (127.28.113.3:42057)
I20251215 21:01:58.868741 30887 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.113.3@KRBTEST.COM'} at 127.28.113.3:53053
I20251215 21:01:58.869228 29124 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20251215 21:01:58.874503 30936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:01:58.868428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.113.3:58621 (local address 127.28.113.62:37805)
1215 21:01:58.868539 (+ 111us) server_negotiation.cc:207] Beginning negotiation
1215 21:01:58.868543 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1215 21:01:58.868730 (+ 187us) server_negotiation.cc:408] Connection header received
1215 21:01:58.868833 (+ 103us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1215 21:01:58.868836 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1215 21:01:58.868879 (+ 43us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1215 21:01:58.868944 (+ 65us) server_negotiation.cc:227] Negotiated authn=SASL
1215 21:01:58.869485 (+ 541us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:01:58.870162 (+ 677us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1215 21:01:58.870917 (+ 755us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:01:58.871072 (+ 155us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:01:58.871907 (+ 835us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1215 21:01:58.871921 (+ 14us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1215 21:01:58.871923 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1215 21:01:58.871944 (+ 21us) server_negotiation.cc:923] calling sasl_server_start()
1215 21:01:58.873184 (+ 1240us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1215 21:01:58.873593 (+ 409us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1215 21:01:58.873596 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1215 21:01:58.873598 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1215 21:01:58.873637 (+ 39us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1215 21:01:58.873852 (+ 215us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1215 21:01:58.873856 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1215 21:01:58.873857 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1215 21:01:58.873983 (+ 126us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1215 21:01:58.874063 (+ 80us) server_negotiation.cc:1036] Waiting for connection context
1215 21:01:58.874281 (+ 218us) server_negotiation.cc:300] Negotiation successful
1215 21:01:58.874373 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20251215 21:01:58.874521 31382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:01:58.868367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.113.62:37805 (local address 127.28.113.3:58621)
1215 21:01:58.868655 (+ 288us) negotiation.cc:107] Waiting for socket to connect
1215 21:01:58.868666 (+ 11us) client_negotiation.cc:174] Beginning negotiation
1215 21:01:58.868750 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1215 21:01:58.868958 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1215 21:01:58.868961 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1215 21:01:58.869120 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
1215 21:01:58.869362 (+ 242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:01:58.869371 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:01:58.870296 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1215 21:01:58.870300 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1215 21:01:58.870799 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:01:58.870808 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:01:58.870903 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:01:58.871529 (+ 626us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1215 21:01:58.871541 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
1215 21:01:58.871805 (+ 264us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1215 21:01:58.873287 (+ 1482us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1215 21:01:58.873291 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1215 21:01:58.873293 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1215 21:01:58.873495 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1215 21:01:58.873722 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1215 21:01:58.873725 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1215 21:01:58.873727 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1215 21:01:58.873771 (+ 44us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1215 21:01:58.874065 (+ 294us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1215 21:01:58.874068 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1215 21:01:58.874143 (+ 75us) client_negotiation.cc:769] Sending connection context
1215 21:01:58.874225 (+ 82us) client_negotiation.cc:240] Negotiation successful
1215 21:01:58.874352 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":88,"threads_started":1}
Dec 15 21:01:58 dist-test-slave-bjsf krb5kdc[30839](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765832518, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Dec 15 21:01:58 dist-test-slave-bjsf krb5kdc[30839](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765832518, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.113.62@KRBTEST.COM
I20251215 21:01:58.892424 30936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:01:58.885436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56856 (local address 127.28.113.62:37805)
1215 21:01:58.885566 (+ 130us) server_negotiation.cc:207] Beginning negotiation
1215 21:01:58.885570 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1215 21:01:58.885582 (+ 12us) server_negotiation.cc:408] Connection header received
1215 21:01:58.885621 (+ 39us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1215 21:01:58.885623 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1215 21:01:58.885663 (+ 40us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1215 21:01:58.885730 (+ 67us) server_negotiation.cc:227] Negotiated authn=SASL
1215 21:01:58.886552 (+ 822us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:01:58.887079 (+ 527us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1215 21:01:58.887702 (+ 623us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:01:58.887833 (+ 131us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:01:58.889892 (+ 2059us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1215 21:01:58.889911 (+ 19us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1215 21:01:58.889914 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1215 21:01:58.889940 (+ 26us) server_negotiation.cc:923] calling sasl_server_start()
1215 21:01:58.891210 (+ 1270us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1215 21:01:58.891547 (+ 337us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1215 21:01:58.891550 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1215 21:01:58.891551 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1215 21:01:58.891588 (+ 37us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1215 21:01:58.891741 (+ 153us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1215 21:01:58.891743 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1215 21:01:58.891744 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1215 21:01:58.891887 (+ 143us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1215 21:01:58.892022 (+ 135us) server_negotiation.cc:1036] Waiting for connection context
1215 21:01:58.892175 (+ 153us) server_negotiation.cc:300] Negotiation successful
1215 21:01:58.892268 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20251215 21:01:58.894878 30887 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56856:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251215 21:01:58.897188 30887 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251215 21:01:58.908910 31393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:01:58.904896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.113.2:35261 (local address 127.0.0.1:34108)
1215 21:01:58.905216 (+ 320us) negotiation.cc:107] Waiting for socket to connect
1215 21:01:58.905257 (+ 41us) client_negotiation.cc:174] Beginning negotiation
1215 21:01:58.905388 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1215 21:01:58.905927 (+ 539us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1215 21:01:58.905933 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
1215 21:01:58.905959 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1215 21:01:58.906271 (+ 312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:01:58.906281 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:01:58.907716 (+ 1435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1215 21:01:58.907719 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1215 21:01:58.908408 (+ 689us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:01:58.908417 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:01:58.908522 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:01:58.908579 (+ 57us) client_negotiation.cc:769] Sending connection context
1215 21:01:58.908666 (+ 87us) client_negotiation.cc:240] Negotiation successful
1215 21:01:58.908754 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":80,"threads_started":1}
I20251215 21:01:58.909402 31394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:01:58.905065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34108 (local address 127.28.113.2:35261)
1215 21:01:58.905422 (+ 357us) server_negotiation.cc:207] Beginning negotiation
1215 21:01:58.905428 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1215 21:01:58.905452 (+ 24us) server_negotiation.cc:408] Connection header received
1215 21:01:58.905539 (+ 87us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1215 21:01:58.905545 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
1215 21:01:58.905736 (+ 191us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1215 21:01:58.906423 (+ 687us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1215 21:01:58.906581 (+ 158us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:01:58.907602 (+ 1021us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1215 21:01:58.908535 (+ 933us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:01:58.909010 (+ 475us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:01:58.909121 (+ 111us) server_negotiation.cc:1036] Waiting for connection context
1215 21:01:58.909179 (+ 58us) server_negotiation.cc:300] Negotiation successful
1215 21:01:58.909258 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":269,"thread_start_us":104,"threads_started":1}
I20251215 21:01:58.909762 31395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:01:58.905329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.113.3:42057 (local address 127.0.0.1:40368)
1215 21:01:58.905752 (+ 423us) negotiation.cc:107] Waiting for socket to connect
1215 21:01:58.905764 (+ 12us) client_negotiation.cc:174] Beginning negotiation
1215 21:01:58.905844 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1215 21:01:58.907041 (+ 1197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1215 21:01:58.907044 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1215 21:01:58.907062 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1215 21:01:58.907285 (+ 223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:01:58.907292 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:01:58.908652 (+ 1360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1215 21:01:58.908655 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1215 21:01:58.909449 (+ 794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:01:58.909458 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:01:58.909528 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:01:58.909543 (+ 15us) client_negotiation.cc:769] Sending connection context
1215 21:01:58.909580 (+ 37us) client_negotiation.cc:240] Negotiation successful
1215 21:01:58.909646 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":352,"thread_start_us":78,"threads_started":1}
I20251215 21:01:58.910586 31396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:01:58.905591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.113.1:33435 (local address 127.0.0.1:45748)
1215 21:01:58.906458 (+ 867us) negotiation.cc:107] Waiting for socket to connect
1215 21:01:58.906473 (+ 15us) client_negotiation.cc:174] Beginning negotiation
1215 21:01:58.906562 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1215 21:01:58.907164 (+ 602us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1215 21:01:58.907167 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1215 21:01:58.907186 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1215 21:01:58.907416 (+ 230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:01:58.907421 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:01:58.908776 (+ 1355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1215 21:01:58.908779 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1215 21:01:58.910268 (+ 1489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:01:58.910277 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:01:58.910375 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:01:58.910390 (+ 15us) client_negotiation.cc:769] Sending connection context
1215 21:01:58.910436 (+ 46us) client_negotiation.cc:240] Negotiation successful
1215 21:01:58.910484 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":805,"thread_start_us":79,"threads_started":1}
I20251215 21:01:58.910893 31397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:01:58.905692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40368 (local address 127.28.113.3:42057)
1215 21:01:58.906636 (+ 944us) server_negotiation.cc:207] Beginning negotiation
1215 21:01:58.906642 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1215 21:01:58.906663 (+ 21us) server_negotiation.cc:408] Connection header received
1215 21:01:58.906744 (+ 81us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1215 21:01:58.906749 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1215 21:01:58.906925 (+ 176us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1215 21:01:58.907057 (+ 132us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1215 21:01:58.907385 (+ 328us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:01:58.908516 (+ 1131us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1215 21:01:58.910059 (+ 1543us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:01:58.910530 (+ 471us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:01:58.910640 (+ 110us) server_negotiation.cc:1036] Waiting for connection context
1215 21:01:58.910704 (+ 64us) server_negotiation.cc:300] Negotiation successful
1215 21:01:58.910782 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":838,"thread_start_us":107,"threads_started":1}
I20251215 21:01:58.911363 31398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:01:58.905995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45748 (local address 127.28.113.1:33435)
1215 21:01:58.906264 (+ 269us) server_negotiation.cc:207] Beginning negotiation
1215 21:01:58.906269 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1215 21:01:58.906649 (+ 380us) server_negotiation.cc:408] Connection header received
1215 21:01:58.906739 (+ 90us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1215 21:01:58.906744 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1215 21:01:58.906894 (+ 150us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1215 21:01:58.907012 (+ 118us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1215 21:01:58.907527 (+ 515us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:01:58.908652 (+ 1125us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1215 21:01:58.910421 (+ 1769us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:01:58.910926 (+ 505us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:01:58.911015 (+ 89us) server_negotiation.cc:1036] Waiting for connection context
1215 21:01:58.911073 (+ 58us) server_negotiation.cc:300] Negotiation successful
1215 21:01:58.911234 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":184,"thread_start_us":97,"threads_started":1}
I20251215 21:01:58.912242 31168 tablet_service.cc:1505] Processing CreateTablet for tablet a24de5579422439992ad5b94ba3db3c2 (DEFAULT_TABLE table=test-table [id=f68a68024bae4b39990c0e16c60f12f3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251215 21:01:58.913249 31168 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a24de5579422439992ad5b94ba3db3c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 21:01:58.913090 31310 tablet_service.cc:1505] Processing CreateTablet for tablet a24de5579422439992ad5b94ba3db3c2 (DEFAULT_TABLE table=test-table [id=f68a68024bae4b39990c0e16c60f12f3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251215 21:01:58.913760 31310 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a24de5579422439992ad5b94ba3db3c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 21:01:58.913873 31024 tablet_service.cc:1505] Processing CreateTablet for tablet a24de5579422439992ad5b94ba3db3c2 (DEFAULT_TABLE table=test-table [id=f68a68024bae4b39990c0e16c60f12f3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251215 21:01:58.914655 31024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a24de5579422439992ad5b94ba3db3c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 21:01:58.918805 31399 tablet_bootstrap.cc:492] T a24de5579422439992ad5b94ba3db3c2 P 5755a662553c4a84b76c8b2c48880567: Bootstrap starting.
I20251215 21:01:58.919977 31400 tablet_bootstrap.cc:492] T a24de5579422439992ad5b94ba3db3c2 P 9899e8262dee40858d8880500e8054b1: Bootstrap starting.
I20251215 21:01:58.920928 31399 tablet_bootstrap.cc:654] T a24de5579422439992ad5b94ba3db3c2 P 5755a662553c4a84b76c8b2c48880567: Neither blocks nor log segments found. Creating new log.
I20251215 21:01:58.921655 31400 tablet_bootstrap.cc:654] T a24de5579422439992ad5b94ba3db3c2 P 9899e8262dee40858d8880500e8054b1: Neither blocks nor log segments found. Creating new log.
I20251215 21:01:58.921901 31399 log.cc:826] T a24de5579422439992ad5b94ba3db3c2 P 5755a662553c4a84b76c8b2c48880567: Log is configured to *not* fsync() on all Append() calls
I20251215 21:01:58.922482 31401 tablet_bootstrap.cc:492] T a24de5579422439992ad5b94ba3db3c2 P 6e18dca41b204cf0a3ebedd00ec07906: Bootstrap starting.
I20251215 21:01:58.922686 31400 log.cc:826] T a24de5579422439992ad5b94ba3db3c2 P 9899e8262dee40858d8880500e8054b1: Log is configured to *not* fsync() on all Append() calls
I20251215 21:01:58.923501 31399 tablet_bootstrap.cc:492] T a24de5579422439992ad5b94ba3db3c2 P 5755a662553c4a84b76c8b2c48880567: No bootstrap required, opened a new log
I20251215 21:01:58.923668 31399 ts_tablet_manager.cc:1403] T a24de5579422439992ad5b94ba3db3c2 P 5755a662553c4a84b76c8b2c48880567: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20251215 21:01:58.924182 31400 tablet_bootstrap.cc:492] T a24de5579422439992ad5b94ba3db3c2 P 9899e8262dee40858d8880500e8054b1: No bootstrap required, opened a new log
I20251215 21:01:58.924347 31400 ts_tablet_manager.cc:1403] T a24de5579422439992ad5b94ba3db3c2 P 9899e8262dee40858d8880500e8054b1: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20251215 21:01:58.926446 31401 tablet_bootstrap.cc:654] T a24de5579422439992ad5b94ba3db3c2 P 6e18dca41b204cf0a3ebedd00ec07906: Neither blocks nor log segments found. Creating new log.
I20251215 21:01:58.926672 31400 raft_consensus.cc:359] T a24de5579422439992ad5b94ba3db3c2 P 9899e8262dee40858d8880500e8054b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9899e8262dee40858d8880500e8054b1" member_type: VOTER last_known_addr { host: "127.28.113.2" port: 35261 } } peers { permanent_uuid: "6e18dca41b204cf0a3ebedd00ec07906" member_type: VOTER last_known_addr { host: "127.28.113.1" port: 33435 } } peers { permanent_uuid: "5755a662553c4a84b76c8b2c48880567" member_type: VOTER last_known_addr { host: "127.28.113.3" port: 42057 } }
I20251215 21:01:58.926656 31399 raft_consensus.cc:359] T a24de5579422439992ad5b94ba3db3c2 P 5755a662553c4a84b76c8b2c48880567 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9899e8262dee40858d8880500e8054b1" member_type: VOTER last_known_addr { host: "127.28.113.2" port: 35261 } } peers { permanent_uuid: "6e18dca41b204cf0a3ebedd00ec07906" member_type: VOTER last_known_addr { host: "127.28.113.1" port: 33435 } } peers { permanent_uuid: "5755a662553c4a84b76c8b2c48880567" member_type: VOTER last_known_addr { host: "127.28.113.3" port: 42057 } }
I20251215 21:01:58.926851 31399 raft_consensus.cc:385] T a24de5579422439992ad5b94ba3db3c2 P 5755a662553c4a84b76c8b2c48880567 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251215 21:01:58.926851 31400 raft_consensus.cc:385] T a24de5579422439992ad5b94ba3db3c2 P 9899e8262dee40858d8880500e8054b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251215 21:01:58.926909 31400 raft_consensus.cc:740] T a24de5579422439992ad5b94ba3db3c2 P 9899e8262dee40858d8880500e8054b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9899e8262dee40858d8880500e8054b1, State: Initialized, Role: FOLLOWER
I20251215 21:01:58.926936 31399 raft_consensus.cc:740] T a24de5579422439992ad5b94ba3db3c2 P 5755a662553c4a84b76c8b2c48880567 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5755a662553c4a84b76c8b2c48880567, State: Initialized, Role: FOLLOWER
I20251215 21:01:58.927218 31401 log.cc:826] T a24de5579422439992ad5b94ba3db3c2 P 6e18dca41b204cf0a3ebedd00ec07906: Log is configured to *not* fsync() on all Append() calls
I20251215 21:01:58.927321 31400 consensus_queue.cc:260] T a24de5579422439992ad5b94ba3db3c2 P 9899e8262dee40858d8880500e8054b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9899e8262dee40858d8880500e8054b1" member_type: VOTER last_known_addr { host: "127.28.113.2" port: 35261 } } peers { permanent_uuid: "6e18dca41b204cf0a3ebedd00ec07906" member_type: VOTER last_known_addr { host: "127.28.113.1" port: 33435 } } peers { permanent_uuid: "5755a662553c4a84b76c8b2c48880567" member_type: VOTER last_known_addr { host: "127.28.113.3" port: 42057 } }
I20251215 21:01:58.927362 31399 consensus_queue.cc:260] T a24de5579422439992ad5b94ba3db3c2 P 5755a662553c4a84b76c8b2c48880567 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9899e8262dee40858d8880500e8054b1" member_type: VOTER last_known_addr { host: "127.28.113.2" port: 35261 } } peers { permanent_uuid: "6e18dca41b204cf0a3ebedd00ec07906" member_type: VOTER last_known_addr { host: "127.28.113.1" port: 33435 } } peers { permanent_uuid: "5755a662553c4a84b76c8b2c48880567" member_type: VOTER last_known_addr { host: "127.28.113.3" port: 42057 } }
I20251215 21:01:58.928030 31376 heartbeater.cc:499] Master 127.28.113.62:37805 was elected leader, sending a full tablet report...
I20251215 21:01:58.928182 31400 ts_tablet_manager.cc:1434] T a24de5579422439992ad5b94ba3db3c2 P 9899e8262dee40858d8880500e8054b1: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20251215 21:01:58.928310 31234 heartbeater.cc:499] Master 127.28.113.62:37805 was elected leader, sending a full tablet report...
I20251215 21:01:58.928176 31399 ts_tablet_manager.cc:1434] T a24de5579422439992ad5b94ba3db3c2 P 5755a662553c4a84b76c8b2c48880567: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20251215 21:01:58.930020 31401 tablet_bootstrap.cc:492] T a24de5579422439992ad5b94ba3db3c2 P 6e18dca41b204cf0a3ebedd00ec07906: No bootstrap required, opened a new log
I20251215 21:01:58.930194 31401 ts_tablet_manager.cc:1403] T a24de5579422439992ad5b94ba3db3c2 P 6e18dca41b204cf0a3ebedd00ec07906: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.000s
I20251215 21:01:58.933218 31401 raft_consensus.cc:359] T a24de5579422439992ad5b94ba3db3c2 P 6e18dca41b204cf0a3ebedd00ec07906 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9899e8262dee40858d8880500e8054b1" member_type: VOTER last_known_addr { host: "127.28.113.2" port: 35261 } } peers { permanent_uuid: "6e18dca41b204cf0a3ebedd00ec07906" member_type: VOTER last_known_addr { host: "127.28.113.1" port: 33435 } } peers { permanent_uuid: "5755a662553c4a84b76c8b2c48880567" member_type: VOTER last_known_addr { host: "127.28.113.3" port: 42057 } }
I20251215 21:01:58.933396 31401 raft_consensus.cc:385] T a24de5579422439992ad5b94ba3db3c2 P 6e18dca41b204cf0a3ebedd00ec07906 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251215 21:01:58.933451 31401 raft_consensus.cc:740] T a24de5579422439992ad5b94ba3db3c2 P 6e18dca41b204cf0a3ebedd00ec07906 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e18dca41b204cf0a3ebedd00ec07906, State: Initialized, Role: FOLLOWER
I20251215 21:01:58.933808 31401 consensus_queue.cc:260] T a24de5579422439992ad5b94ba3db3c2 P 6e18dca41b204cf0a3ebedd00ec07906 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9899e8262dee40858d8880500e8054b1" member_type: VOTER last_known_addr { host: "127.28.113.2" port: 35261 } } peers { permanent_uuid: "6e18dca41b204cf0a3ebedd00ec07906" member_type: VOTER last_known_addr { host: "127.28.113.1" port: 33435 } } peers { permanent_uuid: "5755a662553c4a84b76c8b2c48880567" member_type: VOTER last_known_addr { host: "127.28.113.3" port: 42057 } }
I20251215 21:01:58.934361 31092 heartbeater.cc:499] Master 127.28.113.62:37805 was elected leader, sending a full tablet report...
I20251215 21:01:58.934571 31401 ts_tablet_manager.cc:1434] T a24de5579422439992ad5b94ba3db3c2 P 6e18dca41b204cf0a3ebedd00ec07906: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20251215 21:01:58.976605 31406 raft_consensus.cc:493] T a24de5579422439992ad5b94ba3db3c2 P 5755a662553c4a84b76c8b2c48880567 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 21:01:58.976778 31406 raft_consensus.cc:515] T a24de5579422439992ad5b94ba3db3c2 P 5755a662553c4a84b76c8b2c48880567 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9899e8262dee40858d8880500e8054b1" member_type: VOTER last_known_addr { host: "127.28.113.2" port: 35261 } } peers { permanent_uuid: "6e18dca41b204cf0a3ebedd00ec07906" member_type: VOTER last_known_addr { host: "127.28.113.1" port: 33435 } } peers { permanent_uuid: "5755a662553c4a84b76c8b2c48880567" member_type: VOTER last_known_addr { host: "127.28.113.3" port: 42057 } }
I20251215 21:01:58.977777 31406 leader_election.cc:290] T a24de5579422439992ad5b94ba3db3c2 P 5755a662553c4a84b76c8b2c48880567 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9899e8262dee40858d8880500e8054b1 (127.28.113.2:35261), 6e18dca41b204cf0a3ebedd00ec07906 (127.28.113.1:33435)
I20251215 21:01:58.980516 31269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:01:58.978005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.113.2:35261 (local address 127.28.113.3:54503)
1215 21:01:58.978143 (+ 138us) negotiation.cc:107] Waiting for socket to connect
1215 21:01:58.978159 (+ 16us) client_negotiation.cc:174] Beginning negotiation
1215 21:01:58.978259 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1215 21:01:58.978557 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1215 21:01:58.978560 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1215 21:01:58.978587 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1215 21:01:58.978794 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:01:58.978800 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:01:58.979531 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1215 21:01:58.979533 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1215 21:01:58.980145 (+ 612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:01:58.980153 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:01:58.980252 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:01:58.980275 (+ 23us) client_negotiation.cc:769] Sending connection context
1215 21:01:58.980323 (+ 48us) client_negotiation.cc:240] Negotiation successful
1215 21:01:58.980375 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47}
I20251215 21:01:58.980753 31408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:01:58.978256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.113.1:33435 (local address 127.28.113.3:40079)
1215 21:01:58.978588 (+ 332us) negotiation.cc:107] Waiting for socket to connect
1215 21:01:58.978598 (+ 10us) client_negotiation.cc:174] Beginning negotiation
1215 21:01:58.978683 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1215 21:01:58.978892 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1215 21:01:58.978895 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1215 21:01:58.978910 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1215 21:01:58.979138 (+ 228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:01:58.979144 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:01:58.979817 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1215 21:01:58.979820 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1215 21:01:58.980405 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:01:58.980412 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:01:58.980509 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:01:58.980529 (+ 20us) client_negotiation.cc:769] Sending connection context
1215 21:01:58.980575 (+ 46us) client_negotiation.cc:240] Negotiation successful
1215 21:01:58.980624 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":137,"threads_started":1}
I20251215 21:01:58.980955 31394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:01:58.978096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.113.3:54503 (local address 127.28.113.2:35261)
1215 21:01:58.978222 (+ 126us) server_negotiation.cc:207] Beginning negotiation
1215 21:01:58.978226 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1215 21:01:58.978241 (+ 15us) server_negotiation.cc:408] Connection header received
1215 21:01:58.978395 (+ 154us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1215 21:01:58.978399 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1215 21:01:58.978466 (+ 67us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1215 21:01:58.978545 (+ 79us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1215 21:01:58.978899 (+ 354us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:01:58.979403 (+ 504us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1215 21:01:58.980251 (+ 848us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:01:58.980697 (+ 446us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:01:58.980735 (+ 38us) server_negotiation.cc:1036] Waiting for connection context
1215 21:01:58.980792 (+ 57us) server_negotiation.cc:300] Negotiation successful
1215 21:01:58.980845 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20251215 21:01:58.981178 31398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:01:58.978391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.113.3:40079 (local address 127.28.113.1:33435)
1215 21:01:58.978521 (+ 130us) server_negotiation.cc:207] Beginning negotiation
1215 21:01:58.978525 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1215 21:01:58.978660 (+ 135us) server_negotiation.cc:408] Connection header received
1215 21:01:58.978763 (+ 103us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1215 21:01:58.978766 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1215 21:01:58.978808 (+ 42us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1215 21:01:58.978868 (+ 60us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1215 21:01:58.979242 (+ 374us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:01:58.979715 (+ 473us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1215 21:01:58.980517 (+ 802us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:01:58.980958 (+ 441us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:01:58.980995 (+ 37us) server_negotiation.cc:1036] Waiting for connection context
1215 21:01:58.981042 (+ 47us) server_negotiation.cc:300] Negotiation successful
1215 21:01:58.981096 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20251215 21:01:58.981500 31176 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a24de5579422439992ad5b94ba3db3c2" candidate_uuid: "5755a662553c4a84b76c8b2c48880567" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9899e8262dee40858d8880500e8054b1" is_pre_election: true
I20251215 21:01:58.981637 31045 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a24de5579422439992ad5b94ba3db3c2" candidate_uuid: "5755a662553c4a84b76c8b2c48880567" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e18dca41b204cf0a3ebedd00ec07906" is_pre_election: true
I20251215 21:01:58.981746 31176 raft_consensus.cc:2468] T a24de5579422439992ad5b94ba3db3c2 P 9899e8262dee40858d8880500e8054b1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5755a662553c4a84b76c8b2c48880567 in term 0.
I20251215 21:01:58.981868 31045 raft_consensus.cc:2468] T a24de5579422439992ad5b94ba3db3c2 P 6e18dca41b204cf0a3ebedd00ec07906 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5755a662553c4a84b76c8b2c48880567 in term 0.
I20251215 21:01:58.982146 31262 leader_election.cc:304] T a24de5579422439992ad5b94ba3db3c2 P 5755a662553c4a84b76c8b2c48880567 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5755a662553c4a84b76c8b2c48880567, 9899e8262dee40858d8880500e8054b1; no voters:
I20251215 21:01:58.982343 31406 raft_consensus.cc:2804] T a24de5579422439992ad5b94ba3db3c2 P 5755a662553c4a84b76c8b2c48880567 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251215 21:01:58.982425 31406 raft_consensus.cc:493] T a24de5579422439992ad5b94ba3db3c2 P 5755a662553c4a84b76c8b2c48880567 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 21:01:58.982473 31406 raft_consensus.cc:3060] T a24de5579422439992ad5b94ba3db3c2 P 5755a662553c4a84b76c8b2c48880567 [term 0 FOLLOWER]: Advancing to term 1
I20251215 21:01:58.983408 31406 raft_consensus.cc:515] T a24de5579422439992ad5b94ba3db3c2 P 5755a662553c4a84b76c8b2c48880567 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9899e8262dee40858d8880500e8054b1" member_type: VOTER last_known_addr { host: "127.28.113.2" port: 35261 } } peers { permanent_uuid: "6e18dca41b204cf0a3ebedd00ec07906" member_type: VOTER last_known_addr { host: "127.28.113.1" port: 33435 } } peers { permanent_uuid: "5755a662553c4a84b76c8b2c48880567" member_type: VOTER last_known_addr { host: "127.28.113.3" port: 42057 } }
I20251215 21:01:58.983718 31406 leader_election.cc:290] T a24de5579422439992ad5b94ba3db3c2 P 5755a662553c4a84b76c8b2c48880567 [CANDIDATE]: Term 1 election: Requested vote from peers 9899e8262dee40858d8880500e8054b1 (127.28.113.2:35261), 6e18dca41b204cf0a3ebedd00ec07906 (127.28.113.1:33435)
I20251215 21:01:58.984076 31176 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a24de5579422439992ad5b94ba3db3c2" candidate_uuid: "5755a662553c4a84b76c8b2c48880567" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9899e8262dee40858d8880500e8054b1"
I20251215 21:01:58.984129 31045 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a24de5579422439992ad5b94ba3db3c2" candidate_uuid: "5755a662553c4a84b76c8b2c48880567" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e18dca41b204cf0a3ebedd00ec07906"
I20251215 21:01:58.984194 31176 raft_consensus.cc:3060] T a24de5579422439992ad5b94ba3db3c2 P 9899e8262dee40858d8880500e8054b1 [term 0 FOLLOWER]: Advancing to term 1
I20251215 21:01:58.984251 31045 raft_consensus.cc:3060] T a24de5579422439992ad5b94ba3db3c2 P 6e18dca41b204cf0a3ebedd00ec07906 [term 0 FOLLOWER]: Advancing to term 1
I20251215 21:01:58.985271 31176 raft_consensus.cc:2468] T a24de5579422439992ad5b94ba3db3c2 P 9899e8262dee40858d8880500e8054b1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5755a662553c4a84b76c8b2c48880567 in term 1.
I20251215 21:01:58.985287 31045 raft_consensus.cc:2468] T a24de5579422439992ad5b94ba3db3c2 P 6e18dca41b204cf0a3ebedd00ec07906 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5755a662553c4a84b76c8b2c48880567 in term 1.
I20251215 21:01:58.985538 31262 leader_election.cc:304] T a24de5579422439992ad5b94ba3db3c2 P 5755a662553c4a84b76c8b2c48880567 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5755a662553c4a84b76c8b2c48880567, 9899e8262dee40858d8880500e8054b1; no voters:
I20251215 21:01:58.985747 31406 raft_consensus.cc:2804] T a24de5579422439992ad5b94ba3db3c2 P 5755a662553c4a84b76c8b2c48880567 [term 1 FOLLOWER]: Leader election won for term 1
I20251215 21:01:58.985949 31406 raft_consensus.cc:697] T a24de5579422439992ad5b94ba3db3c2 P 5755a662553c4a84b76c8b2c48880567 [term 1 LEADER]: Becoming Leader. State: Replica: 5755a662553c4a84b76c8b2c48880567, State: Running, Role: LEADER
I20251215 21:01:58.986222 31406 consensus_queue.cc:237] T a24de5579422439992ad5b94ba3db3c2 P 5755a662553c4a84b76c8b2c48880567 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9899e8262dee40858d8880500e8054b1" member_type: VOTER last_known_addr { host: "127.28.113.2" port: 35261 } } peers { permanent_uuid: "6e18dca41b204cf0a3ebedd00ec07906" member_type: VOTER last_known_addr { host: "127.28.113.1" port: 33435 } } peers { permanent_uuid: "5755a662553c4a84b76c8b2c48880567" member_type: VOTER last_known_addr { host: "127.28.113.3" port: 42057 } }
I20251215 21:01:58.989279 30886 catalog_manager.cc:5654] T a24de5579422439992ad5b94ba3db3c2 P 5755a662553c4a84b76c8b2c48880567 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5755a662553c4a84b76c8b2c48880567 (127.28.113.3). New cstate: current_term: 1 leader_uuid: "5755a662553c4a84b76c8b2c48880567" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9899e8262dee40858d8880500e8054b1" member_type: VOTER last_known_addr { host: "127.28.113.2" port: 35261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6e18dca41b204cf0a3ebedd00ec07906" member_type: VOTER last_known_addr { host: "127.28.113.1" port: 33435 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5755a662553c4a84b76c8b2c48880567" member_type: VOTER last_known_addr { host: "127.28.113.3" port: 42057 } health_report { overall_health: HEALTHY } } }
Dec 15 21:01:59 dist-test-slave-bjsf krb5kdc[30839](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765832518, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.113.62@KRBTEST.COM for kudu/127.28.113.62@KRBTEST.COM
I20251215 21:01:59.028642 31416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:01:59.021585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.113.62:37805 (local address 127.0.0.1:56872)
1215 21:01:59.021867 (+ 282us) negotiation.cc:107] Waiting for socket to connect
1215 21:01:59.021881 (+ 14us) client_negotiation.cc:174] Beginning negotiation
1215 21:01:59.021983 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1215 21:01:59.022193 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1215 21:01:59.022196 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1215 21:01:59.022638 (+ 442us) client_negotiation.cc:189] Negotiated authn=SASL
1215 21:01:59.022844 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:01:59.022851 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:01:59.023489 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1215 21:01:59.023491 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1215 21:01:59.023868 (+ 377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:01:59.023874 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:01:59.023988 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:01:59.024515 (+ 527us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1215 21:01:59.024528 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
1215 21:01:59.025937 (+ 1409us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1215 21:01:59.027451 (+ 1514us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1215 21:01:59.027457 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1215 21:01:59.027462 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
1215 21:01:59.027661 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1215 21:01:59.027831 (+ 170us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1215 21:01:59.027833 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1215 21:01:59.027834 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1215 21:01:59.027874 (+ 40us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1215 21:01:59.028180 (+ 306us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1215 21:01:59.028185 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1215 21:01:59.028282 (+ 97us) client_negotiation.cc:769] Sending connection context
1215 21:01:59.028370 (+ 88us) client_negotiation.cc:240] Negotiation successful
1215 21:01:59.028480 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":108,"threads_started":1}
I20251215 21:01:59.028662 30936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:01:59.021623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56872 (local address 127.28.113.62:37805)
1215 21:01:59.021749 (+ 126us) server_negotiation.cc:207] Beginning negotiation
1215 21:01:59.021752 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1215 21:01:59.021943 (+ 191us) server_negotiation.cc:408] Connection header received
1215 21:01:59.022053 (+ 110us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1215 21:01:59.022056 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1215 21:01:59.022096 (+ 40us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1215 21:01:59.022168 (+ 72us) server_negotiation.cc:227] Negotiated authn=SASL
1215 21:01:59.022939 (+ 771us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:01:59.023410 (+ 471us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1215 21:01:59.023944 (+ 534us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:01:59.024093 (+ 149us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:01:59.026043 (+ 1950us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1215 21:01:59.026059 (+ 16us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1215 21:01:59.026061 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1215 21:01:59.026082 (+ 21us) server_negotiation.cc:923] calling sasl_server_start()
1215 21:01:59.027364 (+ 1282us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1215 21:01:59.027739 (+ 375us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1215 21:01:59.027741 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1215 21:01:59.027743 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1215 21:01:59.027775 (+ 32us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1215 21:01:59.027927 (+ 152us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1215 21:01:59.027929 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1215 21:01:59.027930 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1215 21:01:59.028098 (+ 168us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1215 21:01:59.028171 (+ 73us) server_negotiation.cc:1036] Waiting for connection context
1215 21:01:59.028407 (+ 236us) server_negotiation.cc:300] Negotiation successful
1215 21:01:59.028532 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":32,"server-negotiator.queue_time_us":47}
I20251215 21:01:59.032119 30886 catalog_manager.cc:2257] Servicing CreateTable request from {username='kudu', principal='kudu/127.28.113.62@KRBTEST.COM'} at 127.0.0.1:56872:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20251215 21:01:59.032785 30886 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251215 21:01:59.038005 31024 tablet_service.cc:1505] Processing CreateTablet for tablet 5529f04605c642e289a035f8c04815df (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=7251aad19b58416fb864f21ec6ad0343]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20251215 21:01:59.038049 31168 tablet_service.cc:1505] Processing CreateTablet for tablet 5529f04605c642e289a035f8c04815df (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=7251aad19b58416fb864f21ec6ad0343]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20251215 21:01:59.038338 31310 tablet_service.cc:1505] Processing CreateTablet for tablet 5529f04605c642e289a035f8c04815df (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=7251aad19b58416fb864f21ec6ad0343]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20251215 21:01:59.038633 31024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5529f04605c642e289a035f8c04815df. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 21:01:59.038681 31168 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5529f04605c642e289a035f8c04815df. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 21:01:59.038812 31310 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5529f04605c642e289a035f8c04815df. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 21:01:59.041038 31401 tablet_bootstrap.cc:492] T 5529f04605c642e289a035f8c04815df P 6e18dca41b204cf0a3ebedd00ec07906: Bootstrap starting.
I20251215 21:01:59.041281 31399 tablet_bootstrap.cc:492] T 5529f04605c642e289a035f8c04815df P 5755a662553c4a84b76c8b2c48880567: Bootstrap starting.
I20251215 21:01:59.041508 31400 tablet_bootstrap.cc:492] T 5529f04605c642e289a035f8c04815df P 9899e8262dee40858d8880500e8054b1: Bootstrap starting.
I20251215 21:01:59.041996 31401 tablet_bootstrap.cc:654] T 5529f04605c642e289a035f8c04815df P 6e18dca41b204cf0a3ebedd00ec07906: Neither blocks nor log segments found. Creating new log.
I20251215 21:01:59.042281 31399 tablet_bootstrap.cc:654] T 5529f04605c642e289a035f8c04815df P 5755a662553c4a84b76c8b2c48880567: Neither blocks nor log segments found. Creating new log.
I20251215 21:01:59.042460 31400 tablet_bootstrap.cc:654] T 5529f04605c642e289a035f8c04815df P 9899e8262dee40858d8880500e8054b1: Neither blocks nor log segments found. Creating new log.
I20251215 21:01:59.043114 31401 tablet_bootstrap.cc:492] T 5529f04605c642e289a035f8c04815df P 6e18dca41b204cf0a3ebedd00ec07906: No bootstrap required, opened a new log
I20251215 21:01:59.043198 31401 ts_tablet_manager.cc:1403] T 5529f04605c642e289a035f8c04815df P 6e18dca41b204cf0a3ebedd00ec07906: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20251215 21:01:59.043630 31399 tablet_bootstrap.cc:492] T 5529f04605c642e289a035f8c04815df P 5755a662553c4a84b76c8b2c48880567: No bootstrap required, opened a new log
I20251215 21:01:59.043601 31401 raft_consensus.cc:359] T 5529f04605c642e289a035f8c04815df P 6e18dca41b204cf0a3ebedd00ec07906 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e18dca41b204cf0a3ebedd00ec07906" member_type: VOTER last_known_addr { host: "127.28.113.1" port: 33435 } } peers { permanent_uuid: "9899e8262dee40858d8880500e8054b1" member_type: VOTER last_known_addr { host: "127.28.113.2" port: 35261 } } peers { permanent_uuid: "5755a662553c4a84b76c8b2c48880567" member_type: VOTER last_known_addr { host: "127.28.113.3" port: 42057 } }
I20251215 21:01:59.043691 31399 ts_tablet_manager.cc:1403] T 5529f04605c642e289a035f8c04815df P 5755a662553c4a84b76c8b2c48880567: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20251215 21:01:59.043704 31401 raft_consensus.cc:385] T 5529f04605c642e289a035f8c04815df P 6e18dca41b204cf0a3ebedd00ec07906 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251215 21:01:59.043736 31401 raft_consensus.cc:740] T 5529f04605c642e289a035f8c04815df P 6e18dca41b204cf0a3ebedd00ec07906 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e18dca41b204cf0a3ebedd00ec07906, State: Initialized, Role: FOLLOWER
I20251215 21:01:59.043866 31401 consensus_queue.cc:260] T 5529f04605c642e289a035f8c04815df P 6e18dca41b204cf0a3ebedd00ec07906 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e18dca41b204cf0a3ebedd00ec07906" member_type: VOTER last_known_addr { host: "127.28.113.1" port: 33435 } } peers { permanent_uuid: "9899e8262dee40858d8880500e8054b1" member_type: VOTER last_known_addr { host: "127.28.113.2" port: 35261 } } peers { permanent_uuid: "5755a662553c4a84b76c8b2c48880567" member_type: VOTER last_known_addr { host: "127.28.113.3" port: 42057 } }
I20251215 21:01:59.044010 31400 tablet_bootstrap.cc:492] T 5529f04605c642e289a035f8c04815df P 9899e8262dee40858d8880500e8054b1: No bootstrap required, opened a new log
I20251215 21:01:59.044080 31400 ts_tablet_manager.cc:1403] T 5529f04605c642e289a035f8c04815df P 9899e8262dee40858d8880500e8054b1: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20251215 21:01:59.044134 31401 ts_tablet_manager.cc:1434] T 5529f04605c642e289a035f8c04815df P 6e18dca41b204cf0a3ebedd00ec07906: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20251215 21:01:59.044098 31399 raft_consensus.cc:359] T 5529f04605c642e289a035f8c04815df P 5755a662553c4a84b76c8b2c48880567 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e18dca41b204cf0a3ebedd00ec07906" member_type: VOTER last_known_addr { host: "127.28.113.1" port: 33435 } } peers { permanent_uuid: "9899e8262dee40858d8880500e8054b1" member_type: VOTER last_known_addr { host: "127.28.113.2" port: 35261 } } peers { permanent_uuid: "5755a662553c4a84b76c8b2c48880567" member_type: VOTER last_known_addr { host: "127.28.113.3" port: 42057 } }
I20251215 21:01:59.044173 31399 raft_consensus.cc:385] T 5529f04605c642e289a035f8c04815df P 5755a662553c4a84b76c8b2c48880567 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251215 21:01:59.044195 31399 raft_consensus.cc:740] T 5529f04605c642e289a035f8c04815df P 5755a662553c4a84b76c8b2c48880567 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5755a662553c4a84b76c8b2c48880567, State: Initialized, Role: FOLLOWER
I20251215 21:01:59.044297 31399 consensus_queue.cc:260] T 5529f04605c642e289a035f8c04815df P 5755a662553c4a84b76c8b2c48880567 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e18dca41b204cf0a3ebedd00ec07906" member_type: VOTER last_known_addr { host: "127.28.113.1" port: 33435 } } peers { permanent_uuid: "9899e8262dee40858d8880500e8054b1" member_type: VOTER last_known_addr { host: "127.28.113.2" port: 35261 } } peers { permanent_uuid: "5755a662553c4a84b76c8b2c48880567" member_type: VOTER last_known_addr { host: "127.28.113.3" port: 42057 } }
I20251215 21:01:59.044350 31407 tablet_replica.cc:442] T 5529f04605c642e289a035f8c04815df P 6e18dca41b204cf0a3ebedd00ec07906: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e18dca41b204cf0a3ebedd00ec07906" member_type: VOTER last_known_addr { host: "127.28.113.1" port: 33435 } } peers { permanent_uuid: "9899e8262dee40858d8880500e8054b1" member_type: VOTER last_known_addr { host: "127.28.113.2" port: 35261 } } peers { permanent_uuid: "5755a662553c4a84b76c8b2c48880567" member_type: VOTER last_known_addr { host: "127.28.113.3" port: 42057 } } }
I20251215 21:01:59.044456 31407 tablet_replica.cc:445] T 5529f04605c642e289a035f8c04815df P 6e18dca41b204cf0a3ebedd00ec07906: This TxnStatusTablet replica's current role is: FOLLOWER
I20251215 21:01:59.044514 31399 ts_tablet_manager.cc:1434] T 5529f04605c642e289a035f8c04815df P 5755a662553c4a84b76c8b2c48880567: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20251215 21:01:59.044512 31400 raft_consensus.cc:359] T 5529f04605c642e289a035f8c04815df P 9899e8262dee40858d8880500e8054b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e18dca41b204cf0a3ebedd00ec07906" member_type: VOTER last_known_addr { host: "127.28.113.1" port: 33435 } } peers { permanent_uuid: "9899e8262dee40858d8880500e8054b1" member_type: VOTER last_known_addr { host: "127.28.113.2" port: 35261 } } peers { permanent_uuid: "5755a662553c4a84b76c8b2c48880567" member_type: VOTER last_known_addr { host: "127.28.113.3" port: 42057 } }
I20251215 21:01:59.044615 31400 raft_consensus.cc:385] T 5529f04605c642e289a035f8c04815df P 9899e8262dee40858d8880500e8054b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251215 21:01:59.044651 31400 raft_consensus.cc:740] T 5529f04605c642e289a035f8c04815df P 9899e8262dee40858d8880500e8054b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9899e8262dee40858d8880500e8054b1, State: Initialized, Role: FOLLOWER
I20251215 21:01:59.044759 31400 consensus_queue.cc:260] T 5529f04605c642e289a035f8c04815df P 9899e8262dee40858d8880500e8054b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e18dca41b204cf0a3ebedd00ec07906" member_type: VOTER last_known_addr { host: "127.28.113.1" port: 33435 } } peers { permanent_uuid: "9899e8262dee40858d8880500e8054b1" member_type: VOTER last_known_addr { host: "127.28.113.2" port: 35261 } } peers { permanent_uuid: "5755a662553c4a84b76c8b2c48880567" member_type: VOTER last_known_addr { host: "127.28.113.3" port: 42057 } }
I20251215 21:01:59.045001 31400 ts_tablet_manager.cc:1434] T 5529f04605c642e289a035f8c04815df P 9899e8262dee40858d8880500e8054b1: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20251215 21:01:59.045048 31406 tablet_replica.cc:442] T 5529f04605c642e289a035f8c04815df P 5755a662553c4a84b76c8b2c48880567: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e18dca41b204cf0a3ebedd00ec07906" member_type: VOTER last_known_addr { host: "127.28.113.1" port: 33435 } } peers { permanent_uuid: "9899e8262dee40858d8880500e8054b1" member_type: VOTER last_known_addr { host: "127.28.113.2" port: 35261 } } peers { permanent_uuid: "5755a662553c4a84b76c8b2c48880567" member_type: VOTER last_known_addr { host: "127.28.113.3" port: 42057 } } }
I20251215 21:01:59.045156 31406 tablet_replica.cc:445] T 5529f04605c642e289a035f8c04815df P 5755a662553c4a84b76c8b2c48880567: This TxnStatusTablet replica's current role is: FOLLOWER
I20251215 21:01:59.045234 31405 tablet_replica.cc:442] T 5529f04605c642e289a035f8c04815df P 9899e8262dee40858d8880500e8054b1: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e18dca41b204cf0a3ebedd00ec07906" member_type: VOTER last_known_addr { host: "127.28.113.1" port: 33435 } } peers { permanent_uuid: "9899e8262dee40858d8880500e8054b1" member_type: VOTER last_known_addr { host: "127.28.113.2" port: 35261 } } peers { permanent_uuid: "5755a662553c4a84b76c8b2c48880567" member_type: VOTER last_known_addr { host: "127.28.113.3" port: 42057 } } }
I20251215 21:01:59.045342 31405 tablet_replica.cc:445] T 5529f04605c642e289a035f8c04815df P 9899e8262dee40858d8880500e8054b1: This TxnStatusTablet replica's current role is: FOLLOWER
W20251215 21:01:59.115449 31377 tablet.cc:2378] T 5529f04605c642e289a035f8c04815df P 5755a662553c4a84b76c8b2c48880567: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20251215 21:01:59.147697 31235 tablet.cc:2378] T 5529f04605c642e289a035f8c04815df P 9899e8262dee40858d8880500e8054b1: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20251215 21:01:59.173956 31093 tablet.cc:2378] T 5529f04605c642e289a035f8c04815df P 6e18dca41b204cf0a3ebedd00ec07906: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20251215 21:01:59.230430 31407 raft_consensus.cc:493] T 5529f04605c642e289a035f8c04815df P 6e18dca41b204cf0a3ebedd00ec07906 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 21:01:59.230589 31407 raft_consensus.cc:515] T 5529f04605c642e289a035f8c04815df P 6e18dca41b204cf0a3ebedd00ec07906 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e18dca41b204cf0a3ebedd00ec07906" member_type: VOTER last_known_addr { host: "127.28.113.1" port: 33435 } } peers { permanent_uuid: "9899e8262dee40858d8880500e8054b1" member_type: VOTER last_known_addr { host: "127.28.113.2" port: 35261 } } peers { permanent_uuid: "5755a662553c4a84b76c8b2c48880567" member_type: VOTER last_known_addr { host: "127.28.113.3" port: 42057 } }
I20251215 21:01:59.231657 31407 leader_election.cc:290] T 5529f04605c642e289a035f8c04815df P 6e18dca41b204cf0a3ebedd00ec07906 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9899e8262dee40858d8880500e8054b1 (127.28.113.2:35261), 5755a662553c4a84b76c8b2c48880567 (127.28.113.3:42057)
I20251215 21:01:59.234570 31420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:01:59.231828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.113.2:35261 (local address 127.28.113.1:43425)
1215 21:01:59.232218 (+ 390us) negotiation.cc:107] Waiting for socket to connect
1215 21:01:59.232229 (+ 11us) client_negotiation.cc:174] Beginning negotiation
1215 21:01:59.232316 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1215 21:01:59.232543 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1215 21:01:59.232546 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1215 21:01:59.232562 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1215 21:01:59.232820 (+ 258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:01:59.232828 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:01:59.233600 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1215 21:01:59.233604 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1215 21:01:59.234226 (+ 622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:01:59.234234 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:01:59.234314 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:01:59.234330 (+ 16us) client_negotiation.cc:769] Sending connection context
1215 21:01:59.234372 (+ 42us) client_negotiation.cc:240] Negotiation successful
1215 21:01:59.234425 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":331,"thread_start_us":141,"threads_started":1}
I20251215 21:01:59.234942 31421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:01:59.231898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.113.3:42057 (local address 127.28.113.1:43603)
1215 21:01:59.232308 (+ 410us) negotiation.cc:107] Waiting for socket to connect
1215 21:01:59.232321 (+ 13us) client_negotiation.cc:174] Beginning negotiation
1215 21:01:59.232418 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1215 21:01:59.232643 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1215 21:01:59.232645 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1215 21:01:59.232659 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1215 21:01:59.232860 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:01:59.232872 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:01:59.233630 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1215 21:01:59.233633 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1215 21:01:59.234287 (+ 654us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:01:59.234298 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:01:59.234421 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:01:59.234439 (+ 18us) client_negotiation.cc:769] Sending connection context
1215 21:01:59.234485 (+ 46us) client_negotiation.cc:240] Negotiation successful
1215 21:01:59.234565 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":139,"threads_started":1}
I20251215 21:01:59.235054 31394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:01:59.231945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.113.1:43425 (local address 127.28.113.2:35261)
1215 21:01:59.232101 (+ 156us) server_negotiation.cc:207] Beginning negotiation
1215 21:01:59.232105 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1215 21:01:59.232297 (+ 192us) server_negotiation.cc:408] Connection header received
1215 21:01:59.232395 (+ 98us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1215 21:01:59.232398 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1215 21:01:59.232457 (+ 59us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1215 21:01:59.232527 (+ 70us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1215 21:01:59.232933 (+ 406us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:01:59.233496 (+ 563us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1215 21:01:59.234343 (+ 847us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:01:59.234786 (+ 443us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:01:59.234835 (+ 49us) server_negotiation.cc:1036] Waiting for connection context
1215 21:01:59.234887 (+ 52us) server_negotiation.cc:300] Negotiation successful
1215 21:01:59.234944 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20251215 21:01:59.235114 31397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:01:59.231995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.113.1:43603 (local address 127.28.113.3:42057)
1215 21:01:59.232122 (+ 127us) server_negotiation.cc:207] Beginning negotiation
1215 21:01:59.232127 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1215 21:01:59.232399 (+ 272us) server_negotiation.cc:408] Connection header received
1215 21:01:59.232498 (+ 99us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1215 21:01:59.232501 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1215 21:01:59.232551 (+ 50us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1215 21:01:59.232632 (+ 81us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1215 21:01:59.232996 (+ 364us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:01:59.233525 (+ 529us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1215 21:01:59.234413 (+ 888us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:01:59.234877 (+ 464us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:01:59.234912 (+ 35us) server_negotiation.cc:1036] Waiting for connection context
1215 21:01:59.234954 (+ 42us) server_negotiation.cc:300] Negotiation successful
1215 21:01:59.235006 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20251215 21:01:59.235424 31176 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5529f04605c642e289a035f8c04815df" candidate_uuid: "6e18dca41b204cf0a3ebedd00ec07906" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9899e8262dee40858d8880500e8054b1" is_pre_election: true
I20251215 21:01:59.235566 31176 raft_consensus.cc:2468] T 5529f04605c642e289a035f8c04815df P 9899e8262dee40858d8880500e8054b1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e18dca41b204cf0a3ebedd00ec07906 in term 0.
I20251215 21:01:59.235584 31330 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5529f04605c642e289a035f8c04815df" candidate_uuid: "6e18dca41b204cf0a3ebedd00ec07906" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5755a662553c4a84b76c8b2c48880567" is_pre_election: true
I20251215 21:01:59.235770 31330 raft_consensus.cc:2468] T 5529f04605c642e289a035f8c04815df P 5755a662553c4a84b76c8b2c48880567 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e18dca41b204cf0a3ebedd00ec07906 in term 0.
I20251215 21:01:59.235893 30978 leader_election.cc:304] T 5529f04605c642e289a035f8c04815df P 6e18dca41b204cf0a3ebedd00ec07906 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e18dca41b204cf0a3ebedd00ec07906, 9899e8262dee40858d8880500e8054b1; no voters:
I20251215 21:01:59.236152 31407 raft_consensus.cc:2804] T 5529f04605c642e289a035f8c04815df P 6e18dca41b204cf0a3ebedd00ec07906 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251215 21:01:59.236205 31407 raft_consensus.cc:493] T 5529f04605c642e289a035f8c04815df P 6e18dca41b204cf0a3ebedd00ec07906 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 21:01:59.236238 31407 raft_consensus.cc:3060] T 5529f04605c642e289a035f8c04815df P 6e18dca41b204cf0a3ebedd00ec07906 [term 0 FOLLOWER]: Advancing to term 1
I20251215 21:01:59.237046 31407 raft_consensus.cc:515] T 5529f04605c642e289a035f8c04815df P 6e18dca41b204cf0a3ebedd00ec07906 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e18dca41b204cf0a3ebedd00ec07906" member_type: VOTER last_known_addr { host: "127.28.113.1" port: 33435 } } peers { permanent_uuid: "9899e8262dee40858d8880500e8054b1" member_type: VOTER last_known_addr { host: "127.28.113.2" port: 35261 } } peers { permanent_uuid: "5755a662553c4a84b76c8b2c48880567" member_type: VOTER last_known_addr { host: "127.28.113.3" port: 42057 } }
I20251215 21:01:59.237316 31407 leader_election.cc:290] T 5529f04605c642e289a035f8c04815df P 6e18dca41b204cf0a3ebedd00ec07906 [CANDIDATE]: Term 1 election: Requested vote from peers 9899e8262dee40858d8880500e8054b1 (127.28.113.2:35261), 5755a662553c4a84b76c8b2c48880567 (127.28.113.3:42057)
I20251215 21:01:59.237656 31176 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5529f04605c642e289a035f8c04815df" candidate_uuid: "6e18dca41b204cf0a3ebedd00ec07906" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9899e8262dee40858d8880500e8054b1"
I20251215 21:01:59.237772 31176 raft_consensus.cc:3060] T 5529f04605c642e289a035f8c04815df P 9899e8262dee40858d8880500e8054b1 [term 0 FOLLOWER]: Advancing to term 1
I20251215 21:01:59.237871 31330 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5529f04605c642e289a035f8c04815df" candidate_uuid: "6e18dca41b204cf0a3ebedd00ec07906" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5755a662553c4a84b76c8b2c48880567"
I20251215 21:01:59.237977 31330 raft_consensus.cc:3060] T 5529f04605c642e289a035f8c04815df P 5755a662553c4a84b76c8b2c48880567 [term 0 FOLLOWER]: Advancing to term 1
I20251215 21:01:59.238571 31176 raft_consensus.cc:2468] T 5529f04605c642e289a035f8c04815df P 9899e8262dee40858d8880500e8054b1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e18dca41b204cf0a3ebedd00ec07906 in term 1.
I20251215 21:01:59.238828 30978 leader_election.cc:304] T 5529f04605c642e289a035f8c04815df P 6e18dca41b204cf0a3ebedd00ec07906 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e18dca41b204cf0a3ebedd00ec07906, 9899e8262dee40858d8880500e8054b1; no voters:
I20251215 21:01:59.238922 31330 raft_consensus.cc:2468] T 5529f04605c642e289a035f8c04815df P 5755a662553c4a84b76c8b2c48880567 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e18dca41b204cf0a3ebedd00ec07906 in term 1.
I20251215 21:01:59.239024 31407 raft_consensus.cc:2804] T 5529f04605c642e289a035f8c04815df P 6e18dca41b204cf0a3ebedd00ec07906 [term 1 FOLLOWER]: Leader election won for term 1
I20251215 21:01:59.239255 31407 raft_consensus.cc:697] T 5529f04605c642e289a035f8c04815df P 6e18dca41b204cf0a3ebedd00ec07906 [term 1 LEADER]: Becoming Leader. State: Replica: 6e18dca41b204cf0a3ebedd00ec07906, State: Running, Role: LEADER
I20251215 21:01:59.239603 31407 consensus_queue.cc:237] T 5529f04605c642e289a035f8c04815df P 6e18dca41b204cf0a3ebedd00ec07906 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e18dca41b204cf0a3ebedd00ec07906" member_type: VOTER last_known_addr { host: "127.28.113.1" port: 33435 } } peers { permanent_uuid: "9899e8262dee40858d8880500e8054b1" member_type: VOTER last_known_addr { host: "127.28.113.2" port: 35261 } } peers { permanent_uuid: "5755a662553c4a84b76c8b2c48880567" member_type: VOTER last_known_addr { host: "127.28.113.3" port: 42057 } }
I20251215 21:01:59.241775 31422 tablet_replica.cc:442] T 5529f04605c642e289a035f8c04815df P 6e18dca41b204cf0a3ebedd00ec07906: TxnStatusTablet state changed. Reason: New leader 6e18dca41b204cf0a3ebedd00ec07906. Latest consensus state: current_term: 1 leader_uuid: "6e18dca41b204cf0a3ebedd00ec07906" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e18dca41b204cf0a3ebedd00ec07906" member_type: VOTER last_known_addr { host: "127.28.113.1" port: 33435 } } peers { permanent_uuid: "9899e8262dee40858d8880500e8054b1" member_type: VOTER last_known_addr { host: "127.28.113.2" port: 35261 } } peers { permanent_uuid: "5755a662553c4a84b76c8b2c48880567" member_type: VOTER last_known_addr { host: "127.28.113.3" port: 42057 } } }
I20251215 21:01:59.241854 31422 tablet_replica.cc:445] T 5529f04605c642e289a035f8c04815df P 6e18dca41b204cf0a3ebedd00ec07906: This TxnStatusTablet replica's current role is: LEADER
I20251215 21:01:59.242081 31424 txn_status_manager.cc:874] Waiting until node catch up with all replicated operations in previous term...
I20251215 21:01:59.242283 31424 txn_status_manager.cc:930] Loading transaction status metadata into memory...
I20251215 21:01:59.242719 30882 catalog_manager.cc:5654] T 5529f04605c642e289a035f8c04815df P 6e18dca41b204cf0a3ebedd00ec07906 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6e18dca41b204cf0a3ebedd00ec07906 (127.28.113.1). New cstate: current_term: 1 leader_uuid: "6e18dca41b204cf0a3ebedd00ec07906" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e18dca41b204cf0a3ebedd00ec07906" member_type: VOTER last_known_addr { host: "127.28.113.1" port: 33435 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9899e8262dee40858d8880500e8054b1" member_type: VOTER last_known_addr { host: "127.28.113.2" port: 35261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5755a662553c4a84b76c8b2c48880567" member_type: VOTER last_known_addr { host: "127.28.113.3" port: 42057 } health_report { overall_health: UNKNOWN } } }
I20251215 21:01:59.283773 31416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:01:59.280477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.113.1:33435 (local address 127.0.0.1:45756)
1215 21:01:59.280630 (+ 153us) negotiation.cc:107] Waiting for socket to connect
1215 21:01:59.280645 (+ 15us) client_negotiation.cc:174] Beginning negotiation
1215 21:01:59.280749 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1215 21:01:59.281047 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1215 21:01:59.281049 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1215 21:01:59.281056 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
1215 21:01:59.281233 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:01:59.281238 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:01:59.282170 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1215 21:01:59.282172 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1215 21:01:59.282575 (+ 403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:01:59.282580 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:01:59.282659 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:01:59.282698 (+ 39us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
1215 21:01:59.283559 (+ 861us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
1215 21:01:59.283566 (+ 7us) client_negotiation.cc:769] Sending connection context
1215 21:01:59.283600 (+ 34us) client_negotiation.cc:240] Negotiation successful
1215 21:01:59.283662 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20251215 21:01:59.283811 31398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:01:59.280581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45756 (local address 127.28.113.1:33435)
1215 21:01:59.280745 (+ 164us) server_negotiation.cc:207] Beginning negotiation
1215 21:01:59.280750 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1215 21:01:59.280766 (+ 16us) server_negotiation.cc:408] Connection header received
1215 21:01:59.280880 (+ 114us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1215 21:01:59.280884 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1215 21:01:59.280956 (+ 72us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1215 21:01:59.281051 (+ 95us) server_negotiation.cc:227] Negotiated authn=TOKEN
1215 21:01:59.281323 (+ 272us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:01:59.282080 (+ 757us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1215 21:01:59.282667 (+ 587us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:01:59.282856 (+ 189us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:01:59.282918 (+ 62us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1215 21:01:59.283467 (+ 549us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1215 21:01:59.283590 (+ 123us) server_negotiation.cc:1036] Waiting for connection context
1215 21:01:59.283619 (+ 29us) server_negotiation.cc:300] Negotiation successful
1215 21:01:59.283680 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20251215 21:01:59.288528 31330 raft_consensus.cc:1275] T 5529f04605c642e289a035f8c04815df P 5755a662553c4a84b76c8b2c48880567 [term 1 FOLLOWER]: Refusing update from remote peer 6e18dca41b204cf0a3ebedd00ec07906: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251215 21:01:59.288532 31176 raft_consensus.cc:1275] T 5529f04605c642e289a035f8c04815df P 9899e8262dee40858d8880500e8054b1 [term 1 FOLLOWER]: Refusing update from remote peer 6e18dca41b204cf0a3ebedd00ec07906: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251215 21:01:59.289099 31422 consensus_queue.cc:1048] T 5529f04605c642e289a035f8c04815df P 6e18dca41b204cf0a3ebedd00ec07906 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9899e8262dee40858d8880500e8054b1" member_type: VOTER last_known_addr { host: "127.28.113.2" port: 35261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251215 21:01:59.289281 31407 consensus_queue.cc:1048] T 5529f04605c642e289a035f8c04815df P 6e18dca41b204cf0a3ebedd00ec07906 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5755a662553c4a84b76c8b2c48880567" member_type: VOTER last_known_addr { host: "127.28.113.3" port: 42057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251215 21:01:59.292135 31406 tablet_replica.cc:442] T 5529f04605c642e289a035f8c04815df P 5755a662553c4a84b76c8b2c48880567: TxnStatusTablet state changed. Reason: New leader 6e18dca41b204cf0a3ebedd00ec07906. Latest consensus state: current_term: 1 leader_uuid: "6e18dca41b204cf0a3ebedd00ec07906" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e18dca41b204cf0a3ebedd00ec07906" member_type: VOTER last_known_addr { host: "127.28.113.1" port: 33435 } } peers { permanent_uuid: "9899e8262dee40858d8880500e8054b1" member_type: VOTER last_known_addr { host: "127.28.113.2" port: 35261 } } peers { permanent_uuid: "5755a662553c4a84b76c8b2c48880567" member_type: VOTER last_known_addr { host: "127.28.113.3" port: 42057 } } }
I20251215 21:01:59.292245 31406 tablet_replica.cc:445] T 5529f04605c642e289a035f8c04815df P 5755a662553c4a84b76c8b2c48880567: This TxnStatusTablet replica's current role is: FOLLOWER
I20251215 21:01:59.292593 31405 tablet_replica.cc:442] T 5529f04605c642e289a035f8c04815df P 9899e8262dee40858d8880500e8054b1: TxnStatusTablet state changed. Reason: New leader 6e18dca41b204cf0a3ebedd00ec07906. Latest consensus state: current_term: 1 leader_uuid: "6e18dca41b204cf0a3ebedd00ec07906" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e18dca41b204cf0a3ebedd00ec07906" member_type: VOTER last_known_addr { host: "127.28.113.1" port: 33435 } } peers { permanent_uuid: "9899e8262dee40858d8880500e8054b1" member_type: VOTER last_known_addr { host: "127.28.113.2" port: 35261 } } peers { permanent_uuid: "5755a662553c4a84b76c8b2c48880567" member_type: VOTER last_known_addr { host: "127.28.113.3" port: 42057 } } }
I20251215 21:01:59.292702 31405 tablet_replica.cc:445] T 5529f04605c642e289a035f8c04815df P 9899e8262dee40858d8880500e8054b1: This TxnStatusTablet replica's current role is: FOLLOWER
I20251215 21:01:59.295032 31422 tablet_replica.cc:442] T 5529f04605c642e289a035f8c04815df P 6e18dca41b204cf0a3ebedd00ec07906: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6e18dca41b204cf0a3ebedd00ec07906" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e18dca41b204cf0a3ebedd00ec07906" member_type: VOTER last_known_addr { host: "127.28.113.1" port: 33435 } } peers { permanent_uuid: "9899e8262dee40858d8880500e8054b1" member_type: VOTER last_known_addr { host: "127.28.113.2" port: 35261 } } peers { permanent_uuid: "5755a662553c4a84b76c8b2c48880567" member_type: VOTER last_known_addr { host: "127.28.113.3" port: 42057 } } }
I20251215 21:01:59.295135 31422 tablet_replica.cc:445] T 5529f04605c642e289a035f8c04815df P 6e18dca41b204cf0a3ebedd00ec07906: This TxnStatusTablet replica's current role is: LEADER
I20251215 21:01:59.295267 31427 mvcc.cc:204] Tried to move back new op lower bound from 7232849998999445504 to 7232849998807945216. Current Snapshot: MvccSnapshot[applied={T|T < 7232849998999445504}]
I20251215 21:01:59.295428 31405 tablet_replica.cc:442] T 5529f04605c642e289a035f8c04815df P 9899e8262dee40858d8880500e8054b1: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6e18dca41b204cf0a3ebedd00ec07906" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e18dca41b204cf0a3ebedd00ec07906" member_type: VOTER last_known_addr { host: "127.28.113.1" port: 33435 } } peers { permanent_uuid: "9899e8262dee40858d8880500e8054b1" member_type: VOTER last_known_addr { host: "127.28.113.2" port: 35261 } } peers { permanent_uuid: "5755a662553c4a84b76c8b2c48880567" member_type: VOTER last_known_addr { host: "127.28.113.3" port: 42057 } } }
I20251215 21:01:59.295533 31405 tablet_replica.cc:445] T 5529f04605c642e289a035f8c04815df P 9899e8262dee40858d8880500e8054b1: This TxnStatusTablet replica's current role is: FOLLOWER
I20251215 21:01:59.296754 31429 mvcc.cc:204] Tried to move back new op lower bound from 7232849998999445504 to 7232849998807945216. Current Snapshot: MvccSnapshot[applied={T|T < 7232849998999445504}]
I20251215 21:01:59.299561 31407 tablet_replica.cc:442] T 5529f04605c642e289a035f8c04815df P 6e18dca41b204cf0a3ebedd00ec07906: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6e18dca41b204cf0a3ebedd00ec07906" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e18dca41b204cf0a3ebedd00ec07906" member_type: VOTER last_known_addr { host: "127.28.113.1" port: 33435 } } peers { permanent_uuid: "9899e8262dee40858d8880500e8054b1" member_type: VOTER last_known_addr { host: "127.28.113.2" port: 35261 } } peers { permanent_uuid: "5755a662553c4a84b76c8b2c48880567" member_type: VOTER last_known_addr { host: "127.28.113.3" port: 42057 } } }
I20251215 21:01:59.299660 31407 tablet_replica.cc:445] T 5529f04605c642e289a035f8c04815df P 6e18dca41b204cf0a3ebedd00ec07906: This TxnStatusTablet replica's current role is: LEADER
I20251215 21:01:59.300611 31406 tablet_replica.cc:442] T 5529f04605c642e289a035f8c04815df P 5755a662553c4a84b76c8b2c48880567: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6e18dca41b204cf0a3ebedd00ec07906" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e18dca41b204cf0a3ebedd00ec07906" member_type: VOTER last_known_addr { host: "127.28.113.1" port: 33435 } } peers { permanent_uuid: "9899e8262dee40858d8880500e8054b1" member_type: VOTER last_known_addr { host: "127.28.113.2" port: 35261 } } peers { permanent_uuid: "5755a662553c4a84b76c8b2c48880567" member_type: VOTER last_known_addr { host: "127.28.113.3" port: 42057 } } }
I20251215 21:01:59.300719 31406 tablet_replica.cc:445] T 5529f04605c642e289a035f8c04815df P 5755a662553c4a84b76c8b2c48880567: This TxnStatusTablet replica's current role is: FOLLOWER
I20251215 21:01:59.311676 31397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:01:59.308705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40376 (local address 127.28.113.3:42057)
1215 21:01:59.308810 (+ 105us) server_negotiation.cc:207] Beginning negotiation
1215 21:01:59.308814 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1215 21:01:59.308828 (+ 14us) server_negotiation.cc:408] Connection header received
1215 21:01:59.309009 (+ 181us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1215 21:01:59.309013 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1215 21:01:59.309066 (+ 53us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1215 21:01:59.309144 (+ 78us) server_negotiation.cc:227] Negotiated authn=TOKEN
1215 21:01:59.309462 (+ 318us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:01:59.309937 (+ 475us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1215 21:01:59.310528 (+ 591us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:01:59.310692 (+ 164us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:01:59.310740 (+ 48us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1215 21:01:59.311150 (+ 410us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1215 21:01:59.311248 (+ 98us) server_negotiation.cc:1036] Waiting for connection context
1215 21:01:59.311500 (+ 252us) server_negotiation.cc:300] Negotiation successful
1215 21:01:59.311553 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20251215 21:01:59.320683 31398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:01:59.318347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.113.3:33333 (local address 127.28.113.1:33435)
1215 21:01:59.318480 (+ 133us) server_negotiation.cc:207] Beginning negotiation
1215 21:01:59.318484 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1215 21:01:59.318497 (+ 13us) server_negotiation.cc:408] Connection header received
1215 21:01:59.318612 (+ 115us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1215 21:01:59.318615 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1215 21:01:59.318658 (+ 43us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1215 21:01:59.318715 (+ 57us) server_negotiation.cc:227] Negotiated authn=TOKEN
1215 21:01:59.319016 (+ 301us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:01:59.319456 (+ 440us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1215 21:01:59.320089 (+ 633us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:01:59.320258 (+ 169us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:01:59.320299 (+ 41us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1215 21:01:59.320361 (+ 62us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1215 21:01:59.320419 (+ 58us) server_negotiation.cc:1036] Waiting for connection context
1215 21:01:59.320512 (+ 93us) server_negotiation.cc:300] Negotiation successful
1215 21:01:59.320554 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20251215 21:01:59.320683 31382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:01:59.318243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.113.1:33435 (local address 127.28.113.3:33333)
1215 21:01:59.318423 (+ 180us) negotiation.cc:107] Waiting for socket to connect
1215 21:01:59.318437 (+ 14us) client_negotiation.cc:174] Beginning negotiation
1215 21:01:59.318520 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1215 21:01:59.318730 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1215 21:01:59.318733 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1215 21:01:59.318740 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
1215 21:01:59.318923 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:01:59.318928 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:01:59.319549 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1215 21:01:59.319553 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1215 21:01:59.319993 (+ 440us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:01:59.320003 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:01:59.320097 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:01:59.320138 (+ 41us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
1215 21:01:59.320436 (+ 298us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
1215 21:01:59.320441 (+ 5us) client_negotiation.cc:769] Sending connection context
1215 21:01:59.320497 (+ 56us) client_negotiation.cc:240] Negotiation successful
1215 21:01:59.320555 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":97}
I20251215 21:01:59.328446 31176 raft_consensus.cc:1275] T a24de5579422439992ad5b94ba3db3c2 P 9899e8262dee40858d8880500e8054b1 [term 1 FOLLOWER]: Refusing update from remote peer 5755a662553c4a84b76c8b2c48880567: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251215 21:01:59.328704 31045 raft_consensus.cc:1275] T a24de5579422439992ad5b94ba3db3c2 P 6e18dca41b204cf0a3ebedd00ec07906 [term 1 FOLLOWER]: Refusing update from remote peer 5755a662553c4a84b76c8b2c48880567: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251215 21:01:59.329283 31409 consensus_queue.cc:1048] T a24de5579422439992ad5b94ba3db3c2 P 5755a662553c4a84b76c8b2c48880567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9899e8262dee40858d8880500e8054b1" member_type: VOTER last_known_addr { host: "127.28.113.2" port: 35261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251215 21:01:59.329443 31406 consensus_queue.cc:1048] T a24de5579422439992ad5b94ba3db3c2 P 5755a662553c4a84b76c8b2c48880567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e18dca41b204cf0a3ebedd00ec07906" member_type: VOTER last_known_addr { host: "127.28.113.1" port: 33435 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251215 21:01:59.332113 31428 mvcc.cc:204] Tried to move back new op lower bound from 7232849999165444096 to 7232849997770272768. Current Snapshot: MvccSnapshot[applied={T|T < 7232849999165444096}]
I20251215 21:01:59.350409 31442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:01:59.347414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.113.3:42057 (local address 127.28.113.1:41129)
1215 21:01:59.347719 (+ 305us) negotiation.cc:107] Waiting for socket to connect
1215 21:01:59.347734 (+ 15us) client_negotiation.cc:174] Beginning negotiation
1215 21:01:59.347838 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1215 21:01:59.348122 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1215 21:01:59.348126 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
1215 21:01:59.348134 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
1215 21:01:59.348460 (+ 326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:01:59.348469 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:01:59.349271 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1215 21:01:59.349274 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1215 21:01:59.349699 (+ 425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:01:59.349705 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:01:59.349797 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:01:59.349839 (+ 42us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
1215 21:01:59.350153 (+ 314us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
1215 21:01:59.350158 (+ 5us) client_negotiation.cc:769] Sending connection context
1215 21:01:59.350211 (+ 53us) client_negotiation.cc:240] Negotiation successful
1215 21:01:59.350280 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":113,"threads_started":1}
I20251215 21:01:59.350389 31397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:01:59.347475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.113.1:41129 (local address 127.28.113.3:42057)
1215 21:01:59.347582 (+ 107us) server_negotiation.cc:207] Beginning negotiation
1215 21:01:59.347586 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1215 21:01:59.347816 (+ 230us) server_negotiation.cc:408] Connection header received
1215 21:01:59.347943 (+ 127us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1215 21:01:59.347947 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1215 21:01:59.348022 (+ 75us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1215 21:01:59.348102 (+ 80us) server_negotiation.cc:227] Negotiated authn=TOKEN
1215 21:01:59.348551 (+ 449us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:01:59.349176 (+ 625us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1215 21:01:59.349795 (+ 619us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:01:59.349980 (+ 185us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:01:59.350019 (+ 39us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1215 21:01:59.350078 (+ 59us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1215 21:01:59.350136 (+ 58us) server_negotiation.cc:1036] Waiting for connection context
1215 21:01:59.350225 (+ 89us) server_negotiation.cc:300] Negotiation successful
1215 21:01:59.350266 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20251215 21:01:59.371945 30886 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56856:
table { table_name: "test-table" } modify_external_catalogs: true
I20251215 21:01:59.372146 30886 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56856:
table { table_name: "test-table" } modify_external_catalogs: true
I20251215 21:01:59.374636 30886 catalog_manager.cc:5941] T 00000000000000000000000000000000 P 8d6aede0d59c4dbd8c8bca7ade69344b: Sending DeleteTablet for 3 replicas of tablet a24de5579422439992ad5b94ba3db3c2
I20251215 21:01:59.375502 31168 tablet_service.cc:1552] Processing DeleteTablet for tablet a24de5579422439992ad5b94ba3db3c2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-15 21:01:59 UTC) from {username='kudu', principal='kudu/127.28.113.62@KRBTEST.COM'} at 127.0.0.1:34108
I20251215 21:01:59.375514 31310 tablet_service.cc:1552] Processing DeleteTablet for tablet a24de5579422439992ad5b94ba3db3c2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-15 21:01:59 UTC) from {username='kudu', principal='kudu/127.28.113.62@KRBTEST.COM'} at 127.0.0.1:40368
I20251215 21:01:59.376044 31447 tablet_replica.cc:333] T a24de5579422439992ad5b94ba3db3c2 P 5755a662553c4a84b76c8b2c48880567: stopping tablet replica
I20251215 21:01:59.376312 31447 raft_consensus.cc:2243] T a24de5579422439992ad5b94ba3db3c2 P 5755a662553c4a84b76c8b2c48880567 [term 1 LEADER]: Raft consensus shutting down.
I20251215 21:01:59.376709 31447 raft_consensus.cc:2272] T a24de5579422439992ad5b94ba3db3c2 P 5755a662553c4a84b76c8b2c48880567 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251215 21:01:59.376794 30886 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56856:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251215 21:01:59.377210 30886 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251215 21:01:59.378177 31447 ts_tablet_manager.cc:1916] T a24de5579422439992ad5b94ba3db3c2 P 5755a662553c4a84b76c8b2c48880567: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251215 21:01:59.381142 31447 ts_tablet_manager.cc:1929] T a24de5579422439992ad5b94ba3db3c2 P 5755a662553c4a84b76c8b2c48880567: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20251215 21:01:59.381304 31447 log.cc:1199] T a24de5579422439992ad5b94ba3db3c2 P 5755a662553c4a84b76c8b2c48880567: Deleting WAL directory at /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765832506145157-29124-0/minicluster-data/ts-2/wal/wals/a24de5579422439992ad5b94ba3db3c2
I20251215 21:01:59.381654 31447 ts_tablet_manager.cc:1950] T a24de5579422439992ad5b94ba3db3c2 P 5755a662553c4a84b76c8b2c48880567: Deleting consensus metadata
I20251215 21:01:59.382653 31310 tablet_service.cc:1505] Processing CreateTablet for tablet 5899938341e44e15b437baab732d3ece (DEFAULT_TABLE table=test-table [id=f30ecf8355e64fbe95803bd920322b74]), partition=RANGE (key) PARTITION UNBOUNDED
I20251215 21:01:59.382895 31310 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5899938341e44e15b437baab732d3ece. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 21:01:59.382889 30873 catalog_manager.cc:4985] TS 5755a662553c4a84b76c8b2c48880567 (127.28.113.3:42057): tablet a24de5579422439992ad5b94ba3db3c2 (table test-table [id=f68a68024bae4b39990c0e16c60f12f3]) successfully deleted
I20251215 21:01:59.384980 31399 tablet_bootstrap.cc:492] T 5899938341e44e15b437baab732d3ece P 5755a662553c4a84b76c8b2c48880567: Bootstrap starting.
I20251215 21:01:59.385938 31399 tablet_bootstrap.cc:654] T 5899938341e44e15b437baab732d3ece P 5755a662553c4a84b76c8b2c48880567: Neither blocks nor log segments found. Creating new log.
I20251215 21:01:59.387086 31399 tablet_bootstrap.cc:492] T 5899938341e44e15b437baab732d3ece P 5755a662553c4a84b76c8b2c48880567: No bootstrap required, opened a new log
I20251215 21:01:59.387168 31399 ts_tablet_manager.cc:1403] T 5899938341e44e15b437baab732d3ece P 5755a662553c4a84b76c8b2c48880567: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20251215 21:01:59.387128 31026 tablet_service.cc:1505] Processing CreateTablet for tablet 5899938341e44e15b437baab732d3ece (DEFAULT_TABLE table=test-table [id=f30ecf8355e64fbe95803bd920322b74]), partition=RANGE (key) PARTITION UNBOUNDED
I20251215 21:01:59.387388 31026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5899938341e44e15b437baab732d3ece. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 21:01:59.387615 31399 raft_consensus.cc:359] T 5899938341e44e15b437baab732d3ece P 5755a662553c4a84b76c8b2c48880567 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e18dca41b204cf0a3ebedd00ec07906" member_type: VOTER last_known_addr { host: "127.28.113.1" port: 33435 } } peers { permanent_uuid: "9899e8262dee40858d8880500e8054b1" member_type: VOTER last_known_addr { host: "127.28.113.2" port: 35261 } } peers { permanent_uuid: "5755a662553c4a84b76c8b2c48880567" member_type: VOTER last_known_addr { host: "127.28.113.3" port: 42057 } }
I20251215 21:01:59.387748 31399 raft_consensus.cc:385] T 5899938341e44e15b437baab732d3ece P 5755a662553c4a84b76c8b2c48880567 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251215 21:01:59.387779 31399 raft_consensus.cc:740] T 5899938341e44e15b437baab732d3ece P 5755a662553c4a84b76c8b2c48880567 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5755a662553c4a84b76c8b2c48880567, State: Initialized, Role: FOLLOWER
I20251215 21:01:59.387892 31399 consensus_queue.cc:260] T 5899938341e44e15b437baab732d3ece P 5755a662553c4a84b76c8b2c48880567 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e18dca41b204cf0a3ebedd00ec07906" member_type: VOTER last_known_addr { host: "127.28.113.1" port: 33435 } } peers { permanent_uuid: "9899e8262dee40858d8880500e8054b1" member_type: VOTER last_known_addr { host: "127.28.113.2" port: 35261 } } peers { permanent_uuid: "5755a662553c4a84b76c8b2c48880567" member_type: VOTER last_known_addr { host: "127.28.113.3" port: 42057 } }
I20251215 21:01:59.388288 31399 ts_tablet_manager.cc:1434] T 5899938341e44e15b437baab732d3ece P 5755a662553c4a84b76c8b2c48880567: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20251215 21:01:59.389504 31168 tablet_service.cc:1505] Processing CreateTablet for tablet 5899938341e44e15b437baab732d3ece (DEFAULT_TABLE table=test-table [id=f30ecf8355e64fbe95803bd920322b74]), partition=RANGE (key) PARTITION UNBOUNDED
I20251215 21:01:59.389773 31168 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5899938341e44e15b437baab732d3ece. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 21:01:59.391885 31400 tablet_bootstrap.cc:492] T 5899938341e44e15b437baab732d3ece P 9899e8262dee40858d8880500e8054b1: Bootstrap starting.
I20251215 21:01:59.392953 31400 tablet_bootstrap.cc:654] T 5899938341e44e15b437baab732d3ece P 9899e8262dee40858d8880500e8054b1: Neither blocks nor log segments found. Creating new log.
I20251215 21:01:59.394428 31400 tablet_bootstrap.cc:492] T 5899938341e44e15b437baab732d3ece P 9899e8262dee40858d8880500e8054b1: No bootstrap required, opened a new log
I20251215 21:01:59.394559 31400 ts_tablet_manager.cc:1403] T 5899938341e44e15b437baab732d3ece P 9899e8262dee40858d8880500e8054b1: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20251215 21:01:59.394850 31401 tablet_bootstrap.cc:492] T 5899938341e44e15b437baab732d3ece P 6e18dca41b204cf0a3ebedd00ec07906: Bootstrap starting.
I20251215 21:01:59.394953 31024 tablet_service.cc:1552] Processing DeleteTablet for tablet a24de5579422439992ad5b94ba3db3c2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-15 21:01:59 UTC) from {username='kudu', principal='kudu/127.28.113.62@KRBTEST.COM'} at 127.0.0.1:45748
I20251215 21:01:59.395073 31400 raft_consensus.cc:359] T 5899938341e44e15b437baab732d3ece P 9899e8262dee40858d8880500e8054b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e18dca41b204cf0a3ebedd00ec07906" member_type: VOTER last_known_addr { host: "127.28.113.1" port: 33435 } } peers { permanent_uuid: "9899e8262dee40858d8880500e8054b1" member_type: VOTER last_known_addr { host: "127.28.113.2" port: 35261 } } peers { permanent_uuid: "5755a662553c4a84b76c8b2c48880567" member_type: VOTER last_known_addr { host: "127.28.113.3" port: 42057 } }
I20251215 21:01:59.395174 31400 raft_consensus.cc:385] T 5899938341e44e15b437baab732d3ece P 9899e8262dee40858d8880500e8054b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251215 21:01:59.395206 31400 raft_consensus.cc:740] T 5899938341e44e15b437baab732d3ece P 9899e8262dee40858d8880500e8054b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9899e8262dee40858d8880500e8054b1, State: Initialized, Role: FOLLOWER
I20251215 21:01:59.395367 31400 consensus_queue.cc:260] T 5899938341e44e15b437baab732d3ece P 9899e8262dee40858d8880500e8054b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e18dca41b204cf0a3ebedd00ec07906" member_type: VOTER last_known_addr { host: "127.28.113.1" port: 33435 } } peers { permanent_uuid: "9899e8262dee40858d8880500e8054b1" member_type: VOTER last_known_addr { host: "127.28.113.2" port: 35261 } } peers { permanent_uuid: "5755a662553c4a84b76c8b2c48880567" member_type: VOTER last_known_addr { host: "127.28.113.3" port: 42057 } }
I20251215 21:01:59.395766 31401 tablet_bootstrap.cc:654] T 5899938341e44e15b437baab732d3ece P 6e18dca41b204cf0a3ebedd00ec07906: Neither blocks nor log segments found. Creating new log.
I20251215 21:01:59.397198 31401 tablet_bootstrap.cc:492] T 5899938341e44e15b437baab732d3ece P 6e18dca41b204cf0a3ebedd00ec07906: No bootstrap required, opened a new log
I20251215 21:01:59.397276 31401 ts_tablet_manager.cc:1403] T 5899938341e44e15b437baab732d3ece P 6e18dca41b204cf0a3ebedd00ec07906: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20251215 21:01:59.397336 31400 ts_tablet_manager.cc:1434] T 5899938341e44e15b437baab732d3ece P 9899e8262dee40858d8880500e8054b1: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20251215 21:01:59.397719 31401 raft_consensus.cc:359] T 5899938341e44e15b437baab732d3ece P 6e18dca41b204cf0a3ebedd00ec07906 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e18dca41b204cf0a3ebedd00ec07906" member_type: VOTER last_known_addr { host: "127.28.113.1" port: 33435 } } peers { permanent_uuid: "9899e8262dee40858d8880500e8054b1" member_type: VOTER last_known_addr { host: "127.28.113.2" port: 35261 } } peers { permanent_uuid: "5755a662553c4a84b76c8b2c48880567" member_type: VOTER last_known_addr { host: "127.28.113.3" port: 42057 } }
I20251215 21:01:59.397807 31401 raft_consensus.cc:385] T 5899938341e44e15b437baab732d3ece P 6e18dca41b204cf0a3ebedd00ec07906 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251215 21:01:59.397835 31401 raft_consensus.cc:740] T 5899938341e44e15b437baab732d3ece P 6e18dca41b204cf0a3ebedd00ec07906 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e18dca41b204cf0a3ebedd00ec07906, State: Initialized, Role: FOLLOWER
I20251215 21:01:59.397943 31401 consensus_queue.cc:260] T 5899938341e44e15b437baab732d3ece P 6e18dca41b204cf0a3ebedd00ec07906 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e18dca41b204cf0a3ebedd00ec07906" member_type: VOTER last_known_addr { host: "127.28.113.1" port: 33435 } } peers { permanent_uuid: "9899e8262dee40858d8880500e8054b1" member_type: VOTER last_known_addr { host: "127.28.113.2" port: 35261 } } peers { permanent_uuid: "5755a662553c4a84b76c8b2c48880567" member_type: VOTER last_known_addr { host: "127.28.113.3" port: 42057 } }
I20251215 21:01:59.398165 31401 ts_tablet_manager.cc:1434] T 5899938341e44e15b437baab732d3ece P 6e18dca41b204cf0a3ebedd00ec07906: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20251215 21:01:59.400679 31450 tablet_replica.cc:333] T a24de5579422439992ad5b94ba3db3c2 P 6e18dca41b204cf0a3ebedd00ec07906: stopping tablet replica
I20251215 21:01:59.400949 31450 raft_consensus.cc:2243] T a24de5579422439992ad5b94ba3db3c2 P 6e18dca41b204cf0a3ebedd00ec07906 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251215 21:01:59.401121 31450 raft_consensus.cc:2272] T a24de5579422439992ad5b94ba3db3c2 P 6e18dca41b204cf0a3ebedd00ec07906 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251215 21:01:59.402477 31450 ts_tablet_manager.cc:1916] T a24de5579422439992ad5b94ba3db3c2 P 6e18dca41b204cf0a3ebedd00ec07906: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251215 21:01:59.405925 31450 ts_tablet_manager.cc:1929] T a24de5579422439992ad5b94ba3db3c2 P 6e18dca41b204cf0a3ebedd00ec07906: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20251215 21:01:59.406039 31450 log.cc:1199] T a24de5579422439992ad5b94ba3db3c2 P 6e18dca41b204cf0a3ebedd00ec07906: Deleting WAL directory at /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765832506145157-29124-0/minicluster-data/ts-0/wal/wals/a24de5579422439992ad5b94ba3db3c2
I20251215 21:01:59.406391 31450 ts_tablet_manager.cc:1950] T a24de5579422439992ad5b94ba3db3c2 P 6e18dca41b204cf0a3ebedd00ec07906: Deleting consensus metadata
I20251215 21:01:59.407652 30872 catalog_manager.cc:4985] TS 6e18dca41b204cf0a3ebedd00ec07906 (127.28.113.1:33435): tablet a24de5579422439992ad5b94ba3db3c2 (table test-table [id=f68a68024bae4b39990c0e16c60f12f3]) successfully deleted
I20251215 21:01:59.408169 31446 tablet_replica.cc:333] T a24de5579422439992ad5b94ba3db3c2 P 9899e8262dee40858d8880500e8054b1: stopping tablet replica
I20251215 21:01:59.408448 31446 raft_consensus.cc:2243] T a24de5579422439992ad5b94ba3db3c2 P 9899e8262dee40858d8880500e8054b1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251215 21:01:59.408710 31446 raft_consensus.cc:2272] T a24de5579422439992ad5b94ba3db3c2 P 9899e8262dee40858d8880500e8054b1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251215 21:01:59.410055 31446 ts_tablet_manager.cc:1916] T a24de5579422439992ad5b94ba3db3c2 P 9899e8262dee40858d8880500e8054b1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251215 21:01:59.413112 31446 ts_tablet_manager.cc:1929] T a24de5579422439992ad5b94ba3db3c2 P 9899e8262dee40858d8880500e8054b1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20251215 21:01:59.413259 31446 log.cc:1199] T a24de5579422439992ad5b94ba3db3c2 P 9899e8262dee40858d8880500e8054b1: Deleting WAL directory at /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765832506145157-29124-0/minicluster-data/ts-1/wal/wals/a24de5579422439992ad5b94ba3db3c2
I20251215 21:01:59.413622 31446 ts_tablet_manager.cc:1950] T a24de5579422439992ad5b94ba3db3c2 P 9899e8262dee40858d8880500e8054b1: Deleting consensus metadata
I20251215 21:01:59.414906 30872 catalog_manager.cc:4985] TS 9899e8262dee40858d8880500e8054b1 (127.28.113.2:35261): tablet a24de5579422439992ad5b94ba3db3c2 (table test-table [id=f68a68024bae4b39990c0e16c60f12f3]) successfully deleted
I20251215 21:01:59.697496 31405 raft_consensus.cc:493] T 5899938341e44e15b437baab732d3ece P 9899e8262dee40858d8880500e8054b1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 21:01:59.697643 31405 raft_consensus.cc:515] T 5899938341e44e15b437baab732d3ece P 9899e8262dee40858d8880500e8054b1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e18dca41b204cf0a3ebedd00ec07906" member_type: VOTER last_known_addr { host: "127.28.113.1" port: 33435 } } peers { permanent_uuid: "9899e8262dee40858d8880500e8054b1" member_type: VOTER last_known_addr { host: "127.28.113.2" port: 35261 } } peers { permanent_uuid: "5755a662553c4a84b76c8b2c48880567" member_type: VOTER last_known_addr { host: "127.28.113.3" port: 42057 } }
I20251215 21:01:59.698668 31405 leader_election.cc:290] T 5899938341e44e15b437baab732d3ece P 9899e8262dee40858d8880500e8054b1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6e18dca41b204cf0a3ebedd00ec07906 (127.28.113.1:33435), 5755a662553c4a84b76c8b2c48880567 (127.28.113.3:42057)
I20251215 21:01:59.701539 31453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:01:59.698889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.113.1:33435 (local address 127.28.113.2:60927)
1215 21:01:59.699190 (+ 301us) negotiation.cc:107] Waiting for socket to connect
1215 21:01:59.699204 (+ 14us) client_negotiation.cc:174] Beginning negotiation
1215 21:01:59.699300 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1215 21:01:59.699539 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1215 21:01:59.699542 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1215 21:01:59.699556 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1215 21:01:59.699794 (+ 238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:01:59.699801 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:01:59.700505 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1215 21:01:59.700509 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1215 21:01:59.701147 (+ 638us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:01:59.701155 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:01:59.701255 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:01:59.701278 (+ 23us) client_negotiation.cc:769] Sending connection context
1215 21:01:59.701326 (+ 48us) client_negotiation.cc:240] Negotiation successful
1215 21:01:59.701383 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":72,"threads_started":1}
I20251215 21:01:59.701552 31452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:01:59.698919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.113.3:42057 (local address 127.28.113.2:55835)
1215 21:01:59.699277 (+ 358us) negotiation.cc:107] Waiting for socket to connect
1215 21:01:59.699290 (+ 13us) client_negotiation.cc:174] Beginning negotiation
1215 21:01:59.699391 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1215 21:01:59.699632 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1215 21:01:59.699635 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1215 21:01:59.699649 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1215 21:01:59.699832 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:01:59.699837 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:01:59.700599 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1215 21:01:59.700602 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1215 21:01:59.701160 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:01:59.701167 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:01:59.701276 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:01:59.701295 (+ 19us) client_negotiation.cc:769] Sending connection context
1215 21:01:59.701344 (+ 49us) client_negotiation.cc:240] Negotiation successful
1215 21:01:59.701398 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":296,"thread_start_us":75,"threads_started":1}
I20251215 21:01:59.701957 31398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:01:59.698981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.113.2:60927 (local address 127.28.113.1:33435)
1215 21:01:59.699115 (+ 134us) server_negotiation.cc:207] Beginning negotiation
1215 21:01:59.699119 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1215 21:01:59.699276 (+ 157us) server_negotiation.cc:408] Connection header received
1215 21:01:59.699393 (+ 117us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1215 21:01:59.699396 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1215 21:01:59.699449 (+ 53us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1215 21:01:59.699515 (+ 66us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1215 21:01:59.699902 (+ 387us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:01:59.700390 (+ 488us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1215 21:01:59.701261 (+ 871us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:01:59.701687 (+ 426us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:01:59.701728 (+ 41us) server_negotiation.cc:1036] Waiting for connection context
1215 21:01:59.701792 (+ 64us) server_negotiation.cc:300] Negotiation successful
1215 21:01:59.701845 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20251215 21:01:59.701988 31397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:01:59.699028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.113.2:55835 (local address 127.28.113.3:42057)
1215 21:01:59.699150 (+ 122us) server_negotiation.cc:207] Beginning negotiation
1215 21:01:59.699154 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1215 21:01:59.699369 (+ 215us) server_negotiation.cc:408] Connection header received
1215 21:01:59.699481 (+ 112us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1215 21:01:59.699484 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1215 21:01:59.699536 (+ 52us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1215 21:01:59.699619 (+ 83us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1215 21:01:59.700016 (+ 397us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:01:59.700482 (+ 466us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1215 21:01:59.701270 (+ 788us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:01:59.701740 (+ 470us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:01:59.701779 (+ 39us) server_negotiation.cc:1036] Waiting for connection context
1215 21:01:59.701829 (+ 50us) server_negotiation.cc:300] Negotiation successful
1215 21:01:59.701887 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20251215 21:01:59.702353 31330 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5899938341e44e15b437baab732d3ece" candidate_uuid: "9899e8262dee40858d8880500e8054b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5755a662553c4a84b76c8b2c48880567" is_pre_election: true
I20251215 21:01:59.702353 31045 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5899938341e44e15b437baab732d3ece" candidate_uuid: "9899e8262dee40858d8880500e8054b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e18dca41b204cf0a3ebedd00ec07906" is_pre_election: true
I20251215 21:01:59.702509 31045 raft_consensus.cc:2468] T 5899938341e44e15b437baab732d3ece P 6e18dca41b204cf0a3ebedd00ec07906 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9899e8262dee40858d8880500e8054b1 in term 0.
I20251215 21:01:59.702509 31330 raft_consensus.cc:2468] T 5899938341e44e15b437baab732d3ece P 5755a662553c4a84b76c8b2c48880567 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9899e8262dee40858d8880500e8054b1 in term 0.
I20251215 21:01:59.702893 31120 leader_election.cc:304] T 5899938341e44e15b437baab732d3ece P 9899e8262dee40858d8880500e8054b1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 5755a662553c4a84b76c8b2c48880567, 6e18dca41b204cf0a3ebedd00ec07906, 9899e8262dee40858d8880500e8054b1; no voters:
I20251215 21:01:59.703133 31405 raft_consensus.cc:2804] T 5899938341e44e15b437baab732d3ece P 9899e8262dee40858d8880500e8054b1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251215 21:01:59.703188 31405 raft_consensus.cc:493] T 5899938341e44e15b437baab732d3ece P 9899e8262dee40858d8880500e8054b1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 21:01:59.703214 31405 raft_consensus.cc:3060] T 5899938341e44e15b437baab732d3ece P 9899e8262dee40858d8880500e8054b1 [term 0 FOLLOWER]: Advancing to term 1
I20251215 21:01:59.704134 31405 raft_consensus.cc:515] T 5899938341e44e15b437baab732d3ece P 9899e8262dee40858d8880500e8054b1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e18dca41b204cf0a3ebedd00ec07906" member_type: VOTER last_known_addr { host: "127.28.113.1" port: 33435 } } peers { permanent_uuid: "9899e8262dee40858d8880500e8054b1" member_type: VOTER last_known_addr { host: "127.28.113.2" port: 35261 } } peers { permanent_uuid: "5755a662553c4a84b76c8b2c48880567" member_type: VOTER last_known_addr { host: "127.28.113.3" port: 42057 } }
I20251215 21:01:59.704455 31405 leader_election.cc:290] T 5899938341e44e15b437baab732d3ece P 9899e8262dee40858d8880500e8054b1 [CANDIDATE]: Term 1 election: Requested vote from peers 6e18dca41b204cf0a3ebedd00ec07906 (127.28.113.1:33435), 5755a662553c4a84b76c8b2c48880567 (127.28.113.3:42057)
I20251215 21:01:59.704761 31045 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5899938341e44e15b437baab732d3ece" candidate_uuid: "9899e8262dee40858d8880500e8054b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e18dca41b204cf0a3ebedd00ec07906"
I20251215 21:01:59.704792 31330 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5899938341e44e15b437baab732d3ece" candidate_uuid: "9899e8262dee40858d8880500e8054b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5755a662553c4a84b76c8b2c48880567"
I20251215 21:01:59.704869 31045 raft_consensus.cc:3060] T 5899938341e44e15b437baab732d3ece P 6e18dca41b204cf0a3ebedd00ec07906 [term 0 FOLLOWER]: Advancing to term 1
I20251215 21:01:59.704895 31330 raft_consensus.cc:3060] T 5899938341e44e15b437baab732d3ece P 5755a662553c4a84b76c8b2c48880567 [term 0 FOLLOWER]: Advancing to term 1
I20251215 21:01:59.705651 31045 raft_consensus.cc:2468] T 5899938341e44e15b437baab732d3ece P 6e18dca41b204cf0a3ebedd00ec07906 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9899e8262dee40858d8880500e8054b1 in term 1.
I20251215 21:01:59.705665 31330 raft_consensus.cc:2468] T 5899938341e44e15b437baab732d3ece P 5755a662553c4a84b76c8b2c48880567 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9899e8262dee40858d8880500e8054b1 in term 1.
I20251215 21:01:59.705927 31121 leader_election.cc:304] T 5899938341e44e15b437baab732d3ece P 9899e8262dee40858d8880500e8054b1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5755a662553c4a84b76c8b2c48880567, 9899e8262dee40858d8880500e8054b1; no voters:
I20251215 21:01:59.706085 31405 raft_consensus.cc:2804] T 5899938341e44e15b437baab732d3ece P 9899e8262dee40858d8880500e8054b1 [term 1 FOLLOWER]: Leader election won for term 1
I20251215 21:01:59.706238 31405 raft_consensus.cc:697] T 5899938341e44e15b437baab732d3ece P 9899e8262dee40858d8880500e8054b1 [term 1 LEADER]: Becoming Leader. State: Replica: 9899e8262dee40858d8880500e8054b1, State: Running, Role: LEADER
I20251215 21:01:59.706450 31405 consensus_queue.cc:237] T 5899938341e44e15b437baab732d3ece P 9899e8262dee40858d8880500e8054b1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e18dca41b204cf0a3ebedd00ec07906" member_type: VOTER last_known_addr { host: "127.28.113.1" port: 33435 } } peers { permanent_uuid: "9899e8262dee40858d8880500e8054b1" member_type: VOTER last_known_addr { host: "127.28.113.2" port: 35261 } } peers { permanent_uuid: "5755a662553c4a84b76c8b2c48880567" member_type: VOTER last_known_addr { host: "127.28.113.3" port: 42057 } }
I20251215 21:01:59.708999 30886 catalog_manager.cc:5654] T 5899938341e44e15b437baab732d3ece P 9899e8262dee40858d8880500e8054b1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9899e8262dee40858d8880500e8054b1 (127.28.113.2). New cstate: current_term: 1 leader_uuid: "9899e8262dee40858d8880500e8054b1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e18dca41b204cf0a3ebedd00ec07906" member_type: VOTER last_known_addr { host: "127.28.113.1" port: 33435 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9899e8262dee40858d8880500e8054b1" member_type: VOTER last_known_addr { host: "127.28.113.2" port: 35261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5755a662553c4a84b76c8b2c48880567" member_type: VOTER last_known_addr { host: "127.28.113.3" port: 42057 } health_report { overall_health: UNKNOWN } } }
I20251215 21:01:59.771759 31456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:01:59.768778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34122 (local address 127.28.113.2:35261)
1215 21:01:59.769025 (+ 247us) server_negotiation.cc:207] Beginning negotiation
1215 21:01:59.769029 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1215 21:01:59.769042 (+ 13us) server_negotiation.cc:408] Connection header received
1215 21:01:59.769110 (+ 68us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1215 21:01:59.769112 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1215 21:01:59.769162 (+ 50us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1215 21:01:59.769252 (+ 90us) server_negotiation.cc:227] Negotiated authn=TOKEN
1215 21:01:59.769583 (+ 331us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:01:59.770078 (+ 495us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1215 21:01:59.770688 (+ 610us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:01:59.770873 (+ 185us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:01:59.770927 (+ 54us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1215 21:01:59.771368 (+ 441us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1215 21:01:59.771476 (+ 108us) server_negotiation.cc:1036] Waiting for connection context
1215 21:01:59.771560 (+ 84us) server_negotiation.cc:300] Negotiation successful
1215 21:01:59.771608 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":172,"thread_start_us":70,"threads_started":1}
I20251215 21:01:59.780604 31398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:01:59.777981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.113.2:43503 (local address 127.28.113.1:33435)
1215 21:01:59.778113 (+ 132us) server_negotiation.cc:207] Beginning negotiation
1215 21:01:59.778117 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1215 21:01:59.778238 (+ 121us) server_negotiation.cc:408] Connection header received
1215 21:01:59.778371 (+ 133us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1215 21:01:59.778374 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1215 21:01:59.778419 (+ 45us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1215 21:01:59.778484 (+ 65us) server_negotiation.cc:227] Negotiated authn=TOKEN
1215 21:01:59.778863 (+ 379us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:01:59.779315 (+ 452us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1215 21:01:59.779985 (+ 670us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:01:59.780159 (+ 174us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:01:59.780199 (+ 40us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1215 21:01:59.780264 (+ 65us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1215 21:01:59.780325 (+ 61us) server_negotiation.cc:1036] Waiting for connection context
1215 21:01:59.780427 (+ 102us) server_negotiation.cc:300] Negotiation successful
1215 21:01:59.780470 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20251215 21:01:59.780797 31459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:01:59.777879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.113.1:33435 (local address 127.28.113.2:43503)
1215 21:01:59.778160 (+ 281us) negotiation.cc:107] Waiting for socket to connect
1215 21:01:59.778174 (+ 14us) client_negotiation.cc:174] Beginning negotiation
1215 21:01:59.778273 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1215 21:01:59.778501 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1215 21:01:59.778504 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1215 21:01:59.778512 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
1215 21:01:59.778758 (+ 246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:01:59.778764 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:01:59.779427 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1215 21:01:59.779430 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1215 21:01:59.779855 (+ 425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:01:59.779862 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:01:59.779986 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:01:59.780032 (+ 46us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
1215 21:01:59.780349 (+ 317us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
1215 21:01:59.780356 (+ 7us) client_negotiation.cc:769] Sending connection context
1215 21:01:59.780411 (+ 55us) client_negotiation.cc:240] Negotiation successful
1215 21:01:59.780479 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":93,"threads_started":1}
I20251215 21:01:59.786199 31330 raft_consensus.cc:1275] T 5899938341e44e15b437baab732d3ece P 5755a662553c4a84b76c8b2c48880567 [term 1 FOLLOWER]: Refusing update from remote peer 9899e8262dee40858d8880500e8054b1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251215 21:01:59.786288 31045 raft_consensus.cc:1275] T 5899938341e44e15b437baab732d3ece P 6e18dca41b204cf0a3ebedd00ec07906 [term 1 FOLLOWER]: Refusing update from remote peer 9899e8262dee40858d8880500e8054b1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251215 21:01:59.786655 31405 consensus_queue.cc:1048] T 5899938341e44e15b437baab732d3ece P 9899e8262dee40858d8880500e8054b1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e18dca41b204cf0a3ebedd00ec07906" member_type: VOTER last_known_addr { host: "127.28.113.1" port: 33435 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251215 21:01:59.787010 31454 consensus_queue.cc:1048] T 5899938341e44e15b437baab732d3ece P 9899e8262dee40858d8880500e8054b1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5755a662553c4a84b76c8b2c48880567" member_type: VOTER last_known_addr { host: "127.28.113.3" port: 42057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251215 21:01:59.802443 31442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:01:59.800029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.113.2:35261 (local address 127.28.113.1:51617)
1215 21:01:59.800143 (+ 114us) negotiation.cc:107] Waiting for socket to connect
1215 21:01:59.800159 (+ 16us) client_negotiation.cc:174] Beginning negotiation
1215 21:01:59.800239 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1215 21:01:59.800477 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1215 21:01:59.800480 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1215 21:01:59.800488 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
1215 21:01:59.800666 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:01:59.800671 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:01:59.801378 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1215 21:01:59.801381 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1215 21:01:59.801769 (+ 388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:01:59.801786 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:01:59.801868 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:01:59.801890 (+ 22us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
1215 21:01:59.802200 (+ 310us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
1215 21:01:59.802204 (+ 4us) client_negotiation.cc:769] Sending connection context
1215 21:01:59.802259 (+ 55us) client_negotiation.cc:240] Negotiation successful
1215 21:01:59.802329 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26}
I20251215 21:01:59.802443 31456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:01:59.800068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.113.1:51617 (local address 127.28.113.2:35261)
1215 21:01:59.800175 (+ 107us) server_negotiation.cc:207] Beginning negotiation
1215 21:01:59.800180 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1215 21:01:59.800195 (+ 15us) server_negotiation.cc:408] Connection header received
1215 21:01:59.800333 (+ 138us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1215 21:01:59.800336 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1215 21:01:59.800388 (+ 52us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1215 21:01:59.800453 (+ 65us) server_negotiation.cc:227] Negotiated authn=TOKEN
1215 21:01:59.800747 (+ 294us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:01:59.801283 (+ 536us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1215 21:01:59.801863 (+ 580us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:01:59.802007 (+ 144us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:01:59.802049 (+ 42us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1215 21:01:59.802123 (+ 74us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1215 21:01:59.802195 (+ 72us) server_negotiation.cc:1036] Waiting for connection context
1215 21:01:59.802271 (+ 76us) server_negotiation.cc:300] Negotiation successful
1215 21:01:59.802313 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
W20251215 21:01:59.803040 31429 tablet_replica.cc:1306] Aborted: operation has been aborted: cancelling pending write operations
I20251215 21:02:01.356786 30887 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56856:
table { table_name: "test-table" } modify_external_catalogs: true
I20251215 21:02:01.356940 30887 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56856:
table { table_name: "test-table" } modify_external_catalogs: true
I20251215 21:02:01.359270 30887 catalog_manager.cc:5941] T 00000000000000000000000000000000 P 8d6aede0d59c4dbd8c8bca7ade69344b: Sending DeleteTablet for 3 replicas of tablet 5899938341e44e15b437baab732d3ece
I20251215 21:02:01.359813 31024 tablet_service.cc:1552] Processing DeleteTablet for tablet 5899938341e44e15b437baab732d3ece with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-15 21:02:01 UTC) from {username='kudu', principal='kudu/127.28.113.62@KRBTEST.COM'} at 127.0.0.1:45748
I20251215 21:02:01.359822 31168 tablet_service.cc:1552] Processing DeleteTablet for tablet 5899938341e44e15b437baab732d3ece with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-15 21:02:01 UTC) from {username='kudu', principal='kudu/127.28.113.62@KRBTEST.COM'} at 127.0.0.1:34108
I20251215 21:02:01.360193 31488 tablet_replica.cc:333] T 5899938341e44e15b437baab732d3ece P 6e18dca41b204cf0a3ebedd00ec07906: stopping tablet replica
I20251215 21:02:01.360273 31487 tablet_replica.cc:333] T 5899938341e44e15b437baab732d3ece P 9899e8262dee40858d8880500e8054b1: stopping tablet replica
I20251215 21:02:01.360335 31488 raft_consensus.cc:2243] T 5899938341e44e15b437baab732d3ece P 6e18dca41b204cf0a3ebedd00ec07906 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251215 21:02:01.360391 31487 raft_consensus.cc:2243] T 5899938341e44e15b437baab732d3ece P 9899e8262dee40858d8880500e8054b1 [term 1 LEADER]: Raft consensus shutting down.
I20251215 21:02:01.360472 31488 raft_consensus.cc:2272] T 5899938341e44e15b437baab732d3ece P 6e18dca41b204cf0a3ebedd00ec07906 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251215 21:02:01.360689 31487 raft_consensus.cc:2272] T 5899938341e44e15b437baab732d3ece P 9899e8262dee40858d8880500e8054b1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251215 21:02:01.361219 31487 ts_tablet_manager.cc:1916] T 5899938341e44e15b437baab732d3ece P 9899e8262dee40858d8880500e8054b1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251215 21:02:01.361805 31310 tablet_service.cc:1552] Processing DeleteTablet for tablet 5899938341e44e15b437baab732d3ece with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-15 21:02:01 UTC) from {username='kudu', principal='kudu/127.28.113.62@KRBTEST.COM'} at 127.0.0.1:40368
I20251215 21:02:01.362191 31490 tablet_replica.cc:333] T 5899938341e44e15b437baab732d3ece P 5755a662553c4a84b76c8b2c48880567: stopping tablet replica
I20251215 21:02:01.362342 31490 raft_consensus.cc:2243] T 5899938341e44e15b437baab732d3ece P 5755a662553c4a84b76c8b2c48880567 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251215 21:02:01.362527 31490 raft_consensus.cc:2272] T 5899938341e44e15b437baab732d3ece P 5755a662553c4a84b76c8b2c48880567 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251215 21:02:01.363152 31490 ts_tablet_manager.cc:1916] T 5899938341e44e15b437baab732d3ece P 5755a662553c4a84b76c8b2c48880567: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251215 21:02:01.363703 31487 ts_tablet_manager.cc:1929] T 5899938341e44e15b437baab732d3ece P 9899e8262dee40858d8880500e8054b1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20251215 21:02:01.363806 31487 log.cc:1199] T 5899938341e44e15b437baab732d3ece P 9899e8262dee40858d8880500e8054b1: Deleting WAL directory at /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765832506145157-29124-0/minicluster-data/ts-1/wal/wals/5899938341e44e15b437baab732d3ece
I20251215 21:02:01.363816 31488 ts_tablet_manager.cc:1916] T 5899938341e44e15b437baab732d3ece P 6e18dca41b204cf0a3ebedd00ec07906: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251215 21:02:01.364163 31487 ts_tablet_manager.cc:1950] T 5899938341e44e15b437baab732d3ece P 9899e8262dee40858d8880500e8054b1: Deleting consensus metadata
I20251215 21:02:01.364892 30872 catalog_manager.cc:4985] TS 9899e8262dee40858d8880500e8054b1 (127.28.113.2:35261): tablet 5899938341e44e15b437baab732d3ece (table test-table [id=f30ecf8355e64fbe95803bd920322b74]) successfully deleted
I20251215 21:02:01.365448 31490 ts_tablet_manager.cc:1929] T 5899938341e44e15b437baab732d3ece P 5755a662553c4a84b76c8b2c48880567: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20251215 21:02:01.365510 31490 log.cc:1199] T 5899938341e44e15b437baab732d3ece P 5755a662553c4a84b76c8b2c48880567: Deleting WAL directory at /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765832506145157-29124-0/minicluster-data/ts-2/wal/wals/5899938341e44e15b437baab732d3ece
I20251215 21:02:01.365777 31490 ts_tablet_manager.cc:1950] T 5899938341e44e15b437baab732d3ece P 5755a662553c4a84b76c8b2c48880567: Deleting consensus metadata
I20251215 21:02:01.366328 30873 catalog_manager.cc:4985] TS 5755a662553c4a84b76c8b2c48880567 (127.28.113.3:42057): tablet 5899938341e44e15b437baab732d3ece (table test-table [id=f30ecf8355e64fbe95803bd920322b74]) successfully deleted
I20251215 21:02:01.366480 31488 ts_tablet_manager.cc:1929] T 5899938341e44e15b437baab732d3ece P 6e18dca41b204cf0a3ebedd00ec07906: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20251215 21:02:01.366564 31488 log.cc:1199] T 5899938341e44e15b437baab732d3ece P 6e18dca41b204cf0a3ebedd00ec07906: Deleting WAL directory at /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765832506145157-29124-0/minicluster-data/ts-0/wal/wals/5899938341e44e15b437baab732d3ece
I20251215 21:02:01.366833 31488 ts_tablet_manager.cc:1950] T 5899938341e44e15b437baab732d3ece P 6e18dca41b204cf0a3ebedd00ec07906: Deleting consensus metadata
I20251215 21:02:01.367465 30872 catalog_manager.cc:4985] TS 6e18dca41b204cf0a3ebedd00ec07906 (127.28.113.1:33435): tablet 5899938341e44e15b437baab732d3ece (table test-table [id=f30ecf8355e64fbe95803bd920322b74]) successfully deleted
Dec 15 21:02:01 dist-test-slave-bjsf krb5kdc[30839](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765832521, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Dec 15 21:02:01 dist-test-slave-bjsf krb5kdc[30839](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765832521, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.28.113.62@KRBTEST.COM
I20251215 21:02:01.386631 31497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:02:01.376641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56882 (local address 127.28.113.62:37805)
1215 21:02:01.376907 (+ 266us) server_negotiation.cc:207] Beginning negotiation
1215 21:02:01.376910 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1215 21:02:01.376923 (+ 13us) server_negotiation.cc:408] Connection header received
1215 21:02:01.376961 (+ 38us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1215 21:02:01.376963 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1215 21:02:01.377005 (+ 42us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1215 21:02:01.377077 (+ 72us) server_negotiation.cc:227] Negotiated authn=SASL
1215 21:02:01.377774 (+ 697us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:02:01.378299 (+ 525us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1215 21:02:01.378856 (+ 557us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:02:01.378986 (+ 130us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:02:01.383812 (+ 4826us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1215 21:02:01.383831 (+ 19us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1215 21:02:01.383833 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1215 21:02:01.383866 (+ 33us) server_negotiation.cc:923] calling sasl_server_start()
1215 21:02:01.385294 (+ 1428us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1215 21:02:01.385644 (+ 350us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1215 21:02:01.385648 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1215 21:02:01.385650 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1215 21:02:01.385691 (+ 41us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1215 21:02:01.385883 (+ 192us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1215 21:02:01.385886 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1215 21:02:01.385888 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1215 21:02:01.386042 (+ 154us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1215 21:02:01.386180 (+ 138us) server_negotiation.cc:1036] Waiting for connection context
1215 21:02:01.386353 (+ 173us) server_negotiation.cc:300] Negotiation successful
1215 21:02:01.386470 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":185,"thread_start_us":100,"threads_started":1}
I20251215 21:02:01.388857 30887 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56882:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251215 21:02:01.389377 30887 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251215 21:02:01.393594 31024 tablet_service.cc:1505] Processing CreateTablet for tablet ddfd6a00699547f3829226d6b549cb02 (DEFAULT_TABLE table=test-table [id=5344a7c089634484bbe85ad4a56e3356]), partition=RANGE (key) PARTITION UNBOUNDED
I20251215 21:02:01.393888 31024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ddfd6a00699547f3829226d6b549cb02. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 21:02:01.393854 31168 tablet_service.cc:1505] Processing CreateTablet for tablet ddfd6a00699547f3829226d6b549cb02 (DEFAULT_TABLE table=test-table [id=5344a7c089634484bbe85ad4a56e3356]), partition=RANGE (key) PARTITION UNBOUNDED
I20251215 21:02:01.393890 31310 tablet_service.cc:1505] Processing CreateTablet for tablet ddfd6a00699547f3829226d6b549cb02 (DEFAULT_TABLE table=test-table [id=5344a7c089634484bbe85ad4a56e3356]), partition=RANGE (key) PARTITION UNBOUNDED
I20251215 21:02:01.394099 31168 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ddfd6a00699547f3829226d6b549cb02. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 21:02:01.394165 31310 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ddfd6a00699547f3829226d6b549cb02. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 21:02:01.396237 31500 tablet_bootstrap.cc:492] T ddfd6a00699547f3829226d6b549cb02 P 6e18dca41b204cf0a3ebedd00ec07906: Bootstrap starting.
I20251215 21:02:01.396360 31498 tablet_bootstrap.cc:492] T ddfd6a00699547f3829226d6b549cb02 P 5755a662553c4a84b76c8b2c48880567: Bootstrap starting.
I20251215 21:02:01.396696 31499 tablet_bootstrap.cc:492] T ddfd6a00699547f3829226d6b549cb02 P 9899e8262dee40858d8880500e8054b1: Bootstrap starting.
I20251215 21:02:01.397161 31498 tablet_bootstrap.cc:654] T ddfd6a00699547f3829226d6b549cb02 P 5755a662553c4a84b76c8b2c48880567: Neither blocks nor log segments found. Creating new log.
I20251215 21:02:01.397161 31500 tablet_bootstrap.cc:654] T ddfd6a00699547f3829226d6b549cb02 P 6e18dca41b204cf0a3ebedd00ec07906: Neither blocks nor log segments found. Creating new log.
I20251215 21:02:01.397467 31499 tablet_bootstrap.cc:654] T ddfd6a00699547f3829226d6b549cb02 P 9899e8262dee40858d8880500e8054b1: Neither blocks nor log segments found. Creating new log.
I20251215 21:02:01.398308 31500 tablet_bootstrap.cc:492] T ddfd6a00699547f3829226d6b549cb02 P 6e18dca41b204cf0a3ebedd00ec07906: No bootstrap required, opened a new log
I20251215 21:02:01.398386 31500 ts_tablet_manager.cc:1403] T ddfd6a00699547f3829226d6b549cb02 P 6e18dca41b204cf0a3ebedd00ec07906: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20251215 21:02:01.398520 31499 tablet_bootstrap.cc:492] T ddfd6a00699547f3829226d6b549cb02 P 9899e8262dee40858d8880500e8054b1: No bootstrap required, opened a new log
I20251215 21:02:01.398520 31498 tablet_bootstrap.cc:492] T ddfd6a00699547f3829226d6b549cb02 P 5755a662553c4a84b76c8b2c48880567: No bootstrap required, opened a new log
I20251215 21:02:01.398610 31499 ts_tablet_manager.cc:1403] T ddfd6a00699547f3829226d6b549cb02 P 9899e8262dee40858d8880500e8054b1: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20251215 21:02:01.398605 31498 ts_tablet_manager.cc:1403] T ddfd6a00699547f3829226d6b549cb02 P 5755a662553c4a84b76c8b2c48880567: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20251215 21:02:01.398772 31500 raft_consensus.cc:359] T ddfd6a00699547f3829226d6b549cb02 P 6e18dca41b204cf0a3ebedd00ec07906 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e18dca41b204cf0a3ebedd00ec07906" member_type: VOTER last_known_addr { host: "127.28.113.1" port: 33435 } } peers { permanent_uuid: "9899e8262dee40858d8880500e8054b1" member_type: VOTER last_known_addr { host: "127.28.113.2" port: 35261 } } peers { permanent_uuid: "5755a662553c4a84b76c8b2c48880567" member_type: VOTER last_known_addr { host: "127.28.113.3" port: 42057 } }
I20251215 21:02:01.398865 31500 raft_consensus.cc:385] T ddfd6a00699547f3829226d6b549cb02 P 6e18dca41b204cf0a3ebedd00ec07906 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251215 21:02:01.398921 31500 raft_consensus.cc:740] T ddfd6a00699547f3829226d6b549cb02 P 6e18dca41b204cf0a3ebedd00ec07906 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e18dca41b204cf0a3ebedd00ec07906, State: Initialized, Role: FOLLOWER
I20251215 21:02:01.399022 31498 raft_consensus.cc:359] T ddfd6a00699547f3829226d6b549cb02 P 5755a662553c4a84b76c8b2c48880567 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e18dca41b204cf0a3ebedd00ec07906" member_type: VOTER last_known_addr { host: "127.28.113.1" port: 33435 } } peers { permanent_uuid: "9899e8262dee40858d8880500e8054b1" member_type: VOTER last_known_addr { host: "127.28.113.2" port: 35261 } } peers { permanent_uuid: "5755a662553c4a84b76c8b2c48880567" member_type: VOTER last_known_addr { host: "127.28.113.3" port: 42057 } }
I20251215 21:02:01.399022 31499 raft_consensus.cc:359] T ddfd6a00699547f3829226d6b549cb02 P 9899e8262dee40858d8880500e8054b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e18dca41b204cf0a3ebedd00ec07906" member_type: VOTER last_known_addr { host: "127.28.113.1" port: 33435 } } peers { permanent_uuid: "9899e8262dee40858d8880500e8054b1" member_type: VOTER last_known_addr { host: "127.28.113.2" port: 35261 } } peers { permanent_uuid: "5755a662553c4a84b76c8b2c48880567" member_type: VOTER last_known_addr { host: "127.28.113.3" port: 42057 } }
I20251215 21:02:01.399070 31500 consensus_queue.cc:260] T ddfd6a00699547f3829226d6b549cb02 P 6e18dca41b204cf0a3ebedd00ec07906 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e18dca41b204cf0a3ebedd00ec07906" member_type: VOTER last_known_addr { host: "127.28.113.1" port: 33435 } } peers { permanent_uuid: "9899e8262dee40858d8880500e8054b1" member_type: VOTER last_known_addr { host: "127.28.113.2" port: 35261 } } peers { permanent_uuid: "5755a662553c4a84b76c8b2c48880567" member_type: VOTER last_known_addr { host: "127.28.113.3" port: 42057 } }
I20251215 21:02:01.399127 31498 raft_consensus.cc:385] T ddfd6a00699547f3829226d6b549cb02 P 5755a662553c4a84b76c8b2c48880567 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251215 21:02:01.399127 31499 raft_consensus.cc:385] T ddfd6a00699547f3829226d6b549cb02 P 9899e8262dee40858d8880500e8054b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251215 21:02:01.399165 31498 raft_consensus.cc:740] T ddfd6a00699547f3829226d6b549cb02 P 5755a662553c4a84b76c8b2c48880567 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5755a662553c4a84b76c8b2c48880567, State: Initialized, Role: FOLLOWER
I20251215 21:02:01.399165 31499 raft_consensus.cc:740] T ddfd6a00699547f3829226d6b549cb02 P 9899e8262dee40858d8880500e8054b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9899e8262dee40858d8880500e8054b1, State: Initialized, Role: FOLLOWER
I20251215 21:02:01.399282 31498 consensus_queue.cc:260] T ddfd6a00699547f3829226d6b549cb02 P 5755a662553c4a84b76c8b2c48880567 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e18dca41b204cf0a3ebedd00ec07906" member_type: VOTER last_known_addr { host: "127.28.113.1" port: 33435 } } peers { permanent_uuid: "9899e8262dee40858d8880500e8054b1" member_type: VOTER last_known_addr { host: "127.28.113.2" port: 35261 } } peers { permanent_uuid: "5755a662553c4a84b76c8b2c48880567" member_type: VOTER last_known_addr { host: "127.28.113.3" port: 42057 } }
I20251215 21:02:01.399338 31499 consensus_queue.cc:260] T ddfd6a00699547f3829226d6b549cb02 P 9899e8262dee40858d8880500e8054b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e18dca41b204cf0a3ebedd00ec07906" member_type: VOTER last_known_addr { host: "127.28.113.1" port: 33435 } } peers { permanent_uuid: "9899e8262dee40858d8880500e8054b1" member_type: VOTER last_known_addr { host: "127.28.113.2" port: 35261 } } peers { permanent_uuid: "5755a662553c4a84b76c8b2c48880567" member_type: VOTER last_known_addr { host: "127.28.113.3" port: 42057 } }
I20251215 21:02:01.399381 31500 ts_tablet_manager.cc:1434] T ddfd6a00699547f3829226d6b549cb02 P 6e18dca41b204cf0a3ebedd00ec07906: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20251215 21:02:01.399578 31499 ts_tablet_manager.cc:1434] T ddfd6a00699547f3829226d6b549cb02 P 9899e8262dee40858d8880500e8054b1: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20251215 21:02:01.399592 31498 ts_tablet_manager.cc:1434] T ddfd6a00699547f3829226d6b549cb02 P 5755a662553c4a84b76c8b2c48880567: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20251215 21:02:01.454596 31504 raft_consensus.cc:493] T ddfd6a00699547f3829226d6b549cb02 P 5755a662553c4a84b76c8b2c48880567 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 21:02:01.454725 31504 raft_consensus.cc:515] T ddfd6a00699547f3829226d6b549cb02 P 5755a662553c4a84b76c8b2c48880567 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e18dca41b204cf0a3ebedd00ec07906" member_type: VOTER last_known_addr { host: "127.28.113.1" port: 33435 } } peers { permanent_uuid: "9899e8262dee40858d8880500e8054b1" member_type: VOTER last_known_addr { host: "127.28.113.2" port: 35261 } } peers { permanent_uuid: "5755a662553c4a84b76c8b2c48880567" member_type: VOTER last_known_addr { host: "127.28.113.3" port: 42057 } }
I20251215 21:02:01.455154 31504 leader_election.cc:290] T ddfd6a00699547f3829226d6b549cb02 P 5755a662553c4a84b76c8b2c48880567 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6e18dca41b204cf0a3ebedd00ec07906 (127.28.113.1:33435), 9899e8262dee40858d8880500e8054b1 (127.28.113.2:35261)
I20251215 21:02:01.455547 31045 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ddfd6a00699547f3829226d6b549cb02" candidate_uuid: "5755a662553c4a84b76c8b2c48880567" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e18dca41b204cf0a3ebedd00ec07906" is_pre_election: true
I20251215 21:02:01.455619 31170 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ddfd6a00699547f3829226d6b549cb02" candidate_uuid: "5755a662553c4a84b76c8b2c48880567" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9899e8262dee40858d8880500e8054b1" is_pre_election: true
I20251215 21:02:01.455686 31045 raft_consensus.cc:2468] T ddfd6a00699547f3829226d6b549cb02 P 6e18dca41b204cf0a3ebedd00ec07906 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5755a662553c4a84b76c8b2c48880567 in term 0.
I20251215 21:02:01.455775 31170 raft_consensus.cc:2468] T ddfd6a00699547f3829226d6b549cb02 P 9899e8262dee40858d8880500e8054b1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5755a662553c4a84b76c8b2c48880567 in term 0.
I20251215 21:02:01.456030 31262 leader_election.cc:304] T ddfd6a00699547f3829226d6b549cb02 P 5755a662553c4a84b76c8b2c48880567 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5755a662553c4a84b76c8b2c48880567, 6e18dca41b204cf0a3ebedd00ec07906; no voters:
I20251215 21:02:01.456218 31504 raft_consensus.cc:2804] T ddfd6a00699547f3829226d6b549cb02 P 5755a662553c4a84b76c8b2c48880567 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251215 21:02:01.456272 31504 raft_consensus.cc:493] T ddfd6a00699547f3829226d6b549cb02 P 5755a662553c4a84b76c8b2c48880567 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 21:02:01.456300 31504 raft_consensus.cc:3060] T ddfd6a00699547f3829226d6b549cb02 P 5755a662553c4a84b76c8b2c48880567 [term 0 FOLLOWER]: Advancing to term 1
I20251215 21:02:01.457180 31504 raft_consensus.cc:515] T ddfd6a00699547f3829226d6b549cb02 P 5755a662553c4a84b76c8b2c48880567 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e18dca41b204cf0a3ebedd00ec07906" member_type: VOTER last_known_addr { host: "127.28.113.1" port: 33435 } } peers { permanent_uuid: "9899e8262dee40858d8880500e8054b1" member_type: VOTER last_known_addr { host: "127.28.113.2" port: 35261 } } peers { permanent_uuid: "5755a662553c4a84b76c8b2c48880567" member_type: VOTER last_known_addr { host: "127.28.113.3" port: 42057 } }
I20251215 21:02:01.457473 31504 leader_election.cc:290] T ddfd6a00699547f3829226d6b549cb02 P 5755a662553c4a84b76c8b2c48880567 [CANDIDATE]: Term 1 election: Requested vote from peers 6e18dca41b204cf0a3ebedd00ec07906 (127.28.113.1:33435), 9899e8262dee40858d8880500e8054b1 (127.28.113.2:35261)
I20251215 21:02:01.457787 31045 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ddfd6a00699547f3829226d6b549cb02" candidate_uuid: "5755a662553c4a84b76c8b2c48880567" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e18dca41b204cf0a3ebedd00ec07906"
I20251215 21:02:01.457828 31170 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ddfd6a00699547f3829226d6b549cb02" candidate_uuid: "5755a662553c4a84b76c8b2c48880567" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9899e8262dee40858d8880500e8054b1"
I20251215 21:02:01.457893 31045 raft_consensus.cc:3060] T ddfd6a00699547f3829226d6b549cb02 P 6e18dca41b204cf0a3ebedd00ec07906 [term 0 FOLLOWER]: Advancing to term 1
I20251215 21:02:01.457930 31170 raft_consensus.cc:3060] T ddfd6a00699547f3829226d6b549cb02 P 9899e8262dee40858d8880500e8054b1 [term 0 FOLLOWER]: Advancing to term 1
I20251215 21:02:01.458693 31045 raft_consensus.cc:2468] T ddfd6a00699547f3829226d6b549cb02 P 6e18dca41b204cf0a3ebedd00ec07906 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5755a662553c4a84b76c8b2c48880567 in term 1.
I20251215 21:02:01.458709 31170 raft_consensus.cc:2468] T ddfd6a00699547f3829226d6b549cb02 P 9899e8262dee40858d8880500e8054b1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5755a662553c4a84b76c8b2c48880567 in term 1.
I20251215 21:02:01.458949 31262 leader_election.cc:304] T ddfd6a00699547f3829226d6b549cb02 P 5755a662553c4a84b76c8b2c48880567 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5755a662553c4a84b76c8b2c48880567, 9899e8262dee40858d8880500e8054b1; no voters:
I20251215 21:02:01.459105 31504 raft_consensus.cc:2804] T ddfd6a00699547f3829226d6b549cb02 P 5755a662553c4a84b76c8b2c48880567 [term 1 FOLLOWER]: Leader election won for term 1
I20251215 21:02:01.459300 31504 raft_consensus.cc:697] T ddfd6a00699547f3829226d6b549cb02 P 5755a662553c4a84b76c8b2c48880567 [term 1 LEADER]: Becoming Leader. State: Replica: 5755a662553c4a84b76c8b2c48880567, State: Running, Role: LEADER
I20251215 21:02:01.459432 31504 consensus_queue.cc:237] T ddfd6a00699547f3829226d6b549cb02 P 5755a662553c4a84b76c8b2c48880567 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e18dca41b204cf0a3ebedd00ec07906" member_type: VOTER last_known_addr { host: "127.28.113.1" port: 33435 } } peers { permanent_uuid: "9899e8262dee40858d8880500e8054b1" member_type: VOTER last_known_addr { host: "127.28.113.2" port: 35261 } } peers { permanent_uuid: "5755a662553c4a84b76c8b2c48880567" member_type: VOTER last_known_addr { host: "127.28.113.3" port: 42057 } }
I20251215 21:02:01.460839 30886 catalog_manager.cc:5654] T ddfd6a00699547f3829226d6b549cb02 P 5755a662553c4a84b76c8b2c48880567 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5755a662553c4a84b76c8b2c48880567 (127.28.113.3). New cstate: current_term: 1 leader_uuid: "5755a662553c4a84b76c8b2c48880567" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e18dca41b204cf0a3ebedd00ec07906" member_type: VOTER last_known_addr { host: "127.28.113.1" port: 33435 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9899e8262dee40858d8880500e8054b1" member_type: VOTER last_known_addr { host: "127.28.113.2" port: 35261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5755a662553c4a84b76c8b2c48880567" member_type: VOTER last_known_addr { host: "127.28.113.3" port: 42057 } health_report { overall_health: HEALTHY } } }
I20251215 21:02:01.480182 31508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:02:01.478052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40392 (local address 127.28.113.3:42057)
1215 21:02:01.478275 (+ 223us) server_negotiation.cc:207] Beginning negotiation
1215 21:02:01.478279 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1215 21:02:01.478290 (+ 11us) server_negotiation.cc:408] Connection header received
1215 21:02:01.478324 (+ 34us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1215 21:02:01.478326 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1215 21:02:01.478365 (+ 39us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1215 21:02:01.478435 (+ 70us) server_negotiation.cc:227] Negotiated authn=TOKEN
1215 21:02:01.478695 (+ 260us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:02:01.479170 (+ 475us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1215 21:02:01.479700 (+ 530us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:02:01.479820 (+ 120us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:02:01.479853 (+ 33us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1215 21:02:01.479913 (+ 60us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1215 21:02:01.480001 (+ 88us) server_negotiation.cc:1036] Waiting for connection context
1215 21:02:01.480036 (+ 35us) server_negotiation.cc:300] Negotiation successful
1215 21:02:01.480075 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":157,"thread_start_us":92,"threads_started":1}
I20251215 21:02:01.485606 31170 raft_consensus.cc:1275] T ddfd6a00699547f3829226d6b549cb02 P 9899e8262dee40858d8880500e8054b1 [term 1 FOLLOWER]: Refusing update from remote peer 5755a662553c4a84b76c8b2c48880567: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251215 21:02:01.485652 31045 raft_consensus.cc:1275] T ddfd6a00699547f3829226d6b549cb02 P 6e18dca41b204cf0a3ebedd00ec07906 [term 1 FOLLOWER]: Refusing update from remote peer 5755a662553c4a84b76c8b2c48880567: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251215 21:02:01.485954 31505 consensus_queue.cc:1048] T ddfd6a00699547f3829226d6b549cb02 P 5755a662553c4a84b76c8b2c48880567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9899e8262dee40858d8880500e8054b1" member_type: VOTER last_known_addr { host: "127.28.113.2" port: 35261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251215 21:02:01.486115 31504 consensus_queue.cc:1048] T ddfd6a00699547f3829226d6b549cb02 P 5755a662553c4a84b76c8b2c48880567 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e18dca41b204cf0a3ebedd00ec07906" member_type: VOTER last_known_addr { host: "127.28.113.1" port: 33435 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251215 21:02:01.514847 30882 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56882:
table { table_name: "test-table" } modify_external_catalogs: true
I20251215 21:02:01.514997 30882 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56882:
table { table_name: "test-table" } modify_external_catalogs: true
I20251215 21:02:01.516747 30882 catalog_manager.cc:5941] T 00000000000000000000000000000000 P 8d6aede0d59c4dbd8c8bca7ade69344b: Sending DeleteTablet for 3 replicas of tablet ddfd6a00699547f3829226d6b549cb02
I20251215 21:02:01.517225 31024 tablet_service.cc:1552] Processing DeleteTablet for tablet ddfd6a00699547f3829226d6b549cb02 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-15 21:02:01 UTC) from {username='kudu', principal='kudu/127.28.113.62@KRBTEST.COM'} at 127.0.0.1:45748
I20251215 21:02:01.517289 31168 tablet_service.cc:1552] Processing DeleteTablet for tablet ddfd6a00699547f3829226d6b549cb02 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-15 21:02:01 UTC) from {username='kudu', principal='kudu/127.28.113.62@KRBTEST.COM'} at 127.0.0.1:34108
I20251215 21:02:01.517318 31310 tablet_service.cc:1552] Processing DeleteTablet for tablet ddfd6a00699547f3829226d6b549cb02 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-15 21:02:01 UTC) from {username='kudu', principal='kudu/127.28.113.62@KRBTEST.COM'} at 127.0.0.1:40368
I20251215 21:02:01.517408 31488 tablet_replica.cc:333] T ddfd6a00699547f3829226d6b549cb02 P 6e18dca41b204cf0a3ebedd00ec07906: stopping tablet replica
I20251215 21:02:01.517428 31487 tablet_replica.cc:333] T ddfd6a00699547f3829226d6b549cb02 P 9899e8262dee40858d8880500e8054b1: stopping tablet replica
I20251215 21:02:01.517496 31490 tablet_replica.cc:333] T ddfd6a00699547f3829226d6b549cb02 P 5755a662553c4a84b76c8b2c48880567: stopping tablet replica
I20251215 21:02:01.517508 31488 raft_consensus.cc:2243] T ddfd6a00699547f3829226d6b549cb02 P 6e18dca41b204cf0a3ebedd00ec07906 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251215 21:02:01.517508 31487 raft_consensus.cc:2243] T ddfd6a00699547f3829226d6b549cb02 P 9899e8262dee40858d8880500e8054b1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251215 21:02:01.517606 31487 raft_consensus.cc:2272] T ddfd6a00699547f3829226d6b549cb02 P 9899e8262dee40858d8880500e8054b1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251215 21:02:01.517606 31488 raft_consensus.cc:2272] T ddfd6a00699547f3829226d6b549cb02 P 6e18dca41b204cf0a3ebedd00ec07906 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251215 21:02:01.517617 31490 raft_consensus.cc:2243] T ddfd6a00699547f3829226d6b549cb02 P 5755a662553c4a84b76c8b2c48880567 [term 1 LEADER]: Raft consensus shutting down.
I20251215 21:02:01.517788 31490 raft_consensus.cc:2272] T ddfd6a00699547f3829226d6b549cb02 P 5755a662553c4a84b76c8b2c48880567 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251215 21:02:01.518193 31487 ts_tablet_manager.cc:1916] T ddfd6a00699547f3829226d6b549cb02 P 9899e8262dee40858d8880500e8054b1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251215 21:02:01.518492 31488 ts_tablet_manager.cc:1916] T ddfd6a00699547f3829226d6b549cb02 P 6e18dca41b204cf0a3ebedd00ec07906: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251215 21:02:01.518492 31490 ts_tablet_manager.cc:1916] T ddfd6a00699547f3829226d6b549cb02 P 5755a662553c4a84b76c8b2c48880567: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251215 21:02:01.520076 31487 ts_tablet_manager.cc:1929] T ddfd6a00699547f3829226d6b549cb02 P 9899e8262dee40858d8880500e8054b1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20251215 21:02:01.520141 31487 log.cc:1199] T ddfd6a00699547f3829226d6b549cb02 P 9899e8262dee40858d8880500e8054b1: Deleting WAL directory at /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765832506145157-29124-0/minicluster-data/ts-1/wal/wals/ddfd6a00699547f3829226d6b549cb02
I20251215 21:02:01.520331 31487 ts_tablet_manager.cc:1950] T ddfd6a00699547f3829226d6b549cb02 P 9899e8262dee40858d8880500e8054b1: Deleting consensus metadata
I20251215 21:02:01.520627 31490 ts_tablet_manager.cc:1929] T ddfd6a00699547f3829226d6b549cb02 P 5755a662553c4a84b76c8b2c48880567: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20251215 21:02:01.520666 31488 ts_tablet_manager.cc:1929] T ddfd6a00699547f3829226d6b549cb02 P 6e18dca41b204cf0a3ebedd00ec07906: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20251215 21:02:01.520702 31490 log.cc:1199] T ddfd6a00699547f3829226d6b549cb02 P 5755a662553c4a84b76c8b2c48880567: Deleting WAL directory at /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765832506145157-29124-0/minicluster-data/ts-2/wal/wals/ddfd6a00699547f3829226d6b549cb02
I20251215 21:02:01.520720 31488 log.cc:1199] T ddfd6a00699547f3829226d6b549cb02 P 6e18dca41b204cf0a3ebedd00ec07906: Deleting WAL directory at /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765832506145157-29124-0/minicluster-data/ts-0/wal/wals/ddfd6a00699547f3829226d6b549cb02
I20251215 21:02:01.520879 30872 catalog_manager.cc:4985] TS 9899e8262dee40858d8880500e8054b1 (127.28.113.2:35261): tablet ddfd6a00699547f3829226d6b549cb02 (table test-table [id=5344a7c089634484bbe85ad4a56e3356]) successfully deleted
I20251215 21:02:01.520939 31488 ts_tablet_manager.cc:1950] T ddfd6a00699547f3829226d6b549cb02 P 6e18dca41b204cf0a3ebedd00ec07906: Deleting consensus metadata
I20251215 21:02:01.520939 31490 ts_tablet_manager.cc:1950] T ddfd6a00699547f3829226d6b549cb02 P 5755a662553c4a84b76c8b2c48880567: Deleting consensus metadata
I20251215 21:02:01.520980 30882 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56882:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251215 21:02:01.521364 30882 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251215 21:02:01.521510 30872 catalog_manager.cc:4985] TS 6e18dca41b204cf0a3ebedd00ec07906 (127.28.113.1:33435): tablet ddfd6a00699547f3829226d6b549cb02 (table test-table [id=5344a7c089634484bbe85ad4a56e3356]) successfully deleted
I20251215 21:02:01.521515 30873 catalog_manager.cc:4985] TS 5755a662553c4a84b76c8b2c48880567 (127.28.113.3:42057): tablet ddfd6a00699547f3829226d6b549cb02 (table test-table [id=5344a7c089634484bbe85ad4a56e3356]) successfully deleted
I20251215 21:02:01.525230 31024 tablet_service.cc:1505] Processing CreateTablet for tablet c7d65525d480409693e3975d5fec9976 (DEFAULT_TABLE table=test-table [id=b1d179e616a141c88a3c1b3b3703c9e5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251215 21:02:01.525472 31024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c7d65525d480409693e3975d5fec9976. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 21:02:01.525614 31310 tablet_service.cc:1505] Processing CreateTablet for tablet c7d65525d480409693e3975d5fec9976 (DEFAULT_TABLE table=test-table [id=b1d179e616a141c88a3c1b3b3703c9e5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251215 21:02:01.525847 31310 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c7d65525d480409693e3975d5fec9976. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 21:02:01.527117 31500 tablet_bootstrap.cc:492] T c7d65525d480409693e3975d5fec9976 P 6e18dca41b204cf0a3ebedd00ec07906: Bootstrap starting.
I20251215 21:02:01.527262 31168 tablet_service.cc:1505] Processing CreateTablet for tablet c7d65525d480409693e3975d5fec9976 (DEFAULT_TABLE table=test-table [id=b1d179e616a141c88a3c1b3b3703c9e5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251215 21:02:01.527408 31498 tablet_bootstrap.cc:492] T c7d65525d480409693e3975d5fec9976 P 5755a662553c4a84b76c8b2c48880567: Bootstrap starting.
I20251215 21:02:01.527475 31168 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c7d65525d480409693e3975d5fec9976. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 21:02:01.527932 31500 tablet_bootstrap.cc:654] T c7d65525d480409693e3975d5fec9976 P 6e18dca41b204cf0a3ebedd00ec07906: Neither blocks nor log segments found. Creating new log.
I20251215 21:02:01.528234 31498 tablet_bootstrap.cc:654] T c7d65525d480409693e3975d5fec9976 P 5755a662553c4a84b76c8b2c48880567: Neither blocks nor log segments found. Creating new log.
I20251215 21:02:01.528846 31500 tablet_bootstrap.cc:492] T c7d65525d480409693e3975d5fec9976 P 6e18dca41b204cf0a3ebedd00ec07906: No bootstrap required, opened a new log
I20251215 21:02:01.528930 31500 ts_tablet_manager.cc:1403] T c7d65525d480409693e3975d5fec9976 P 6e18dca41b204cf0a3ebedd00ec07906: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20251215 21:02:01.529261 31498 tablet_bootstrap.cc:492] T c7d65525d480409693e3975d5fec9976 P 5755a662553c4a84b76c8b2c48880567: No bootstrap required, opened a new log
I20251215 21:02:01.529335 31498 ts_tablet_manager.cc:1403] T c7d65525d480409693e3975d5fec9976 P 5755a662553c4a84b76c8b2c48880567: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20251215 21:02:01.529284 31500 raft_consensus.cc:359] T c7d65525d480409693e3975d5fec9976 P 6e18dca41b204cf0a3ebedd00ec07906 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e18dca41b204cf0a3ebedd00ec07906" member_type: VOTER last_known_addr { host: "127.28.113.1" port: 33435 } } peers { permanent_uuid: "9899e8262dee40858d8880500e8054b1" member_type: VOTER last_known_addr { host: "127.28.113.2" port: 35261 } } peers { permanent_uuid: "5755a662553c4a84b76c8b2c48880567" member_type: VOTER last_known_addr { host: "127.28.113.3" port: 42057 } }
I20251215 21:02:01.529382 31500 raft_consensus.cc:385] T c7d65525d480409693e3975d5fec9976 P 6e18dca41b204cf0a3ebedd00ec07906 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251215 21:02:01.529386 31499 tablet_bootstrap.cc:492] T c7d65525d480409693e3975d5fec9976 P 9899e8262dee40858d8880500e8054b1: Bootstrap starting.
I20251215 21:02:01.529415 31500 raft_consensus.cc:740] T c7d65525d480409693e3975d5fec9976 P 6e18dca41b204cf0a3ebedd00ec07906 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e18dca41b204cf0a3ebedd00ec07906, State: Initialized, Role: FOLLOWER
I20251215 21:02:01.529522 31500 consensus_queue.cc:260] T c7d65525d480409693e3975d5fec9976 P 6e18dca41b204cf0a3ebedd00ec07906 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e18dca41b204cf0a3ebedd00ec07906" member_type: VOTER last_known_addr { host: "127.28.113.1" port: 33435 } } peers { permanent_uuid: "9899e8262dee40858d8880500e8054b1" member_type: VOTER last_known_addr { host: "127.28.113.2" port: 35261 } } peers { permanent_uuid: "5755a662553c4a84b76c8b2c48880567" member_type: VOTER last_known_addr { host: "127.28.113.3" port: 42057 } }
I20251215 21:02:01.529702 31498 raft_consensus.cc:359] T c7d65525d480409693e3975d5fec9976 P 5755a662553c4a84b76c8b2c48880567 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e18dca41b204cf0a3ebedd00ec07906" member_type: VOTER last_known_addr { host: "127.28.113.1" port: 33435 } } peers { permanent_uuid: "9899e8262dee40858d8880500e8054b1" member_type: VOTER last_known_addr { host: "127.28.113.2" port: 35261 } } peers { permanent_uuid: "5755a662553c4a84b76c8b2c48880567" member_type: VOTER last_known_addr { host: "127.28.113.3" port: 42057 } }
I20251215 21:02:01.529778 31500 ts_tablet_manager.cc:1434] T c7d65525d480409693e3975d5fec9976 P 6e18dca41b204cf0a3ebedd00ec07906: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20251215 21:02:01.529791 31498 raft_consensus.cc:385] T c7d65525d480409693e3975d5fec9976 P 5755a662553c4a84b76c8b2c48880567 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251215 21:02:01.529821 31498 raft_consensus.cc:740] T c7d65525d480409693e3975d5fec9976 P 5755a662553c4a84b76c8b2c48880567 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5755a662553c4a84b76c8b2c48880567, State: Initialized, Role: FOLLOWER
I20251215 21:02:01.529927 31498 consensus_queue.cc:260] T c7d65525d480409693e3975d5fec9976 P 5755a662553c4a84b76c8b2c48880567 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e18dca41b204cf0a3ebedd00ec07906" member_type: VOTER last_known_addr { host: "127.28.113.1" port: 33435 } } peers { permanent_uuid: "9899e8262dee40858d8880500e8054b1" member_type: VOTER last_known_addr { host: "127.28.113.2" port: 35261 } } peers { permanent_uuid: "5755a662553c4a84b76c8b2c48880567" member_type: VOTER last_known_addr { host: "127.28.113.3" port: 42057 } }
I20251215 21:02:01.530157 31499 tablet_bootstrap.cc:654] T c7d65525d480409693e3975d5fec9976 P 9899e8262dee40858d8880500e8054b1: Neither blocks nor log segments found. Creating new log.
I20251215 21:02:01.530157 31498 ts_tablet_manager.cc:1434] T c7d65525d480409693e3975d5fec9976 P 5755a662553c4a84b76c8b2c48880567: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20251215 21:02:01.531251 31499 tablet_bootstrap.cc:492] T c7d65525d480409693e3975d5fec9976 P 9899e8262dee40858d8880500e8054b1: No bootstrap required, opened a new log
I20251215 21:02:01.531329 31499 ts_tablet_manager.cc:1403] T c7d65525d480409693e3975d5fec9976 P 9899e8262dee40858d8880500e8054b1: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20251215 21:02:01.531700 31499 raft_consensus.cc:359] T c7d65525d480409693e3975d5fec9976 P 9899e8262dee40858d8880500e8054b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e18dca41b204cf0a3ebedd00ec07906" member_type: VOTER last_known_addr { host: "127.28.113.1" port: 33435 } } peers { permanent_uuid: "9899e8262dee40858d8880500e8054b1" member_type: VOTER last_known_addr { host: "127.28.113.2" port: 35261 } } peers { permanent_uuid: "5755a662553c4a84b76c8b2c48880567" member_type: VOTER last_known_addr { host: "127.28.113.3" port: 42057 } }
I20251215 21:02:01.531791 31499 raft_consensus.cc:385] T c7d65525d480409693e3975d5fec9976 P 9899e8262dee40858d8880500e8054b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251215 21:02:01.531823 31499 raft_consensus.cc:740] T c7d65525d480409693e3975d5fec9976 P 9899e8262dee40858d8880500e8054b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9899e8262dee40858d8880500e8054b1, State: Initialized, Role: FOLLOWER
I20251215 21:02:01.531945 31499 consensus_queue.cc:260] T c7d65525d480409693e3975d5fec9976 P 9899e8262dee40858d8880500e8054b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e18dca41b204cf0a3ebedd00ec07906" member_type: VOTER last_known_addr { host: "127.28.113.1" port: 33435 } } peers { permanent_uuid: "9899e8262dee40858d8880500e8054b1" member_type: VOTER last_known_addr { host: "127.28.113.2" port: 35261 } } peers { permanent_uuid: "5755a662553c4a84b76c8b2c48880567" member_type: VOTER last_known_addr { host: "127.28.113.3" port: 42057 } }
I20251215 21:02:01.532204 31499 ts_tablet_manager.cc:1434] T c7d65525d480409693e3975d5fec9976 P 9899e8262dee40858d8880500e8054b1: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20251215 21:02:01.690958 31474 raft_consensus.cc:493] T c7d65525d480409693e3975d5fec9976 P 6e18dca41b204cf0a3ebedd00ec07906 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 21:02:01.691094 31474 raft_consensus.cc:515] T c7d65525d480409693e3975d5fec9976 P 6e18dca41b204cf0a3ebedd00ec07906 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e18dca41b204cf0a3ebedd00ec07906" member_type: VOTER last_known_addr { host: "127.28.113.1" port: 33435 } } peers { permanent_uuid: "9899e8262dee40858d8880500e8054b1" member_type: VOTER last_known_addr { host: "127.28.113.2" port: 35261 } } peers { permanent_uuid: "5755a662553c4a84b76c8b2c48880567" member_type: VOTER last_known_addr { host: "127.28.113.3" port: 42057 } }
I20251215 21:02:01.691499 31474 leader_election.cc:290] T c7d65525d480409693e3975d5fec9976 P 6e18dca41b204cf0a3ebedd00ec07906 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9899e8262dee40858d8880500e8054b1 (127.28.113.2:35261), 5755a662553c4a84b76c8b2c48880567 (127.28.113.3:42057)
I20251215 21:02:01.691936 31170 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c7d65525d480409693e3975d5fec9976" candidate_uuid: "6e18dca41b204cf0a3ebedd00ec07906" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9899e8262dee40858d8880500e8054b1" is_pre_election: true
I20251215 21:02:01.691939 31330 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c7d65525d480409693e3975d5fec9976" candidate_uuid: "6e18dca41b204cf0a3ebedd00ec07906" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5755a662553c4a84b76c8b2c48880567" is_pre_election: true
I20251215 21:02:01.692097 31170 raft_consensus.cc:2468] T c7d65525d480409693e3975d5fec9976 P 9899e8262dee40858d8880500e8054b1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e18dca41b204cf0a3ebedd00ec07906 in term 0.
I20251215 21:02:01.692116 31330 raft_consensus.cc:2468] T c7d65525d480409693e3975d5fec9976 P 5755a662553c4a84b76c8b2c48880567 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e18dca41b204cf0a3ebedd00ec07906 in term 0.
I20251215 21:02:01.692425 30978 leader_election.cc:304] T c7d65525d480409693e3975d5fec9976 P 6e18dca41b204cf0a3ebedd00ec07906 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e18dca41b204cf0a3ebedd00ec07906, 9899e8262dee40858d8880500e8054b1; no voters:
I20251215 21:02:01.692580 31474 raft_consensus.cc:2804] T c7d65525d480409693e3975d5fec9976 P 6e18dca41b204cf0a3ebedd00ec07906 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251215 21:02:01.692651 31474 raft_consensus.cc:493] T c7d65525d480409693e3975d5fec9976 P 6e18dca41b204cf0a3ebedd00ec07906 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 21:02:01.692679 31474 raft_consensus.cc:3060] T c7d65525d480409693e3975d5fec9976 P 6e18dca41b204cf0a3ebedd00ec07906 [term 0 FOLLOWER]: Advancing to term 1
I20251215 21:02:01.693550 31474 raft_consensus.cc:515] T c7d65525d480409693e3975d5fec9976 P 6e18dca41b204cf0a3ebedd00ec07906 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e18dca41b204cf0a3ebedd00ec07906" member_type: VOTER last_known_addr { host: "127.28.113.1" port: 33435 } } peers { permanent_uuid: "9899e8262dee40858d8880500e8054b1" member_type: VOTER last_known_addr { host: "127.28.113.2" port: 35261 } } peers { permanent_uuid: "5755a662553c4a84b76c8b2c48880567" member_type: VOTER last_known_addr { host: "127.28.113.3" port: 42057 } }
I20251215 21:02:01.693852 31474 leader_election.cc:290] T c7d65525d480409693e3975d5fec9976 P 6e18dca41b204cf0a3ebedd00ec07906 [CANDIDATE]: Term 1 election: Requested vote from peers 9899e8262dee40858d8880500e8054b1 (127.28.113.2:35261), 5755a662553c4a84b76c8b2c48880567 (127.28.113.3:42057)
I20251215 21:02:01.694142 31330 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c7d65525d480409693e3975d5fec9976" candidate_uuid: "6e18dca41b204cf0a3ebedd00ec07906" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5755a662553c4a84b76c8b2c48880567"
I20251215 21:02:01.694147 31170 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c7d65525d480409693e3975d5fec9976" candidate_uuid: "6e18dca41b204cf0a3ebedd00ec07906" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9899e8262dee40858d8880500e8054b1"
I20251215 21:02:01.694228 31330 raft_consensus.cc:3060] T c7d65525d480409693e3975d5fec9976 P 5755a662553c4a84b76c8b2c48880567 [term 0 FOLLOWER]: Advancing to term 1
I20251215 21:02:01.694236 31170 raft_consensus.cc:3060] T c7d65525d480409693e3975d5fec9976 P 9899e8262dee40858d8880500e8054b1 [term 0 FOLLOWER]: Advancing to term 1
I20251215 21:02:01.694994 31330 raft_consensus.cc:2468] T c7d65525d480409693e3975d5fec9976 P 5755a662553c4a84b76c8b2c48880567 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e18dca41b204cf0a3ebedd00ec07906 in term 1.
I20251215 21:02:01.695034 31170 raft_consensus.cc:2468] T c7d65525d480409693e3975d5fec9976 P 9899e8262dee40858d8880500e8054b1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e18dca41b204cf0a3ebedd00ec07906 in term 1.
I20251215 21:02:01.695264 30979 leader_election.cc:304] T c7d65525d480409693e3975d5fec9976 P 6e18dca41b204cf0a3ebedd00ec07906 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5755a662553c4a84b76c8b2c48880567, 6e18dca41b204cf0a3ebedd00ec07906; no voters:
I20251215 21:02:01.695469 31474 raft_consensus.cc:2804] T c7d65525d480409693e3975d5fec9976 P 6e18dca41b204cf0a3ebedd00ec07906 [term 1 FOLLOWER]: Leader election won for term 1
I20251215 21:02:01.695539 31474 raft_consensus.cc:697] T c7d65525d480409693e3975d5fec9976 P 6e18dca41b204cf0a3ebedd00ec07906 [term 1 LEADER]: Becoming Leader. State: Replica: 6e18dca41b204cf0a3ebedd00ec07906, State: Running, Role: LEADER
I20251215 21:02:01.695676 31474 consensus_queue.cc:237] T c7d65525d480409693e3975d5fec9976 P 6e18dca41b204cf0a3ebedd00ec07906 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e18dca41b204cf0a3ebedd00ec07906" member_type: VOTER last_known_addr { host: "127.28.113.1" port: 33435 } } peers { permanent_uuid: "9899e8262dee40858d8880500e8054b1" member_type: VOTER last_known_addr { host: "127.28.113.2" port: 35261 } } peers { permanent_uuid: "5755a662553c4a84b76c8b2c48880567" member_type: VOTER last_known_addr { host: "127.28.113.3" port: 42057 } }
I20251215 21:02:01.697168 30882 catalog_manager.cc:5654] T c7d65525d480409693e3975d5fec9976 P 6e18dca41b204cf0a3ebedd00ec07906 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6e18dca41b204cf0a3ebedd00ec07906 (127.28.113.1). New cstate: current_term: 1 leader_uuid: "6e18dca41b204cf0a3ebedd00ec07906" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e18dca41b204cf0a3ebedd00ec07906" member_type: VOTER last_known_addr { host: "127.28.113.1" port: 33435 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9899e8262dee40858d8880500e8054b1" member_type: VOTER last_known_addr { host: "127.28.113.2" port: 35261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5755a662553c4a84b76c8b2c48880567" member_type: VOTER last_known_addr { host: "127.28.113.3" port: 42057 } health_report { overall_health: UNKNOWN } } }
I20251215 21:02:01.723469 31517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:02:01.721100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45764 (local address 127.28.113.1:33435)
1215 21:02:01.721359 (+ 259us) server_negotiation.cc:207] Beginning negotiation
1215 21:02:01.721362 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1215 21:02:01.721371 (+ 9us) server_negotiation.cc:408] Connection header received
1215 21:02:01.721404 (+ 33us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1215 21:02:01.721406 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1215 21:02:01.721446 (+ 40us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1215 21:02:01.721509 (+ 63us) server_negotiation.cc:227] Negotiated authn=TOKEN
1215 21:02:01.721835 (+ 326us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:02:01.722367 (+ 532us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1215 21:02:01.722948 (+ 581us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:02:01.723071 (+ 123us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:02:01.723106 (+ 35us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1215 21:02:01.723166 (+ 60us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1215 21:02:01.723217 (+ 51us) server_negotiation.cc:1036] Waiting for connection context
1215 21:02:01.723288 (+ 71us) server_negotiation.cc:300] Negotiation successful
1215 21:02:01.723325 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":184,"thread_start_us":105,"threads_started":1}
I20251215 21:02:01.730787 31517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:02:01.728616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.113.1:50517 (local address 127.28.113.1:33435)
1215 21:02:01.728701 (+ 85us) server_negotiation.cc:207] Beginning negotiation
1215 21:02:01.728706 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1215 21:02:01.728760 (+ 54us) server_negotiation.cc:408] Connection header received
1215 21:02:01.728880 (+ 120us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1215 21:02:01.728883 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1215 21:02:01.728918 (+ 35us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1215 21:02:01.728969 (+ 51us) server_negotiation.cc:227] Negotiated authn=TOKEN
1215 21:02:01.729261 (+ 292us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:02:01.729686 (+ 425us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1215 21:02:01.730235 (+ 549us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:02:01.730382 (+ 147us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:02:01.730414 (+ 32us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1215 21:02:01.730471 (+ 57us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1215 21:02:01.730528 (+ 57us) server_negotiation.cc:1036] Waiting for connection context
1215 21:02:01.730615 (+ 87us) server_negotiation.cc:300] Negotiation successful
1215 21:02:01.730652 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20251215 21:02:01.730792 31520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:02:01.728449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.113.1:33435 (local address 127.28.113.1:50517)
1215 21:02:01.728682 (+ 233us) negotiation.cc:107] Waiting for socket to connect
1215 21:02:01.728696 (+ 14us) client_negotiation.cc:174] Beginning negotiation
1215 21:02:01.728789 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1215 21:02:01.728983 (+ 194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1215 21:02:01.728986 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1215 21:02:01.728992 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
1215 21:02:01.729174 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:02:01.729179 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:02:01.729772 (+ 593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1215 21:02:01.729775 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1215 21:02:01.730145 (+ 370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:02:01.730151 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:02:01.730244 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:02:01.730268 (+ 24us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
1215 21:02:01.730543 (+ 275us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
1215 21:02:01.730548 (+ 5us) client_negotiation.cc:769] Sending connection context
1215 21:02:01.730602 (+ 54us) client_negotiation.cc:240] Negotiation successful
1215 21:02:01.730692 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":73,"threads_started":1}
I20251215 21:02:01.735718 31170 raft_consensus.cc:1275] T c7d65525d480409693e3975d5fec9976 P 9899e8262dee40858d8880500e8054b1 [term 1 FOLLOWER]: Refusing update from remote peer 6e18dca41b204cf0a3ebedd00ec07906: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251215 21:02:01.736047 31330 raft_consensus.cc:1275] T c7d65525d480409693e3975d5fec9976 P 5755a662553c4a84b76c8b2c48880567 [term 1 FOLLOWER]: Refusing update from remote peer 6e18dca41b204cf0a3ebedd00ec07906: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251215 21:02:01.736079 31474 consensus_queue.cc:1048] T c7d65525d480409693e3975d5fec9976 P 6e18dca41b204cf0a3ebedd00ec07906 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9899e8262dee40858d8880500e8054b1" member_type: VOTER last_known_addr { host: "127.28.113.2" port: 35261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251215 21:02:01.736389 31474 consensus_queue.cc:1048] T c7d65525d480409693e3975d5fec9976 P 6e18dca41b204cf0a3ebedd00ec07906 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5755a662553c4a84b76c8b2c48880567" member_type: VOTER last_known_addr { host: "127.28.113.3" port: 42057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20251215 21:02:01.748306 31466 tablet_replica.cc:1306] Aborted: operation has been aborted: cancelling pending write operations
I20251215 21:02:03.297392 30887 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56882:
table { table_name: "test-table" } modify_external_catalogs: true
I20251215 21:02:03.297559 30887 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56882:
table { table_name: "test-table" } modify_external_catalogs: true
I20251215 21:02:03.299846 30887 catalog_manager.cc:5941] T 00000000000000000000000000000000 P 8d6aede0d59c4dbd8c8bca7ade69344b: Sending DeleteTablet for 3 replicas of tablet c7d65525d480409693e3975d5fec9976
I20251215 21:02:03.300379 31024 tablet_service.cc:1552] Processing DeleteTablet for tablet c7d65525d480409693e3975d5fec9976 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-15 21:02:03 UTC) from {username='kudu', principal='kudu/127.28.113.62@KRBTEST.COM'} at 127.0.0.1:45748
I20251215 21:02:03.300418 31310 tablet_service.cc:1552] Processing DeleteTablet for tablet c7d65525d480409693e3975d5fec9976 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-15 21:02:03 UTC) from {username='kudu', principal='kudu/127.28.113.62@KRBTEST.COM'} at 127.0.0.1:40368
I20251215 21:02:03.300719 31543 tablet_replica.cc:333] T c7d65525d480409693e3975d5fec9976 P 6e18dca41b204cf0a3ebedd00ec07906: stopping tablet replica
I20251215 21:02:03.300760 29124 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu with pid 30961
I20251215 21:02:03.300846 31168 tablet_service.cc:1552] Processing DeleteTablet for tablet c7d65525d480409693e3975d5fec9976 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-15 21:02:03 UTC) from {username='kudu', principal='kudu/127.28.113.62@KRBTEST.COM'} at 127.0.0.1:34108
I20251215 21:02:03.302028 31545 tablet_replica.cc:333] T c7d65525d480409693e3975d5fec9976 P 9899e8262dee40858d8880500e8054b1: stopping tablet replica
I20251215 21:02:03.302146 31545 raft_consensus.cc:2243] T c7d65525d480409693e3975d5fec9976 P 9899e8262dee40858d8880500e8054b1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251215 21:02:03.302256 31545 raft_consensus.cc:2272] T c7d65525d480409693e3975d5fec9976 P 9899e8262dee40858d8880500e8054b1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251215 21:02:03.304383 31545 ts_tablet_manager.cc:1916] T c7d65525d480409693e3975d5fec9976 P 9899e8262dee40858d8880500e8054b1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251215 21:02:03.305608 31544 tablet_replica.cc:333] T c7d65525d480409693e3975d5fec9976 P 5755a662553c4a84b76c8b2c48880567: stopping tablet replica
I20251215 21:02:03.305729 31544 raft_consensus.cc:2243] T c7d65525d480409693e3975d5fec9976 P 5755a662553c4a84b76c8b2c48880567 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251215 21:02:03.305848 31544 raft_consensus.cc:2272] T c7d65525d480409693e3975d5fec9976 P 5755a662553c4a84b76c8b2c48880567 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251215 21:02:03.306372 31544 ts_tablet_manager.cc:1916] T c7d65525d480409693e3975d5fec9976 P 5755a662553c4a84b76c8b2c48880567: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251215 21:02:03.307951 30872 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.28.113.1:33435 (error 108)
I20251215 21:02:03.308193 29124 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu with pid 31103
I20251215 21:02:03.309264 31547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:02:03.308814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.113.1:33435 (local address 127.0.0.1:45768)
1215 21:02:03.309030 (+ 216us) negotiation.cc:107] Waiting for socket to connect
1215 21:02:03.309085 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.113.1:33435: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":80,"threads_started":1}
W20251215 21:02:03.309387 30872 catalog_manager.cc:4712] TS 6e18dca41b204cf0a3ebedd00ec07906 (127.28.113.1:33435): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c7d65525d480409693e3975d5fec9976: Network error: Client connection negotiation failed: client connection to 127.28.113.1:33435: connect: Connection refused (error 111)
I20251215 21:02:03.362496 31547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:02:03.362148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.113.1:33435 (local address 127.0.0.1:45776)
1215 21:02:03.362330 (+ 182us) negotiation.cc:107] Waiting for socket to connect
1215 21:02:03.362403 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.113.1:33435: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20251215 21:02:03.363140 31547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:02:03.362967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.113.1:33435 (local address 127.0.0.1:45780)
1215 21:02:03.363045 (+ 78us) negotiation.cc:107] Waiting for socket to connect
1215 21:02:03.363085 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.113.1:33435: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
I20251215 21:02:03.393407 31544 ts_tablet_manager.cc:1929] T c7d65525d480409693e3975d5fec9976 P 5755a662553c4a84b76c8b2c48880567: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20251215 21:02:03.393541 31544 log.cc:1199] T c7d65525d480409693e3975d5fec9976 P 5755a662553c4a84b76c8b2c48880567: Deleting WAL directory at /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765832506145157-29124-0/minicluster-data/ts-2/wal/wals/c7d65525d480409693e3975d5fec9976
I20251215 21:02:03.393848 31544 ts_tablet_manager.cc:1950] T c7d65525d480409693e3975d5fec9976 P 5755a662553c4a84b76c8b2c48880567: Deleting consensus metadata
I20251215 21:02:03.397958 29124 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu with pid 31245
I20251215 21:02:03.398293 31547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:02:03.397997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.113.2:35261 (local address 127.0.0.1:34132)
1215 21:02:03.398152 (+ 155us) negotiation.cc:107] Waiting for socket to connect
1215 21:02:03.398205 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.113.2:35261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20251215 21:02:03.411242 29124 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu with pid 30855
2025-12-15T21:02:03Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (7346 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765832506145157-29124-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Dec 15 21:02:05 dist-test-slave-bjsf krb5kdc[31550](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Dec 15 21:02:05 dist-test-slave-bjsf krb5kdc[31550](info): set up 2 sockets
Dec 15 21:02:05 dist-test-slave-bjsf krb5kdc[31550](info): commencing operation
krb5kdc: starting...
W20251215 21:02:07.199824 29124 mini_kdc.cc:121] Time spent starting KDC: real 3.767s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765832506145157-29124-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765832506145157-29124-0/krb5kdc/test-user.keytab.
Dec 15 21:02:07 dist-test-slave-bjsf krb5kdc[31550](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765832527, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-12-15T21:02:07Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-12-15T21:02:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.113.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.113.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.113.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765832506145157-29124-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.28.113.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765832506145157-29124-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.28.113.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.28.113.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.28.113.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765832506145157-29124-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.28.113.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765832506145157-29124-0/krb5kdc/kudu.keytab.
I20251215 21:02:07.323017 29124 external_mini_cluster.cc:1366] Running /tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu
/tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765832506145157-29124-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765832506145157-29124-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765832506145157-29124-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765832506145157-29124-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.113.62:40857
--webserver_interface=127.28.113.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765832506145157-29124-0/krb5kdc/kudu.keytab
--principal=kudu/127.28.113.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.28.113.20:36977
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.113.62:40857
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765832506145157-29124-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765832506145157-29124-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765832506145157-29124-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251215 21:02:07.435040 31566 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251215 21:02:07.435293 31566 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251215 21:02:07.435372 31566 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251215 21:02:07.438654 31566 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251215 21:02:07.438723 31566 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251215 21:02:07.438755 31566 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251215 21:02:07.438779 31566 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251215 21:02:07.438817 31566 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251215 21:02:07.443115 31566 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.113.20:36977
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765832506145157-29124-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765832506145157-29124-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.113.62:40857
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765832506145157-29124-0/krb5kdc/kudu.keytab
--principal=kudu/127.28.113.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.113.62:40857
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765832506145157-29124-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.113.62
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31566
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765832506145157-29124-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision dc741732bb6a40213e11a13869fad06a49a209a3
build type DEBUG
built by None at 15 Dec 2025 20:43:14 UTC on 24a791456cd2
build id 9530
I20251215 21:02:07.444301 31566 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251215 21:02:07.445176 31566 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251215 21:02:07.451095 31571 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251215 21:02:07.451115 31572 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251215 21:02:07.451196 31566 server_base.cc:1047] running on GCE node
W20251215 21:02:07.451107 31574 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251215 21:02:07.452035 31566 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251215 21:02:07.453037 31566 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251215 21:02:07.454181 31566 hybrid_clock.cc:648] HybridClock initialized: now 1765832527454169 us; error 31 us; skew 500 ppm
Dec 15 21:02:07 dist-test-slave-bjsf krb5kdc[31550](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765832527, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.113.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251215 21:02:07.456642 31566 init.cc:377] Logged in from keytab as kudu/127.28.113.62@KRBTEST.COM (short username kudu)
I20251215 21:02:07.457705 31566 webserver.cc:492] Webserver started at http://127.28.113.62:35319/ using document root <none> and password file <none>
I20251215 21:02:07.458247 31566 fs_manager.cc:362] Metadata directory not provided
I20251215 21:02:07.458292 31566 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251215 21:02:07.458463 31566 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251215 21:02:07.460253 31566 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765832506145157-29124-0/minicluster-data/master-0/data/instance:
uuid: "21e2551aa9204ed59fbc1fd14490c4b5"
format_stamp: "Formatted at 2025-12-15 21:02:07 on dist-test-slave-bjsf"
server_key: "f510e6a94d07674d4f0f77c3e1a7d28c"
server_key_iv: "ab12dd9d301cea51c87600e29635b483"
server_key_version: "encryptionkey@0"
I20251215 21:02:07.460695 31566 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765832506145157-29124-0/minicluster-data/master-0/wal/instance:
uuid: "21e2551aa9204ed59fbc1fd14490c4b5"
format_stamp: "Formatted at 2025-12-15 21:02:07 on dist-test-slave-bjsf"
server_key: "f510e6a94d07674d4f0f77c3e1a7d28c"
server_key_iv: "ab12dd9d301cea51c87600e29635b483"
server_key_version: "encryptionkey@0"
I20251215 21:02:07.464223 31566 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20251215 21:02:07.466629 31581 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251215 21:02:07.467793 31566 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20251215 21:02:07.467916 31566 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765832506145157-29124-0/minicluster-data/master-0/data,/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765832506145157-29124-0/minicluster-data/master-0/wal
uuid: "21e2551aa9204ed59fbc1fd14490c4b5"
format_stamp: "Formatted at 2025-12-15 21:02:07 on dist-test-slave-bjsf"
server_key: "f510e6a94d07674d4f0f77c3e1a7d28c"
server_key_iv: "ab12dd9d301cea51c87600e29635b483"
server_key_version: "encryptionkey@0"
I20251215 21:02:07.468062 31566 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765832506145157-29124-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765832506145157-29124-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765832506145157-29124-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251215 21:02:07.487807 31566 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251215 21:02:07.494968 31566 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251215 21:02:07.495159 31566 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251215 21:02:07.502362 31633 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.28.113.62:40857 every 8 connection(s)
I20251215 21:02:07.502370 31566 rpc_server.cc:307] RPC server started. Bound to: 127.28.113.62:40857
I20251215 21:02:07.503429 31566 server_base.cc:1179] Dumped server information to /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765832506145157-29124-0/minicluster-data/master-0/data/info.pb
I20251215 21:02:07.506323 31634 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 21:02:07.507776 29124 external_mini_cluster.cc:1428] Started /tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu as pid 31566
I20251215 21:02:07.507899 29124 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765832506145157-29124-0/minicluster-data/master-0/wal/instance
I20251215 21:02:07.508164 29124 external_mini_cluster.cc:1468] Setting key df3acc83672d4d6765255de9cb8df8a6
I20251215 21:02:07.512429 31634 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 21e2551aa9204ed59fbc1fd14490c4b5: Bootstrap starting.
I20251215 21:02:07.514731 31634 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 21e2551aa9204ed59fbc1fd14490c4b5: Neither blocks nor log segments found. Creating new log.
Dec 15 21:02:07 dist-test-slave-bjsf krb5kdc[31550](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765832527, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.113.62@KRBTEST.COM
I20251215 21:02:07.515475 31634 log.cc:826] T 00000000000000000000000000000000 P 21e2551aa9204ed59fbc1fd14490c4b5: Log is configured to *not* fsync() on all Append() calls
I20251215 21:02:07.517669 31634 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 21e2551aa9204ed59fbc1fd14490c4b5: No bootstrap required, opened a new log
I20251215 21:02:07.520520 31637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:02:07.509483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40090 (local address 127.28.113.62:40857)
1215 21:02:07.509930 (+ 447us) server_negotiation.cc:207] Beginning negotiation
1215 21:02:07.509940 (+ 10us) server_negotiation.cc:400] Waiting for connection header
1215 21:02:07.509976 (+ 36us) server_negotiation.cc:408] Connection header received
1215 21:02:07.510486 (+ 510us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1215 21:02:07.510502 (+ 16us) server_negotiation.cc:462] Received NEGOTIATE request from client
1215 21:02:07.510807 (+ 305us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1215 21:02:07.511115 (+ 308us) server_negotiation.cc:227] Negotiated authn=SASL
1215 21:02:07.512009 (+ 894us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:02:07.512781 (+ 772us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1215 21:02:07.513383 (+ 602us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:02:07.513654 (+ 271us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:02:07.515739 (+ 2085us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1215 21:02:07.515759 (+ 20us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1215 21:02:07.515773 (+ 14us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1215 21:02:07.515815 (+ 42us) server_negotiation.cc:923] calling sasl_server_start()
1215 21:02:07.517866 (+ 2051us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1215 21:02:07.518303 (+ 437us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1215 21:02:07.518309 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1215 21:02:07.518315 (+ 6us) server_negotiation.cc:968] calling sasl_server_step()
1215 21:02:07.518407 (+ 92us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1215 21:02:07.518599 (+ 192us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1215 21:02:07.518602 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1215 21:02:07.518604 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1215 21:02:07.519036 (+ 432us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1215 21:02:07.519246 (+ 210us) server_negotiation.cc:1036] Waiting for connection context
1215 21:02:07.519413 (+ 167us) server_negotiation.cc:300] Negotiation successful
1215 21:02:07.519687 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":310,"thread_start_us":145,"threads_started":1}
I20251215 21:02:07.520933 31634 raft_consensus.cc:359] T 00000000000000000000000000000000 P 21e2551aa9204ed59fbc1fd14490c4b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21e2551aa9204ed59fbc1fd14490c4b5" member_type: VOTER last_known_addr { host: "127.28.113.62" port: 40857 } }
I20251215 21:02:07.521152 31634 raft_consensus.cc:385] T 00000000000000000000000000000000 P 21e2551aa9204ed59fbc1fd14490c4b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251215 21:02:07.521206 31634 raft_consensus.cc:740] T 00000000000000000000000000000000 P 21e2551aa9204ed59fbc1fd14490c4b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 21e2551aa9204ed59fbc1fd14490c4b5, State: Initialized, Role: FOLLOWER
I20251215 21:02:07.521646 31634 consensus_queue.cc:260] T 00000000000000000000000000000000 P 21e2551aa9204ed59fbc1fd14490c4b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21e2551aa9204ed59fbc1fd14490c4b5" member_type: VOTER last_known_addr { host: "127.28.113.62" port: 40857 } }
I20251215 21:02:07.521766 31634 raft_consensus.cc:399] T 00000000000000000000000000000000 P 21e2551aa9204ed59fbc1fd14490c4b5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251215 21:02:07.521826 31634 raft_consensus.cc:493] T 00000000000000000000000000000000 P 21e2551aa9204ed59fbc1fd14490c4b5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251215 21:02:07.521919 31634 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 21e2551aa9204ed59fbc1fd14490c4b5 [term 0 FOLLOWER]: Advancing to term 1
I20251215 21:02:07.522835 31634 raft_consensus.cc:515] T 00000000000000000000000000000000 P 21e2551aa9204ed59fbc1fd14490c4b5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21e2551aa9204ed59fbc1fd14490c4b5" member_type: VOTER last_known_addr { host: "127.28.113.62" port: 40857 } }
I20251215 21:02:07.523170 31634 leader_election.cc:304] T 00000000000000000000000000000000 P 21e2551aa9204ed59fbc1fd14490c4b5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 21e2551aa9204ed59fbc1fd14490c4b5; no voters:
I20251215 21:02:07.523448 31634 leader_election.cc:290] T 00000000000000000000000000000000 P 21e2551aa9204ed59fbc1fd14490c4b5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20251215 21:02:07.523648 31639 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 21e2551aa9204ed59fbc1fd14490c4b5 [term 1 FOLLOWER]: Leader election won for term 1
I20251215 21:02:07.523836 31639 raft_consensus.cc:697] T 00000000000000000000000000000000 P 21e2551aa9204ed59fbc1fd14490c4b5 [term 1 LEADER]: Becoming Leader. State: Replica: 21e2551aa9204ed59fbc1fd14490c4b5, State: Running, Role: LEADER
I20251215 21:02:07.524150 31639 consensus_queue.cc:237] T 00000000000000000000000000000000 P 21e2551aa9204ed59fbc1fd14490c4b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21e2551aa9204ed59fbc1fd14490c4b5" member_type: VOTER last_known_addr { host: "127.28.113.62" port: 40857 } }
I20251215 21:02:07.524574 31634 sys_catalog.cc:565] T 00000000000000000000000000000000 P 21e2551aa9204ed59fbc1fd14490c4b5 [sys.catalog]: configured and running, proceeding with master startup.
I20251215 21:02:07.525698 31640 sys_catalog.cc:455] T 00000000000000000000000000000000 P 21e2551aa9204ed59fbc1fd14490c4b5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "21e2551aa9204ed59fbc1fd14490c4b5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21e2551aa9204ed59fbc1fd14490c4b5" member_type: VOTER last_known_addr { host: "127.28.113.62" port: 40857 } } }
I20251215 21:02:07.525838 31640 sys_catalog.cc:458] T 00000000000000000000000000000000 P 21e2551aa9204ed59fbc1fd14490c4b5 [sys.catalog]: This master's current role is: LEADER
I20251215 21:02:07.525698 31641 sys_catalog.cc:455] T 00000000000000000000000000000000 P 21e2551aa9204ed59fbc1fd14490c4b5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 21e2551aa9204ed59fbc1fd14490c4b5. Latest consensus state: current_term: 1 leader_uuid: "21e2551aa9204ed59fbc1fd14490c4b5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21e2551aa9204ed59fbc1fd14490c4b5" member_type: VOTER last_known_addr { host: "127.28.113.62" port: 40857 } } }
I20251215 21:02:07.525899 31641 sys_catalog.cc:458] T 00000000000000000000000000000000 P 21e2551aa9204ed59fbc1fd14490c4b5 [sys.catalog]: This master's current role is: LEADER
I20251215 21:02:07.526170 31648 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251215 21:02:07.529033 31648 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251215 21:02:07.534554 31648 catalog_manager.cc:1357] Generated new cluster ID: feaab5f91bc24142926ba1642ab54000
I20251215 21:02:07.534700 31648 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251215 21:02:07.547798 31648 catalog_manager.cc:1380] Generated new certificate authority record
I20251215 21:02:07.549134 31648 catalog_manager.cc:1514] Loading token signing keys...
I20251215 21:02:07.557895 31648 catalog_manager.cc:6027] T 00000000000000000000000000000000 P 21e2551aa9204ed59fbc1fd14490c4b5: Generated new TSK 0
I20251215 21:02:07.558718 31648 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.113.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.113.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.113.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765832506145157-29124-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.28.113.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765832506145157-29124-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.28.113.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.28.113.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.28.113.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765832506145157-29124-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.28.113.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765832506145157-29124-0/krb5kdc/kudu.keytab.
I20251215 21:02:07.603829 29124 external_mini_cluster.cc:1366] Running /tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu
/tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765832506145157-29124-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765832506145157-29124-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765832506145157-29124-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765832506145157-29124-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.113.1:0
--local_ip_for_outbound_sockets=127.28.113.1
--webserver_interface=127.28.113.1
--webserver_port=0
--tserver_master_addrs=127.28.113.62:40857
--keytab_file=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765832506145157-29124-0/krb5kdc/kudu.keytab
--principal=kudu/127.28.113.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.28.113.20:36977
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765832506145157-29124-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765832506145157-29124-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765832506145157-29124-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251215 21:02:07.710696 31662 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251215 21:02:07.710925 31662 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251215 21:02:07.710988 31662 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251215 21:02:07.714236 31662 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251215 21:02:07.714308 31662 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251215 21:02:07.714408 31662 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.113.1
I20251215 21:02:07.718662 31662 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.113.20:36977
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765832506145157-29124-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765832506145157-29124-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765832506145157-29124-0/krb5kdc/kudu.keytab
--principal=kudu/127.28.113.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.113.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765832506145157-29124-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.113.1
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.28.113.62:40857
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31662
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.113.1
--log_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765832506145157-29124-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision dc741732bb6a40213e11a13869fad06a49a209a3
build type DEBUG
built by None at 15 Dec 2025 20:43:14 UTC on 24a791456cd2
build id 9530
I20251215 21:02:07.719733 31662 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251215 21:02:07.720602 31662 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251215 21:02:07.727021 31668 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251215 21:02:07.727020 31667 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251215 21:02:07.727093 31662 server_base.cc:1047] running on GCE node
W20251215 21:02:07.727020 31670 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251215 21:02:07.727624 31662 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251215 21:02:07.728192 31662 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251215 21:02:07.729341 31662 hybrid_clock.cc:648] HybridClock initialized: now 1765832527729322 us; error 35 us; skew 500 ppm
Dec 15 21:02:07 dist-test-slave-bjsf krb5kdc[31550](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765832527, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.113.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251215 21:02:07.731786 31662 init.cc:377] Logged in from keytab as kudu/127.28.113.1@KRBTEST.COM (short username kudu)
I20251215 21:02:07.732836 31662 webserver.cc:492] Webserver started at http://127.28.113.1:37839/ using document root <none> and password file <none>
I20251215 21:02:07.733377 31662 fs_manager.cc:362] Metadata directory not provided
I20251215 21:02:07.733423 31662 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251215 21:02:07.733589 31662 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251215 21:02:07.735217 31662 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765832506145157-29124-0/minicluster-data/ts-0/data/instance:
uuid: "32041cec4c054c4292859828606be32d"
format_stamp: "Formatted at 2025-12-15 21:02:07 on dist-test-slave-bjsf"
server_key: "50157591eddc342b9e3e707f554e63ce"
server_key_iv: "2d8ba3e5064c8ed673578ab2345c59e7"
server_key_version: "encryptionkey@0"
I20251215 21:02:07.735651 31662 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765832506145157-29124-0/minicluster-data/ts-0/wal/instance:
uuid: "32041cec4c054c4292859828606be32d"
format_stamp: "Formatted at 2025-12-15 21:02:07 on dist-test-slave-bjsf"
server_key: "50157591eddc342b9e3e707f554e63ce"
server_key_iv: "2d8ba3e5064c8ed673578ab2345c59e7"
server_key_version: "encryptionkey@0"
I20251215 21:02:07.738893 31662 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20251215 21:02:07.741043 31677 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251215 21:02:07.742025 31662 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20251215 21:02:07.742125 31662 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765832506145157-29124-0/minicluster-data/ts-0/data,/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765832506145157-29124-0/minicluster-data/ts-0/wal
uuid: "32041cec4c054c4292859828606be32d"
format_stamp: "Formatted at 2025-12-15 21:02:07 on dist-test-slave-bjsf"
server_key: "50157591eddc342b9e3e707f554e63ce"
server_key_iv: "2d8ba3e5064c8ed673578ab2345c59e7"
server_key_version: "encryptionkey@0"
I20251215 21:02:07.742210 31662 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765832506145157-29124-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765832506145157-29124-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765832506145157-29124-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251215 21:02:07.751137 31662 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251215 21:02:07.754024 31662 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251215 21:02:07.754189 31662 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251215 21:02:07.754719 31662 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251215 21:02:07.755542 31662 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251215 21:02:07.755589 31662 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251215 21:02:07.755626 31662 ts_tablet_manager.cc:616] Registered 0 tablets
I20251215 21:02:07.755661 31662 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251215 21:02:07.763875 31662 rpc_server.cc:307] RPC server started. Bound to: 127.28.113.1:45887
I20251215 21:02:07.763916 31790 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.28.113.1:45887 every 8 connection(s)
I20251215 21:02:07.764845 31662 server_base.cc:1179] Dumped server information to /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765832506145157-29124-0/minicluster-data/ts-0/data/info.pb
I20251215 21:02:07.769069 29124 external_mini_cluster.cc:1428] Started /tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu as pid 31662
I20251215 21:02:07.769167 29124 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765832506145157-29124-0/minicluster-data/ts-0/wal/instance
I20251215 21:02:07.769371 29124 external_mini_cluster.cc:1468] Setting key 7a3f5fbbc7f61e01b4145a557f6449e4
Dec 15 21:02:07 dist-test-slave-bjsf krb5kdc[31550](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765832527, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.113.1@KRBTEST.COM for kudu/127.28.113.62@KRBTEST.COM
I20251215 21:02:07.776716 31637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:02:07.766495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.113.1:40369 (local address 127.28.113.62:40857)
1215 21:02:07.766681 (+ 186us) server_negotiation.cc:207] Beginning negotiation
1215 21:02:07.766686 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1215 21:02:07.767311 (+ 625us) server_negotiation.cc:408] Connection header received
1215 21:02:07.768175 (+ 864us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1215 21:02:07.768179 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1215 21:02:07.768227 (+ 48us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1215 21:02:07.768292 (+ 65us) server_negotiation.cc:227] Negotiated authn=SASL
1215 21:02:07.769970 (+ 1678us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:02:07.770471 (+ 501us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1215 21:02:07.771086 (+ 615us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:02:07.771258 (+ 172us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:02:07.773470 (+ 2212us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1215 21:02:07.773489 (+ 19us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1215 21:02:07.773491 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1215 21:02:07.773515 (+ 24us) server_negotiation.cc:923] calling sasl_server_start()
1215 21:02:07.774894 (+ 1379us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1215 21:02:07.775333 (+ 439us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1215 21:02:07.775337 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1215 21:02:07.775339 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1215 21:02:07.775381 (+ 42us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1215 21:02:07.775618 (+ 237us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1215 21:02:07.775622 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1215 21:02:07.775624 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1215 21:02:07.775787 (+ 163us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1215 21:02:07.775871 (+ 84us) server_negotiation.cc:1036] Waiting for connection context
1215 21:02:07.776401 (+ 530us) server_negotiation.cc:300] Negotiation successful
1215 21:02:07.776541 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
WARNING: no policy specified for kudu/127.28.113.2@KRBTEST.COM; defaulting to no policy
I20251215 21:02:07.777398 31793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:02:07.766766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.113.62:40857 (local address 127.28.113.1:40369)
1215 21:02:07.767178 (+ 412us) negotiation.cc:107] Waiting for socket to connect
1215 21:02:07.767214 (+ 36us) client_negotiation.cc:174] Beginning negotiation
1215 21:02:07.767941 (+ 727us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1215 21:02:07.768437 (+ 496us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1215 21:02:07.768445 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
1215 21:02:07.768821 (+ 376us) client_negotiation.cc:189] Negotiated authn=SASL
1215 21:02:07.769811 (+ 990us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:02:07.769829 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:02:07.770581 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1215 21:02:07.770584 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1215 21:02:07.770985 (+ 401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:02:07.770992 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:02:07.771200 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:02:07.771845 (+ 645us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1215 21:02:07.771867 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
1215 21:02:07.773325 (+ 1458us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1215 21:02:07.775006 (+ 1681us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1215 21:02:07.775012 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1215 21:02:07.775025 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
1215 21:02:07.775244 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1215 21:02:07.775455 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1215 21:02:07.775458 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1215 21:02:07.775460 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1215 21:02:07.775541 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1215 21:02:07.775890 (+ 349us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1215 21:02:07.775897 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1215 21:02:07.776172 (+ 275us) client_negotiation.cc:769] Sending connection context
1215 21:02:07.776360 (+ 188us) client_negotiation.cc:240] Negotiation successful
1215 21:02:07.776626 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":103,"threads_started":1}
I20251215 21:02:07.778558 31791 heartbeater.cc:344] Connected to a master server at 127.28.113.62:40857
I20251215 21:02:07.778813 31791 heartbeater.cc:461] Registering TS with master...
I20251215 21:02:07.779356 31791 heartbeater.cc:507] Master 127.28.113.62:40857 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.113.2@KRBTEST.COM" created.
I20251215 21:02:07.780915 31598 ts_manager.cc:194] Registered new tserver with Master: 32041cec4c054c4292859828606be32d (127.28.113.1:45887)
I20251215 21:02:07.782397 31598 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.113.1@KRBTEST.COM'} at 127.28.113.1:40369
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.113.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765832506145157-29124-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.28.113.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765832506145157-29124-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.28.113.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.28.113.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.28.113.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765832506145157-29124-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.28.113.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765832506145157-29124-0/krb5kdc/kudu.keytab.
I20251215 21:02:07.812522 29124 external_mini_cluster.cc:1366] Running /tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu
/tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765832506145157-29124-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765832506145157-29124-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765832506145157-29124-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765832506145157-29124-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.113.2:0
--local_ip_for_outbound_sockets=127.28.113.2
--webserver_interface=127.28.113.2
--webserver_port=0
--tserver_master_addrs=127.28.113.62:40857
--keytab_file=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765832506145157-29124-0/krb5kdc/kudu.keytab
--principal=kudu/127.28.113.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.28.113.20:36977
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765832506145157-29124-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765832506145157-29124-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765832506145157-29124-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251215 21:02:07.919584 31798 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251215 21:02:07.919834 31798 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251215 21:02:07.919898 31798 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251215 21:02:07.923277 31798 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251215 21:02:07.923362 31798 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251215 21:02:07.923466 31798 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.113.2
I20251215 21:02:07.928030 31798 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.113.20:36977
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765832506145157-29124-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765832506145157-29124-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765832506145157-29124-0/krb5kdc/kudu.keytab
--principal=kudu/127.28.113.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.113.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765832506145157-29124-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.113.2
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.28.113.62:40857
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31798
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.113.2
--log_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765832506145157-29124-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision dc741732bb6a40213e11a13869fad06a49a209a3
build type DEBUG
built by None at 15 Dec 2025 20:43:14 UTC on 24a791456cd2
build id 9530
I20251215 21:02:07.929154 31798 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251215 21:02:07.929997 31798 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251215 21:02:07.936398 31804 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251215 21:02:07.936383 31803 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251215 21:02:07.936384 31806 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251215 21:02:07.936729 31798 server_base.cc:1047] running on GCE node
I20251215 21:02:07.937114 31798 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251215 21:02:07.937650 31798 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251215 21:02:07.938792 31798 hybrid_clock.cc:648] HybridClock initialized: now 1765832527938778 us; error 30 us; skew 500 ppm
Dec 15 21:02:07 dist-test-slave-bjsf krb5kdc[31550](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765832527, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.113.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251215 21:02:07.941274 31798 init.cc:377] Logged in from keytab as kudu/127.28.113.2@KRBTEST.COM (short username kudu)
I20251215 21:02:07.942278 31798 webserver.cc:492] Webserver started at http://127.28.113.2:32769/ using document root <none> and password file <none>
I20251215 21:02:07.942803 31798 fs_manager.cc:362] Metadata directory not provided
I20251215 21:02:07.942848 31798 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251215 21:02:07.943010 31798 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251215 21:02:07.944754 31798 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765832506145157-29124-0/minicluster-data/ts-1/data/instance:
uuid: "0c55f91c48774c9094cb5caff081bb66"
format_stamp: "Formatted at 2025-12-15 21:02:07 on dist-test-slave-bjsf"
server_key: "f4092ebfeb6519cbb4fc23daac436c24"
server_key_iv: "d036c2178d396d08cca1c3aa58b82102"
server_key_version: "encryptionkey@0"
I20251215 21:02:07.945199 31798 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765832506145157-29124-0/minicluster-data/ts-1/wal/instance:
uuid: "0c55f91c48774c9094cb5caff081bb66"
format_stamp: "Formatted at 2025-12-15 21:02:07 on dist-test-slave-bjsf"
server_key: "f4092ebfeb6519cbb4fc23daac436c24"
server_key_iv: "d036c2178d396d08cca1c3aa58b82102"
server_key_version: "encryptionkey@0"
I20251215 21:02:07.948594 31798 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.002s
I20251215 21:02:07.950683 31813 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251215 21:02:07.951664 31798 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20251215 21:02:07.951758 31798 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765832506145157-29124-0/minicluster-data/ts-1/data,/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765832506145157-29124-0/minicluster-data/ts-1/wal
uuid: "0c55f91c48774c9094cb5caff081bb66"
format_stamp: "Formatted at 2025-12-15 21:02:07 on dist-test-slave-bjsf"
server_key: "f4092ebfeb6519cbb4fc23daac436c24"
server_key_iv: "d036c2178d396d08cca1c3aa58b82102"
server_key_version: "encryptionkey@0"
I20251215 21:02:07.951848 31798 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765832506145157-29124-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765832506145157-29124-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765832506145157-29124-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251215 21:02:07.963106 31798 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251215 21:02:07.965847 31798 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251215 21:02:07.966028 31798 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251215 21:02:07.966539 31798 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251215 21:02:07.967372 31798 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251215 21:02:07.967418 31798 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251215 21:02:07.967469 31798 ts_tablet_manager.cc:616] Registered 0 tablets
I20251215 21:02:07.967494 31798 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251215 21:02:07.975909 31798 rpc_server.cc:307] RPC server started. Bound to: 127.28.113.2:35675
I20251215 21:02:07.975934 31926 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.28.113.2:35675 every 8 connection(s)
I20251215 21:02:07.976876 31798 server_base.cc:1179] Dumped server information to /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765832506145157-29124-0/minicluster-data/ts-1/data/info.pb
I20251215 21:02:07.977602 29124 external_mini_cluster.cc:1428] Started /tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu as pid 31798
I20251215 21:02:07.977710 29124 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765832506145157-29124-0/minicluster-data/ts-1/wal/instance
I20251215 21:02:07.977968 29124 external_mini_cluster.cc:1468] Setting key de230495c14f33e19ed609f08669460e
Dec 15 21:02:07 dist-test-slave-bjsf krb5kdc[31550](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765832527, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.113.2@KRBTEST.COM for kudu/127.28.113.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.28.113.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.113.3@KRBTEST.COM" created.
I20251215 21:02:07.990259 31637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:02:07.978733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.113.2:46435 (local address 127.28.113.62:40857)
1215 21:02:07.978878 (+ 145us) server_negotiation.cc:207] Beginning negotiation
1215 21:02:07.978883 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1215 21:02:07.979681 (+ 798us) server_negotiation.cc:408] Connection header received
1215 21:02:07.980571 (+ 890us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1215 21:02:07.980575 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1215 21:02:07.980633 (+ 58us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1215 21:02:07.980705 (+ 72us) server_negotiation.cc:227] Negotiated authn=SASL
1215 21:02:07.982693 (+ 1988us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:02:07.983245 (+ 552us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1215 21:02:07.984201 (+ 956us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:02:07.984335 (+ 134us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:02:07.986844 (+ 2509us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1215 21:02:07.986867 (+ 23us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1215 21:02:07.986869 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1215 21:02:07.986899 (+ 30us) server_negotiation.cc:923] calling sasl_server_start()
1215 21:02:07.988264 (+ 1365us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1215 21:02:07.988818 (+ 554us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1215 21:02:07.988821 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1215 21:02:07.988849 (+ 28us) server_negotiation.cc:968] calling sasl_server_step()
1215 21:02:07.988911 (+ 62us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1215 21:02:07.989179 (+ 268us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1215 21:02:07.989183 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1215 21:02:07.989185 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1215 21:02:07.989354 (+ 169us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1215 21:02:07.989441 (+ 87us) server_negotiation.cc:1036] Waiting for connection context
1215 21:02:07.989954 (+ 513us) server_negotiation.cc:300] Negotiation successful
1215 21:02:07.990089 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20251215 21:02:07.990969 31929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:02:07.979056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.113.62:40857 (local address 127.28.113.2:46435)
1215 21:02:07.979551 (+ 495us) negotiation.cc:107] Waiting for socket to connect
1215 21:02:07.979592 (+ 41us) client_negotiation.cc:174] Beginning negotiation
1215 21:02:07.980374 (+ 782us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1215 21:02:07.980882 (+ 508us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1215 21:02:07.980890 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
1215 21:02:07.981277 (+ 387us) client_negotiation.cc:189] Negotiated authn=SASL
1215 21:02:07.982495 (+ 1218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:02:07.982517 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:02:07.983389 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1215 21:02:07.983393 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1215 21:02:07.984076 (+ 683us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:02:07.984087 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:02:07.984362 (+ 275us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:02:07.985009 (+ 647us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1215 21:02:07.985045 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
1215 21:02:07.986677 (+ 1632us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1215 21:02:07.988406 (+ 1729us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1215 21:02:07.988412 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1215 21:02:07.988428 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
1215 21:02:07.988712 (+ 284us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1215 21:02:07.988992 (+ 280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1215 21:02:07.988995 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1215 21:02:07.988996 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1215 21:02:07.989092 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1215 21:02:07.989453 (+ 361us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1215 21:02:07.989461 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1215 21:02:07.989714 (+ 253us) client_negotiation.cc:769] Sending connection context
1215 21:02:07.989918 (+ 204us) client_negotiation.cc:240] Negotiation successful
1215 21:02:07.990205 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":307,"thread_start_us":140,"threads_started":1}
I20251215 21:02:07.992166 31927 heartbeater.cc:344] Connected to a master server at 127.28.113.62:40857
I20251215 21:02:07.992408 31927 heartbeater.cc:461] Registering TS with master...
I20251215 21:02:07.992944 31927 heartbeater.cc:507] Master 127.28.113.62:40857 requested a full tablet report, sending...
I20251215 21:02:07.994042 31598 ts_manager.cc:194] Registered new tserver with Master: 0c55f91c48774c9094cb5caff081bb66 (127.28.113.2:35675)
I20251215 21:02:07.994858 31598 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.113.2@KRBTEST.COM'} at 127.28.113.2:46435
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.113.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765832506145157-29124-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.28.113.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765832506145157-29124-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.28.113.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.28.113.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.28.113.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765832506145157-29124-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.28.113.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765832506145157-29124-0/krb5kdc/kudu.keytab.
I20251215 21:02:08.022995 29124 external_mini_cluster.cc:1366] Running /tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu
/tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765832506145157-29124-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765832506145157-29124-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765832506145157-29124-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765832506145157-29124-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.113.3:0
--local_ip_for_outbound_sockets=127.28.113.3
--webserver_interface=127.28.113.3
--webserver_port=0
--tserver_master_addrs=127.28.113.62:40857
--keytab_file=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765832506145157-29124-0/krb5kdc/kudu.keytab
--principal=kudu/127.28.113.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.28.113.20:36977
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765832506145157-29124-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765832506145157-29124-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765832506145157-29124-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251215 21:02:08.128446 31934 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251215 21:02:08.128679 31934 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251215 21:02:08.128755 31934 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251215 21:02:08.131949 31934 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251215 21:02:08.132051 31934 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251215 21:02:08.132149 31934 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.113.3
I20251215 21:02:08.136471 31934 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.113.20:36977
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765832506145157-29124-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765832506145157-29124-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765832506145157-29124-0/krb5kdc/kudu.keytab
--principal=kudu/127.28.113.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.113.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765832506145157-29124-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.113.3
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.28.113.62:40857
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31934
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.113.3
--log_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765832506145157-29124-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision dc741732bb6a40213e11a13869fad06a49a209a3
build type DEBUG
built by None at 15 Dec 2025 20:43:14 UTC on 24a791456cd2
build id 9530
I20251215 21:02:08.137552 31934 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251215 21:02:08.138338 31934 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251215 21:02:08.144541 31939 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251215 21:02:08.144603 31942 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251215 21:02:08.144604 31940 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251215 21:02:08.144729 31934 server_base.cc:1047] running on GCE node
I20251215 21:02:08.145131 31934 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251215 21:02:08.145609 31934 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251215 21:02:08.146747 31934 hybrid_clock.cc:648] HybridClock initialized: now 1765832528146731 us; error 31 us; skew 500 ppm
Dec 15 21:02:08 dist-test-slave-bjsf krb5kdc[31550](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765832528, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.113.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251215 21:02:08.149168 31934 init.cc:377] Logged in from keytab as kudu/127.28.113.3@KRBTEST.COM (short username kudu)
I20251215 21:02:08.150131 31934 webserver.cc:492] Webserver started at http://127.28.113.3:38041/ using document root <none> and password file <none>
I20251215 21:02:08.150624 31934 fs_manager.cc:362] Metadata directory not provided
I20251215 21:02:08.150668 31934 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251215 21:02:08.150826 31934 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251215 21:02:08.152482 31934 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765832506145157-29124-0/minicluster-data/ts-2/data/instance:
uuid: "f5ecb317fbad464bafd4330fe2dbeb4b"
format_stamp: "Formatted at 2025-12-15 21:02:08 on dist-test-slave-bjsf"
server_key: "9d67eec335af78a3313b7716423fba47"
server_key_iv: "c7fd8689be6afd0dc97bb3f4637cf345"
server_key_version: "encryptionkey@0"
I20251215 21:02:08.152869 31934 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765832506145157-29124-0/minicluster-data/ts-2/wal/instance:
uuid: "f5ecb317fbad464bafd4330fe2dbeb4b"
format_stamp: "Formatted at 2025-12-15 21:02:08 on dist-test-slave-bjsf"
server_key: "9d67eec335af78a3313b7716423fba47"
server_key_iv: "c7fd8689be6afd0dc97bb3f4637cf345"
server_key_version: "encryptionkey@0"
I20251215 21:02:08.156111 31934 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20251215 21:02:08.158212 31949 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251215 21:02:08.159161 31934 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20251215 21:02:08.159258 31934 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765832506145157-29124-0/minicluster-data/ts-2/data,/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765832506145157-29124-0/minicluster-data/ts-2/wal
uuid: "f5ecb317fbad464bafd4330fe2dbeb4b"
format_stamp: "Formatted at 2025-12-15 21:02:08 on dist-test-slave-bjsf"
server_key: "9d67eec335af78a3313b7716423fba47"
server_key_iv: "c7fd8689be6afd0dc97bb3f4637cf345"
server_key_version: "encryptionkey@0"
I20251215 21:02:08.159338 31934 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765832506145157-29124-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765832506145157-29124-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765832506145157-29124-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251215 21:02:08.188244 31934 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251215 21:02:08.190819 31934 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251215 21:02:08.190997 31934 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251215 21:02:08.191516 31934 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251215 21:02:08.192404 31934 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251215 21:02:08.192451 31934 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251215 21:02:08.192503 31934 ts_tablet_manager.cc:616] Registered 0 tablets
I20251215 21:02:08.192531 31934 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251215 21:02:08.201053 31934 rpc_server.cc:307] RPC server started. Bound to: 127.28.113.3:40271
I20251215 21:02:08.201077 32062 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.28.113.3:40271 every 8 connection(s)
I20251215 21:02:08.201967 31934 server_base.cc:1179] Dumped server information to /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765832506145157-29124-0/minicluster-data/ts-2/data/info.pb
I20251215 21:02:08.208529 29124 external_mini_cluster.cc:1428] Started /tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu as pid 31934
I20251215 21:02:08.208653 29124 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765832506145157-29124-0/minicluster-data/ts-2/wal/instance
I20251215 21:02:08.208877 29124 external_mini_cluster.cc:1468] Setting key b74dc4e91f8552891b115d3c6815906d
Dec 15 21:02:08 dist-test-slave-bjsf krb5kdc[31550](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765832528, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.113.3@KRBTEST.COM for kudu/127.28.113.62@KRBTEST.COM
I20251215 21:02:08.213681 31637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:02:08.203687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.113.3:52335 (local address 127.28.113.62:40857)
1215 21:02:08.203859 (+ 172us) server_negotiation.cc:207] Beginning negotiation
1215 21:02:08.203864 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1215 21:02:08.204506 (+ 642us) server_negotiation.cc:408] Connection header received
1215 21:02:08.205317 (+ 811us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1215 21:02:08.205320 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1215 21:02:08.205371 (+ 51us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1215 21:02:08.205449 (+ 78us) server_negotiation.cc:227] Negotiated authn=SASL
1215 21:02:08.207115 (+ 1666us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:02:08.207613 (+ 498us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1215 21:02:08.208260 (+ 647us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:02:08.208423 (+ 163us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:02:08.210551 (+ 2128us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1215 21:02:08.210565 (+ 14us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1215 21:02:08.210568 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1215 21:02:08.210601 (+ 33us) server_negotiation.cc:923] calling sasl_server_start()
1215 21:02:08.211941 (+ 1340us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1215 21:02:08.212416 (+ 475us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1215 21:02:08.212419 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1215 21:02:08.212420 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1215 21:02:08.212461 (+ 41us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1215 21:02:08.212717 (+ 256us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1215 21:02:08.212720 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1215 21:02:08.212722 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1215 21:02:08.212902 (+ 180us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1215 21:02:08.212991 (+ 89us) server_negotiation.cc:1036] Waiting for connection context
1215 21:02:08.213444 (+ 453us) server_negotiation.cc:300] Negotiation successful
1215 21:02:08.213555 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20251215 21:02:08.214336 32065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:02:08.203939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.113.62:40857 (local address 127.28.113.3:52335)
1215 21:02:08.204374 (+ 435us) negotiation.cc:107] Waiting for socket to connect
1215 21:02:08.204411 (+ 37us) client_negotiation.cc:174] Beginning negotiation
1215 21:02:08.205124 (+ 713us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1215 21:02:08.205639 (+ 515us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1215 21:02:08.205647 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
1215 21:02:08.206047 (+ 400us) client_negotiation.cc:189] Negotiated authn=SASL
1215 21:02:08.206961 (+ 914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:02:08.206976 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:02:08.207721 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1215 21:02:08.207724 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1215 21:02:08.208178 (+ 454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:02:08.208186 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:02:08.208360 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:02:08.208969 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1215 21:02:08.209002 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
1215 21:02:08.210428 (+ 1426us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1215 21:02:08.212094 (+ 1666us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1215 21:02:08.212099 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1215 21:02:08.212109 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
1215 21:02:08.212337 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1215 21:02:08.212523 (+ 186us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1215 21:02:08.212525 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1215 21:02:08.212527 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1215 21:02:08.212606 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1215 21:02:08.212991 (+ 385us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1215 21:02:08.212996 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1215 21:02:08.213230 (+ 234us) client_negotiation.cc:769] Sending connection context
1215 21:02:08.213399 (+ 169us) client_negotiation.cc:240] Negotiation successful
1215 21:02:08.213616 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":98,"threads_started":1}
I20251215 21:02:08.215363 32063 heartbeater.cc:344] Connected to a master server at 127.28.113.62:40857
I20251215 21:02:08.215588 32063 heartbeater.cc:461] Registering TS with master...
I20251215 21:02:08.216105 32063 heartbeater.cc:507] Master 127.28.113.62:40857 requested a full tablet report, sending...
I20251215 21:02:08.217063 31598 ts_manager.cc:194] Registered new tserver with Master: f5ecb317fbad464bafd4330fe2dbeb4b (127.28.113.3:40271)
I20251215 21:02:08.217620 31598 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.113.3@KRBTEST.COM'} at 127.28.113.3:52335
I20251215 21:02:08.222901 29124 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
W20251215 21:02:08.233338 31637 negotiation.cc:337] Failed RPC negotiation. Trace:
1215 21:02:08.232155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46114 (local address 127.28.113.62:40857)
1215 21:02:08.232275 (+ 120us) server_negotiation.cc:207] Beginning negotiation
1215 21:02:08.232278 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1215 21:02:08.232378 (+ 100us) server_negotiation.cc:408] Connection header received
1215 21:02:08.232504 (+ 126us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1215 21:02:08.232507 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1215 21:02:08.232553 (+ 46us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1215 21:02:08.232619 (+ 66us) server_negotiation.cc:227] Negotiated authn=SASL
1215 21:02:08.233230 (+ 611us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46114: BlockingRecv error: recv got EOF from 127.0.0.1:46114 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
I20251215 21:02:08.234503 29124 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu with pid 31662
I20251215 21:02:08.239324 29124 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu with pid 31798
I20251215 21:02:08.243716 29124 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu with pid 31934
I20251215 21:02:08.248020 29124 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu with pid 31566
2025-12-15T21:02:08Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (4835 ms)
[ RUN ] SecurityITest.TestRebalanceReportUnauthorized
Loading random data
Initializing database '/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765832506145157-29124-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Dec 15 21:02:08 dist-test-slave-bjsf krb5kdc[32075](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Dec 15 21:02:08 dist-test-slave-bjsf krb5kdc[32075](info): set up 2 sockets
Dec 15 21:02:08 dist-test-slave-bjsf krb5kdc[32075](info): commencing operation
krb5kdc: starting...
W20251215 21:02:10.063884 29124 mini_kdc.cc:121] Time spent starting KDC: real 1.796s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765832506145157-29124-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765832506145157-29124-0/krb5kdc/test-user.keytab.
Dec 15 21:02:10 dist-test-slave-bjsf krb5kdc[32075](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765832530, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-12-15T21:02:10Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-12-15T21:02:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.113.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.113.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.113.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765832506145157-29124-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.28.113.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765832506145157-29124-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.28.113.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.28.113.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.28.113.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765832506145157-29124-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.28.113.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765832506145157-29124-0/krb5kdc/kudu.keytab.
I20251215 21:02:10.185827 29124 external_mini_cluster.cc:1366] Running /tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu
/tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765832506145157-29124-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765832506145157-29124-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765832506145157-29124-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765832506145157-29124-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.113.62:37619
--webserver_interface=127.28.113.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765832506145157-29124-0/krb5kdc/kudu.keytab
--principal=kudu/127.28.113.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.28.113.20:41313
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.113.62:37619
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765832506145157-29124-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765832506145157-29124-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765832506145157-29124-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251215 21:02:10.288424 32091 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251215 21:02:10.288628 32091 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251215 21:02:10.288691 32091 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251215 21:02:10.291821 32091 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251215 21:02:10.291872 32091 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251215 21:02:10.291898 32091 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251215 21:02:10.291929 32091 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251215 21:02:10.291982 32091 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251215 21:02:10.296061 32091 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.113.20:41313
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765832506145157-29124-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765832506145157-29124-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.113.62:37619
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765832506145157-29124-0/krb5kdc/kudu.keytab
--principal=kudu/127.28.113.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.113.62:37619
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765832506145157-29124-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.113.62
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32091
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765832506145157-29124-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision dc741732bb6a40213e11a13869fad06a49a209a3
build type DEBUG
built by None at 15 Dec 2025 20:43:14 UTC on 24a791456cd2
build id 9530
I20251215 21:02:10.297019 32091 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251215 21:02:10.297792 32091 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251215 21:02:10.302915 32096 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251215 21:02:10.302899 32099 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251215 21:02:10.302891 32097 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251215 21:02:10.303122 32091 server_base.cc:1047] running on GCE node
I20251215 21:02:10.303583 32091 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251215 21:02:10.304427 32091 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251215 21:02:10.305565 32091 hybrid_clock.cc:648] HybridClock initialized: now 1765832530305557 us; error 25 us; skew 500 ppm
Dec 15 21:02:10 dist-test-slave-bjsf krb5kdc[32075](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765832530, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.113.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251215 21:02:10.307782 32091 init.cc:377] Logged in from keytab as kudu/127.28.113.62@KRBTEST.COM (short username kudu)
I20251215 21:02:10.308717 32091 webserver.cc:492] Webserver started at http://127.28.113.62:33671/ using document root <none> and password file <none>
I20251215 21:02:10.309207 32091 fs_manager.cc:362] Metadata directory not provided
I20251215 21:02:10.309254 32091 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251215 21:02:10.309427 32091 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251215 21:02:10.310993 32091 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765832506145157-29124-0/minicluster-data/master-0/data/instance:
uuid: "b4479ad8072e4ea49b14893a8b0661cb"
format_stamp: "Formatted at 2025-12-15 21:02:10 on dist-test-slave-bjsf"
server_key: "48d498f305cc91e3eb26151d5e17cc3b"
server_key_iv: "76f9c8f52bb3b2fe70552f9ac8c4729f"
server_key_version: "encryptionkey@0"
I20251215 21:02:10.311426 32091 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765832506145157-29124-0/minicluster-data/master-0/wal/instance:
uuid: "b4479ad8072e4ea49b14893a8b0661cb"
format_stamp: "Formatted at 2025-12-15 21:02:10 on dist-test-slave-bjsf"
server_key: "48d498f305cc91e3eb26151d5e17cc3b"
server_key_iv: "76f9c8f52bb3b2fe70552f9ac8c4729f"
server_key_version: "encryptionkey@0"
I20251215 21:02:10.314488 32091 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20251215 21:02:10.316488 32106 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251215 21:02:10.317370 32091 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20251215 21:02:10.317469 32091 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765832506145157-29124-0/minicluster-data/master-0/data,/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765832506145157-29124-0/minicluster-data/master-0/wal
uuid: "b4479ad8072e4ea49b14893a8b0661cb"
format_stamp: "Formatted at 2025-12-15 21:02:10 on dist-test-slave-bjsf"
server_key: "48d498f305cc91e3eb26151d5e17cc3b"
server_key_iv: "76f9c8f52bb3b2fe70552f9ac8c4729f"
server_key_version: "encryptionkey@0"
I20251215 21:02:10.317548 32091 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765832506145157-29124-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765832506145157-29124-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765832506145157-29124-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251215 21:02:10.337453 32091 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251215 21:02:10.339881 32091 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251215 21:02:10.340058 32091 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251215 21:02:10.346464 32091 rpc_server.cc:307] RPC server started. Bound to: 127.28.113.62:37619
I20251215 21:02:10.346469 32158 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.28.113.62:37619 every 8 connection(s)
I20251215 21:02:10.347342 32091 server_base.cc:1179] Dumped server information to /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765832506145157-29124-0/minicluster-data/master-0/data/info.pb
I20251215 21:02:10.349866 32159 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 21:02:10.350162 29124 external_mini_cluster.cc:1428] Started /tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu as pid 32091
I20251215 21:02:10.350256 29124 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765832506145157-29124-0/minicluster-data/master-0/wal/instance
I20251215 21:02:10.350466 29124 external_mini_cluster.cc:1468] Setting key 62feb2d92fe6bbc9c10c3f37743de611
I20251215 21:02:10.356138 32159 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b4479ad8072e4ea49b14893a8b0661cb: Bootstrap starting.
Dec 15 21:02:10 dist-test-slave-bjsf krb5kdc[32075](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765832530, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.113.62@KRBTEST.COM
I20251215 21:02:10.358429 32159 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b4479ad8072e4ea49b14893a8b0661cb: Neither blocks nor log segments found. Creating new log.
I20251215 21:02:10.359227 32159 log.cc:826] T 00000000000000000000000000000000 P b4479ad8072e4ea49b14893a8b0661cb: Log is configured to *not* fsync() on all Append() calls
I20251215 21:02:10.361095 32159 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b4479ad8072e4ea49b14893a8b0661cb: No bootstrap required, opened a new log
I20251215 21:02:10.362591 32162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:02:10.351547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51422 (local address 127.28.113.62:37619)
1215 21:02:10.351995 (+ 448us) server_negotiation.cc:207] Beginning negotiation
1215 21:02:10.352007 (+ 12us) server_negotiation.cc:400] Waiting for connection header
1215 21:02:10.352052 (+ 45us) server_negotiation.cc:408] Connection header received
1215 21:02:10.352974 (+ 922us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1215 21:02:10.352999 (+ 25us) server_negotiation.cc:462] Received NEGOTIATE request from client
1215 21:02:10.353337 (+ 338us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1215 21:02:10.353661 (+ 324us) server_negotiation.cc:227] Negotiated authn=SASL
1215 21:02:10.354362 (+ 701us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:02:10.355484 (+ 1122us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1215 21:02:10.356093 (+ 609us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:02:10.356682 (+ 589us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:02:10.358430 (+ 1748us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1215 21:02:10.358448 (+ 18us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1215 21:02:10.358461 (+ 13us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1215 21:02:10.358500 (+ 39us) server_negotiation.cc:923] calling sasl_server_start()
1215 21:02:10.360263 (+ 1763us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1215 21:02:10.360620 (+ 357us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1215 21:02:10.360626 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1215 21:02:10.360630 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
1215 21:02:10.360714 (+ 84us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1215 21:02:10.360896 (+ 182us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1215 21:02:10.360899 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1215 21:02:10.360901 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1215 21:02:10.361277 (+ 376us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1215 21:02:10.361466 (+ 189us) server_negotiation.cc:1036] Waiting for connection context
1215 21:02:10.361649 (+ 183us) server_negotiation.cc:300] Negotiation successful
1215 21:02:10.361894 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":262,"thread_start_us":98,"threads_started":1}
I20251215 21:02:10.363756 32159 raft_consensus.cc:359] T 00000000000000000000000000000000 P b4479ad8072e4ea49b14893a8b0661cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4479ad8072e4ea49b14893a8b0661cb" member_type: VOTER last_known_addr { host: "127.28.113.62" port: 37619 } }
I20251215 21:02:10.363930 32159 raft_consensus.cc:385] T 00000000000000000000000000000000 P b4479ad8072e4ea49b14893a8b0661cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251215 21:02:10.363992 32159 raft_consensus.cc:740] T 00000000000000000000000000000000 P b4479ad8072e4ea49b14893a8b0661cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4479ad8072e4ea49b14893a8b0661cb, State: Initialized, Role: FOLLOWER
I20251215 21:02:10.364409 32159 consensus_queue.cc:260] T 00000000000000000000000000000000 P b4479ad8072e4ea49b14893a8b0661cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4479ad8072e4ea49b14893a8b0661cb" member_type: VOTER last_known_addr { host: "127.28.113.62" port: 37619 } }
I20251215 21:02:10.364558 32159 raft_consensus.cc:399] T 00000000000000000000000000000000 P b4479ad8072e4ea49b14893a8b0661cb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251215 21:02:10.364620 32159 raft_consensus.cc:493] T 00000000000000000000000000000000 P b4479ad8072e4ea49b14893a8b0661cb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251215 21:02:10.364705 32159 raft_consensus.cc:3060] T 00000000000000000000000000000000 P b4479ad8072e4ea49b14893a8b0661cb [term 0 FOLLOWER]: Advancing to term 1
I20251215 21:02:10.365409 32159 raft_consensus.cc:515] T 00000000000000000000000000000000 P b4479ad8072e4ea49b14893a8b0661cb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4479ad8072e4ea49b14893a8b0661cb" member_type: VOTER last_known_addr { host: "127.28.113.62" port: 37619 } }
I20251215 21:02:10.365715 32159 leader_election.cc:304] T 00000000000000000000000000000000 P b4479ad8072e4ea49b14893a8b0661cb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b4479ad8072e4ea49b14893a8b0661cb; no voters:
I20251215 21:02:10.365943 32159 leader_election.cc:290] T 00000000000000000000000000000000 P b4479ad8072e4ea49b14893a8b0661cb [CANDIDATE]: Term 1 election: Requested vote from peers
I20251215 21:02:10.366025 32164 raft_consensus.cc:2804] T 00000000000000000000000000000000 P b4479ad8072e4ea49b14893a8b0661cb [term 1 FOLLOWER]: Leader election won for term 1
I20251215 21:02:10.366218 32164 raft_consensus.cc:697] T 00000000000000000000000000000000 P b4479ad8072e4ea49b14893a8b0661cb [term 1 LEADER]: Becoming Leader. State: Replica: b4479ad8072e4ea49b14893a8b0661cb, State: Running, Role: LEADER
I20251215 21:02:10.366492 32164 consensus_queue.cc:237] T 00000000000000000000000000000000 P b4479ad8072e4ea49b14893a8b0661cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4479ad8072e4ea49b14893a8b0661cb" member_type: VOTER last_known_addr { host: "127.28.113.62" port: 37619 } }
I20251215 21:02:10.366852 32159 sys_catalog.cc:565] T 00000000000000000000000000000000 P b4479ad8072e4ea49b14893a8b0661cb [sys.catalog]: configured and running, proceeding with master startup.
I20251215 21:02:10.367861 32166 sys_catalog.cc:455] T 00000000000000000000000000000000 P b4479ad8072e4ea49b14893a8b0661cb [sys.catalog]: SysCatalogTable state changed. Reason: New leader b4479ad8072e4ea49b14893a8b0661cb. Latest consensus state: current_term: 1 leader_uuid: "b4479ad8072e4ea49b14893a8b0661cb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4479ad8072e4ea49b14893a8b0661cb" member_type: VOTER last_known_addr { host: "127.28.113.62" port: 37619 } } }
I20251215 21:02:10.367993 32166 sys_catalog.cc:458] T 00000000000000000000000000000000 P b4479ad8072e4ea49b14893a8b0661cb [sys.catalog]: This master's current role is: LEADER
I20251215 21:02:10.368273 32165 sys_catalog.cc:455] T 00000000000000000000000000000000 P b4479ad8072e4ea49b14893a8b0661cb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b4479ad8072e4ea49b14893a8b0661cb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4479ad8072e4ea49b14893a8b0661cb" member_type: VOTER last_known_addr { host: "127.28.113.62" port: 37619 } } }
I20251215 21:02:10.368376 32165 sys_catalog.cc:458] T 00000000000000000000000000000000 P b4479ad8072e4ea49b14893a8b0661cb [sys.catalog]: This master's current role is: LEADER
W20251215 21:02:10.371189 32179 catalog_manager.cc:1568] T 00000000000000000000000000000000 P b4479ad8072e4ea49b14893a8b0661cb: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20251215 21:02:10.371260 32179 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20251215 21:02:10.371325 32173 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251215 21:02:10.372140 32173 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251215 21:02:10.377003 32173 catalog_manager.cc:1357] Generated new cluster ID: f611f9ed817c4a6aaad1adabf71dd56d
I20251215 21:02:10.377058 32173 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251215 21:02:10.406309 32173 catalog_manager.cc:1380] Generated new certificate authority record
I20251215 21:02:10.407249 32173 catalog_manager.cc:1514] Loading token signing keys...
I20251215 21:02:10.416556 32173 catalog_manager.cc:6027] T 00000000000000000000000000000000 P b4479ad8072e4ea49b14893a8b0661cb: Generated new TSK 0
I20251215 21:02:10.417266 32173 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.113.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.113.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.113.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765832506145157-29124-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.28.113.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765832506145157-29124-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.28.113.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.28.113.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.28.113.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765832506145157-29124-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.28.113.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765832506145157-29124-0/krb5kdc/kudu.keytab.
I20251215 21:02:10.457916 29124 external_mini_cluster.cc:1366] Running /tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu
/tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765832506145157-29124-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765832506145157-29124-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765832506145157-29124-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765832506145157-29124-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.113.1:0
--local_ip_for_outbound_sockets=127.28.113.1
--webserver_interface=127.28.113.1
--webserver_port=0
--tserver_master_addrs=127.28.113.62:37619
--keytab_file=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765832506145157-29124-0/krb5kdc/kudu.keytab
--principal=kudu/127.28.113.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.28.113.20:41313
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765832506145157-29124-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765832506145157-29124-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765832506145157-29124-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251215 21:02:10.561965 32187 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251215 21:02:10.562184 32187 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251215 21:02:10.562243 32187 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251215 21:02:10.565515 32187 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251215 21:02:10.565586 32187 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251215 21:02:10.565665 32187 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.113.1
I20251215 21:02:10.569854 32187 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.113.20:41313
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765832506145157-29124-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765832506145157-29124-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765832506145157-29124-0/krb5kdc/kudu.keytab
--principal=kudu/127.28.113.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.113.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765832506145157-29124-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.113.1
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.28.113.62:37619
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32187
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.113.1
--log_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765832506145157-29124-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision dc741732bb6a40213e11a13869fad06a49a209a3
build type DEBUG
built by None at 15 Dec 2025 20:43:14 UTC on 24a791456cd2
build id 9530
I20251215 21:02:10.570845 32187 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251215 21:02:10.571647 32187 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251215 21:02:10.577971 32195 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251215 21:02:10.578011 32193 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251215 21:02:10.578047 32192 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251215 21:02:10.578294 32187 server_base.cc:1047] running on GCE node
I20251215 21:02:10.578667 32187 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251215 21:02:10.579202 32187 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251215 21:02:10.580348 32187 hybrid_clock.cc:648] HybridClock initialized: now 1765832530580322 us; error 42 us; skew 500 ppm
Dec 15 21:02:10 dist-test-slave-bjsf krb5kdc[32075](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765832530, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.113.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251215 21:02:10.582856 32187 init.cc:377] Logged in from keytab as kudu/127.28.113.1@KRBTEST.COM (short username kudu)
I20251215 21:02:10.583838 32187 webserver.cc:492] Webserver started at http://127.28.113.1:39243/ using document root <none> and password file <none>
I20251215 21:02:10.584388 32187 fs_manager.cc:362] Metadata directory not provided
I20251215 21:02:10.584437 32187 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251215 21:02:10.584586 32187 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251215 21:02:10.586197 32187 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765832506145157-29124-0/minicluster-data/ts-0/data/instance:
uuid: "c3cbc13d29b94cce95df116fec47f524"
format_stamp: "Formatted at 2025-12-15 21:02:10 on dist-test-slave-bjsf"
server_key: "a7af97bd8f1a323bcf6ff9cb568634b0"
server_key_iv: "4801e9cf8b67d646ec1da3d9730751fc"
server_key_version: "encryptionkey@0"
I20251215 21:02:10.586609 32187 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765832506145157-29124-0/minicluster-data/ts-0/wal/instance:
uuid: "c3cbc13d29b94cce95df116fec47f524"
format_stamp: "Formatted at 2025-12-15 21:02:10 on dist-test-slave-bjsf"
server_key: "a7af97bd8f1a323bcf6ff9cb568634b0"
server_key_iv: "4801e9cf8b67d646ec1da3d9730751fc"
server_key_version: "encryptionkey@0"
I20251215 21:02:10.590303 32187 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20251215 21:02:10.592686 32202 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251215 21:02:10.593744 32187 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20251215 21:02:10.593852 32187 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765832506145157-29124-0/minicluster-data/ts-0/data,/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765832506145157-29124-0/minicluster-data/ts-0/wal
uuid: "c3cbc13d29b94cce95df116fec47f524"
format_stamp: "Formatted at 2025-12-15 21:02:10 on dist-test-slave-bjsf"
server_key: "a7af97bd8f1a323bcf6ff9cb568634b0"
server_key_iv: "4801e9cf8b67d646ec1da3d9730751fc"
server_key_version: "encryptionkey@0"
I20251215 21:02:10.593948 32187 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765832506145157-29124-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765832506145157-29124-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765832506145157-29124-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251215 21:02:10.615167 32187 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251215 21:02:10.618281 32187 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251215 21:02:10.618467 32187 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251215 21:02:10.619500 32187 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251215 21:02:10.620468 32187 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251215 21:02:10.620518 32187 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251215 21:02:10.620579 32187 ts_tablet_manager.cc:616] Registered 0 tablets
I20251215 21:02:10.620601 32187 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251215 21:02:10.629673 32187 rpc_server.cc:307] RPC server started. Bound to: 127.28.113.1:39597
I20251215 21:02:10.629745 32315 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.28.113.1:39597 every 8 connection(s)
I20251215 21:02:10.630683 32187 server_base.cc:1179] Dumped server information to /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765832506145157-29124-0/minicluster-data/ts-0/data/info.pb
I20251215 21:02:10.633136 29124 external_mini_cluster.cc:1428] Started /tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu as pid 32187
I20251215 21:02:10.633241 29124 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765832506145157-29124-0/minicluster-data/ts-0/wal/instance
I20251215 21:02:10.633487 29124 external_mini_cluster.cc:1468] Setting key 8d85bd97a5301811e545d3e17cac1e9a
Dec 15 21:02:10 dist-test-slave-bjsf krb5kdc[32075](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765832530, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.113.1@KRBTEST.COM for kudu/127.28.113.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.28.113.2@KRBTEST.COM; defaulting to no policy
I20251215 21:02:10.643656 32162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:02:10.632342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.113.1:33007 (local address 127.28.113.62:37619)
1215 21:02:10.632470 (+ 128us) server_negotiation.cc:207] Beginning negotiation
1215 21:02:10.632474 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1215 21:02:10.633181 (+ 707us) server_negotiation.cc:408] Connection header received
1215 21:02:10.634152 (+ 971us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1215 21:02:10.634155 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1215 21:02:10.634211 (+ 56us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1215 21:02:10.634260 (+ 49us) server_negotiation.cc:227] Negotiated authn=SASL
1215 21:02:10.636232 (+ 1972us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:02:10.636908 (+ 676us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1215 21:02:10.637768 (+ 860us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:02:10.637949 (+ 181us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:02:10.640430 (+ 2481us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1215 21:02:10.640456 (+ 26us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1215 21:02:10.640459 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1215 21:02:10.640493 (+ 34us) server_negotiation.cc:923] calling sasl_server_start()
1215 21:02:10.642012 (+ 1519us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1215 21:02:10.642516 (+ 504us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1215 21:02:10.642519 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1215 21:02:10.642521 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1215 21:02:10.642565 (+ 44us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1215 21:02:10.642783 (+ 218us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1215 21:02:10.642785 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1215 21:02:10.642786 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1215 21:02:10.642923 (+ 137us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1215 21:02:10.642985 (+ 62us) server_negotiation.cc:1036] Waiting for connection context
1215 21:02:10.643414 (+ 429us) server_negotiation.cc:300] Negotiation successful
1215 21:02:10.643512 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20251215 21:02:10.644356 32318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:02:10.632587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.113.62:37619 (local address 127.28.113.1:33007)
1215 21:02:10.633013 (+ 426us) negotiation.cc:107] Waiting for socket to connect
1215 21:02:10.633059 (+ 46us) client_negotiation.cc:174] Beginning negotiation
1215 21:02:10.633944 (+ 885us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1215 21:02:10.634659 (+ 715us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1215 21:02:10.634668 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
1215 21:02:10.635124 (+ 456us) client_negotiation.cc:189] Negotiated authn=SASL
1215 21:02:10.636091 (+ 967us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:02:10.636105 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:02:10.637048 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1215 21:02:10.637052 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1215 21:02:10.637632 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:02:10.637641 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:02:10.637855 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:02:10.638475 (+ 620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1215 21:02:10.638506 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
1215 21:02:10.640281 (+ 1775us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1215 21:02:10.642136 (+ 1855us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1215 21:02:10.642140 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1215 21:02:10.642149 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
1215 21:02:10.642402 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1215 21:02:10.642641 (+ 239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1215 21:02:10.642643 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1215 21:02:10.642644 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1215 21:02:10.642724 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1215 21:02:10.642993 (+ 269us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1215 21:02:10.642997 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1215 21:02:10.643228 (+ 231us) client_negotiation.cc:769] Sending connection context
1215 21:02:10.643390 (+ 162us) client_negotiation.cc:240] Negotiation successful
1215 21:02:10.643615 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":88,"threads_started":1}
I20251215 21:02:10.645409 32316 heartbeater.cc:344] Connected to a master server at 127.28.113.62:37619
I20251215 21:02:10.645645 32316 heartbeater.cc:461] Registering TS with master...
I20251215 21:02:10.646148 32316 heartbeater.cc:507] Master 127.28.113.62:37619 requested a full tablet report, sending...
I20251215 21:02:10.647492 32123 ts_manager.cc:194] Registered new tserver with Master: c3cbc13d29b94cce95df116fec47f524 (127.28.113.1:39597)
I20251215 21:02:10.648624 32123 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.113.1@KRBTEST.COM'} at 127.28.113.1:33007
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.113.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.113.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765832506145157-29124-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.28.113.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765832506145157-29124-0/krb5kdc/kudu.keytab.
W20251215 21:02:10.968611 29124 mini_kdc.cc:268] Time spent creating service keytab for kudu/127.28.113.2: real 0.333s user 0.000s sys 0.003s
WARNING: no policy specified for HTTP/127.28.113.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.28.113.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.28.113.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765832506145157-29124-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.28.113.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765832506145157-29124-0/krb5kdc/kudu.keytab.
I20251215 21:02:10.987865 29124 external_mini_cluster.cc:1366] Running /tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu
/tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765832506145157-29124-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765832506145157-29124-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765832506145157-29124-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765832506145157-29124-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.113.2:0
--local_ip_for_outbound_sockets=127.28.113.2
--webserver_interface=127.28.113.2
--webserver_port=0
--tserver_master_addrs=127.28.113.62:37619
--keytab_file=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765832506145157-29124-0/krb5kdc/kudu.keytab
--principal=kudu/127.28.113.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.28.113.20:41313
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765832506145157-29124-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765832506145157-29124-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765832506145157-29124-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251215 21:02:11.091018 32323 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251215 21:02:11.091228 32323 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251215 21:02:11.091295 32323 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251215 21:02:11.094563 32323 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251215 21:02:11.094621 32323 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251215 21:02:11.094717 32323 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.113.2
I20251215 21:02:11.098732 32323 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.113.20:41313
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765832506145157-29124-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765832506145157-29124-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765832506145157-29124-0/krb5kdc/kudu.keytab
--principal=kudu/127.28.113.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.113.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765832506145157-29124-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.113.2
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.28.113.62:37619
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32323
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.113.2
--log_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765832506145157-29124-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision dc741732bb6a40213e11a13869fad06a49a209a3
build type DEBUG
built by None at 15 Dec 2025 20:43:14 UTC on 24a791456cd2
build id 9530
I20251215 21:02:11.099702 32323 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251215 21:02:11.100576 32323 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251215 21:02:11.106717 32329 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251215 21:02:11.106698 32328 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251215 21:02:11.106698 32331 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251215 21:02:11.106916 32323 server_base.cc:1047] running on GCE node
I20251215 21:02:11.107299 32323 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251215 21:02:11.107776 32323 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251215 21:02:11.108916 32323 hybrid_clock.cc:648] HybridClock initialized: now 1765832531108904 us; error 27 us; skew 500 ppm
Dec 15 21:02:11 dist-test-slave-bjsf krb5kdc[32075](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765832531, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.113.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251215 21:02:11.111333 32323 init.cc:377] Logged in from keytab as kudu/127.28.113.2@KRBTEST.COM (short username kudu)
I20251215 21:02:11.112355 32323 webserver.cc:492] Webserver started at http://127.28.113.2:33421/ using document root <none> and password file <none>
I20251215 21:02:11.112874 32323 fs_manager.cc:362] Metadata directory not provided
I20251215 21:02:11.112920 32323 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251215 21:02:11.113109 32323 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251215 21:02:11.114710 32323 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765832506145157-29124-0/minicluster-data/ts-1/data/instance:
uuid: "78d3fc23080440ddb526721f3add4e7f"
format_stamp: "Formatted at 2025-12-15 21:02:11 on dist-test-slave-bjsf"
server_key: "12064b9c9641c71c0c2f91928d488185"
server_key_iv: "512f878c558dfe4fb27520bb9cce06d5"
server_key_version: "encryptionkey@0"
I20251215 21:02:11.115139 32323 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765832506145157-29124-0/minicluster-data/ts-1/wal/instance:
uuid: "78d3fc23080440ddb526721f3add4e7f"
format_stamp: "Formatted at 2025-12-15 21:02:11 on dist-test-slave-bjsf"
server_key: "12064b9c9641c71c0c2f91928d488185"
server_key_iv: "512f878c558dfe4fb27520bb9cce06d5"
server_key_version: "encryptionkey@0"
I20251215 21:02:11.118335 32323 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20251215 21:02:11.120432 32338 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251215 21:02:11.121392 32323 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20251215 21:02:11.121486 32323 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765832506145157-29124-0/minicluster-data/ts-1/data,/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765832506145157-29124-0/minicluster-data/ts-1/wal
uuid: "78d3fc23080440ddb526721f3add4e7f"
format_stamp: "Formatted at 2025-12-15 21:02:11 on dist-test-slave-bjsf"
server_key: "12064b9c9641c71c0c2f91928d488185"
server_key_iv: "512f878c558dfe4fb27520bb9cce06d5"
server_key_version: "encryptionkey@0"
I20251215 21:02:11.121580 32323 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765832506145157-29124-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765832506145157-29124-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765832506145157-29124-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251215 21:02:11.134848 32323 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251215 21:02:11.137300 32323 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251215 21:02:11.137441 32323 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251215 21:02:11.137900 32323 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251215 21:02:11.138671 32323 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251215 21:02:11.138715 32323 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251215 21:02:11.138751 32323 ts_tablet_manager.cc:616] Registered 0 tablets
I20251215 21:02:11.138772 32323 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251215 21:02:11.146901 32323 rpc_server.cc:307] RPC server started. Bound to: 127.28.113.2:45169
I20251215 21:02:11.146935 32451 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.28.113.2:45169 every 8 connection(s)
I20251215 21:02:11.147749 32323 server_base.cc:1179] Dumped server information to /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765832506145157-29124-0/minicluster-data/ts-1/data/info.pb
I20251215 21:02:11.152127 29124 external_mini_cluster.cc:1428] Started /tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu as pid 32323
I20251215 21:02:11.152230 29124 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765832506145157-29124-0/minicluster-data/ts-1/wal/instance
I20251215 21:02:11.152437 29124 external_mini_cluster.cc:1468] Setting key 382c61b6bc6bed362605bbb8a762abaf
Dec 15 21:02:11 dist-test-slave-bjsf krb5kdc[32075](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765832531, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.113.2@KRBTEST.COM for kudu/127.28.113.62@KRBTEST.COM
I20251215 21:02:11.159520 32454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:02:11.149310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.113.2:44203 (local address 127.28.113.62:37619)
1215 21:02:11.149528 (+ 218us) server_negotiation.cc:207] Beginning negotiation
1215 21:02:11.149531 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1215 21:02:11.150050 (+ 519us) server_negotiation.cc:408] Connection header received
1215 21:02:11.150834 (+ 784us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1215 21:02:11.150838 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1215 21:02:11.150893 (+ 55us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1215 21:02:11.150971 (+ 78us) server_negotiation.cc:227] Negotiated authn=SASL
1215 21:02:11.152578 (+ 1607us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:02:11.153199 (+ 621us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1215 21:02:11.153821 (+ 622us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:02:11.153992 (+ 171us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:02:11.156158 (+ 2166us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1215 21:02:11.156174 (+ 16us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1215 21:02:11.156177 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1215 21:02:11.156200 (+ 23us) server_negotiation.cc:923] calling sasl_server_start()
1215 21:02:11.157780 (+ 1580us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1215 21:02:11.158206 (+ 426us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1215 21:02:11.158210 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1215 21:02:11.158212 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1215 21:02:11.158268 (+ 56us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1215 21:02:11.158499 (+ 231us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1215 21:02:11.158503 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1215 21:02:11.158505 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1215 21:02:11.158692 (+ 187us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1215 21:02:11.158788 (+ 96us) server_negotiation.cc:1036] Waiting for connection context
1215 21:02:11.159219 (+ 431us) server_negotiation.cc:300] Negotiation successful
1215 21:02:11.159338 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":69,"threads_started":1}
I20251215 21:02:11.160108 32455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:02:11.149544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.113.62:37619 (local address 127.28.113.2:44203)
1215 21:02:11.149932 (+ 388us) negotiation.cc:107] Waiting for socket to connect
1215 21:02:11.149965 (+ 33us) client_negotiation.cc:174] Beginning negotiation
1215 21:02:11.150659 (+ 694us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1215 21:02:11.151121 (+ 462us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1215 21:02:11.151128 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
1215 21:02:11.151493 (+ 365us) client_negotiation.cc:189] Negotiated authn=SASL
1215 21:02:11.152443 (+ 950us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:02:11.152457 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:02:11.153309 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1215 21:02:11.153312 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1215 21:02:11.153727 (+ 415us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:02:11.153735 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:02:11.153936 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:02:11.154482 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1215 21:02:11.154503 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
1215 21:02:11.156019 (+ 1516us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1215 21:02:11.157889 (+ 1870us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1215 21:02:11.157894 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1215 21:02:11.157903 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
1215 21:02:11.158117 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1215 21:02:11.158346 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1215 21:02:11.158348 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1215 21:02:11.158350 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1215 21:02:11.158426 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1215 21:02:11.158775 (+ 349us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1215 21:02:11.158780 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1215 21:02:11.158992 (+ 212us) client_negotiation.cc:769] Sending connection context
1215 21:02:11.159151 (+ 159us) client_negotiation.cc:240] Negotiation successful
1215 21:02:11.159354 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":88,"threads_started":1}
WARNING: no policy specified for kudu/127.28.113.3@KRBTEST.COM; defaulting to no policy
I20251215 21:02:11.161180 32452 heartbeater.cc:344] Connected to a master server at 127.28.113.62:37619
I20251215 21:02:11.161388 32452 heartbeater.cc:461] Registering TS with master...
I20251215 21:02:11.161902 32452 heartbeater.cc:507] Master 127.28.113.62:37619 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.113.3@KRBTEST.COM" created.
I20251215 21:02:11.162957 32123 ts_manager.cc:194] Registered new tserver with Master: 78d3fc23080440ddb526721f3add4e7f (127.28.113.2:45169)
I20251215 21:02:11.163635 32123 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.113.2@KRBTEST.COM'} at 127.28.113.2:44203
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.113.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765832506145157-29124-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.28.113.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765832506145157-29124-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.28.113.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.28.113.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.28.113.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765832506145157-29124-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.28.113.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765832506145157-29124-0/krb5kdc/kudu.keytab.
I20251215 21:02:11.194413 29124 external_mini_cluster.cc:1366] Running /tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu
/tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765832506145157-29124-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765832506145157-29124-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765832506145157-29124-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765832506145157-29124-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.113.3:0
--local_ip_for_outbound_sockets=127.28.113.3
--webserver_interface=127.28.113.3
--webserver_port=0
--tserver_master_addrs=127.28.113.62:37619
--keytab_file=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765832506145157-29124-0/krb5kdc/kudu.keytab
--principal=kudu/127.28.113.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.28.113.20:41313
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765832506145157-29124-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765832506145157-29124-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765832506145157-29124-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251215 21:02:11.293895 32460 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251215 21:02:11.294118 32460 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251215 21:02:11.294174 32460 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251215 21:02:11.297267 32460 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251215 21:02:11.297319 32460 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251215 21:02:11.297392 32460 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.113.3
I20251215 21:02:11.301369 32460 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.113.20:41313
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765832506145157-29124-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765832506145157-29124-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765832506145157-29124-0/krb5kdc/kudu.keytab
--principal=kudu/127.28.113.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.113.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765832506145157-29124-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.113.3
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.28.113.62:37619
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32460
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.113.3
--log_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765832506145157-29124-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision dc741732bb6a40213e11a13869fad06a49a209a3
build type DEBUG
built by None at 15 Dec 2025 20:43:14 UTC on 24a791456cd2
build id 9530
I20251215 21:02:11.302263 32460 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251215 21:02:11.303045 32460 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251215 21:02:11.309016 32468 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251215 21:02:11.309018 32466 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251215 21:02:11.309211 32460 server_base.cc:1047] running on GCE node
W20251215 21:02:11.309262 32465 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251215 21:02:11.309682 32460 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251215 21:02:11.310164 32460 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251215 21:02:11.311302 32460 hybrid_clock.cc:648] HybridClock initialized: now 1765832531311292 us; error 26 us; skew 500 ppm
Dec 15 21:02:11 dist-test-slave-bjsf krb5kdc[32075](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765832531, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.113.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251215 21:02:11.313782 32460 init.cc:377] Logged in from keytab as kudu/127.28.113.3@KRBTEST.COM (short username kudu)
I20251215 21:02:11.314721 32460 webserver.cc:492] Webserver started at http://127.28.113.3:35057/ using document root <none> and password file <none>
I20251215 21:02:11.315214 32460 fs_manager.cc:362] Metadata directory not provided
I20251215 21:02:11.315261 32460 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251215 21:02:11.315420 32460 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251215 21:02:11.317023 32460 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765832506145157-29124-0/minicluster-data/ts-2/data/instance:
uuid: "b3cfc98882704df692868cd89b2dd1ee"
format_stamp: "Formatted at 2025-12-15 21:02:11 on dist-test-slave-bjsf"
server_key: "14360df38871c895a1a334a2e894200b"
server_key_iv: "30aaefc73d5460379ff56fd203e849c1"
server_key_version: "encryptionkey@0"
I20251215 21:02:11.317453 32460 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765832506145157-29124-0/minicluster-data/ts-2/wal/instance:
uuid: "b3cfc98882704df692868cd89b2dd1ee"
format_stamp: "Formatted at 2025-12-15 21:02:11 on dist-test-slave-bjsf"
server_key: "14360df38871c895a1a334a2e894200b"
server_key_iv: "30aaefc73d5460379ff56fd203e849c1"
server_key_version: "encryptionkey@0"
I20251215 21:02:11.320554 32460 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20251215 21:02:11.322638 32475 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251215 21:02:11.323550 32460 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20251215 21:02:11.323644 32460 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765832506145157-29124-0/minicluster-data/ts-2/data,/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765832506145157-29124-0/minicluster-data/ts-2/wal
uuid: "b3cfc98882704df692868cd89b2dd1ee"
format_stamp: "Formatted at 2025-12-15 21:02:11 on dist-test-slave-bjsf"
server_key: "14360df38871c895a1a334a2e894200b"
server_key_iv: "30aaefc73d5460379ff56fd203e849c1"
server_key_version: "encryptionkey@0"
I20251215 21:02:11.323729 32460 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765832506145157-29124-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765832506145157-29124-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765832506145157-29124-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251215 21:02:11.338714 32460 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251215 21:02:11.341089 32460 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251215 21:02:11.341244 32460 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251215 21:02:11.341732 32460 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251215 21:02:11.342537 32460 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251215 21:02:11.342583 32460 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251215 21:02:11.342633 32460 ts_tablet_manager.cc:616] Registered 0 tablets
I20251215 21:02:11.342665 32460 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251215 21:02:11.351722 32460 rpc_server.cc:307] RPC server started. Bound to: 127.28.113.3:33827
I20251215 21:02:11.351794 32588 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.28.113.3:33827 every 8 connection(s)
I20251215 21:02:11.352622 32460 server_base.cc:1179] Dumped server information to /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765832506145157-29124-0/minicluster-data/ts-2/data/info.pb
I20251215 21:02:11.359087 29124 external_mini_cluster.cc:1428] Started /tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu as pid 32460
I20251215 21:02:11.359187 29124 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765832506145157-29124-0/minicluster-data/ts-2/wal/instance
I20251215 21:02:11.359400 29124 external_mini_cluster.cc:1468] Setting key 3e1c27d9a25be2bf8b891e88c2be0a21
Dec 15 21:02:11 dist-test-slave-bjsf krb5kdc[32075](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765832531, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.113.3@KRBTEST.COM for kudu/127.28.113.62@KRBTEST.COM
I20251215 21:02:11.364907 32454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:02:11.354328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.113.3:58599 (local address 127.28.113.62:37619)
1215 21:02:11.354480 (+ 152us) server_negotiation.cc:207] Beginning negotiation
1215 21:02:11.354485 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1215 21:02:11.355158 (+ 673us) server_negotiation.cc:408] Connection header received
1215 21:02:11.356168 (+ 1010us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1215 21:02:11.356171 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1215 21:02:11.356216 (+ 45us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1215 21:02:11.356284 (+ 68us) server_negotiation.cc:227] Negotiated authn=SASL
1215 21:02:11.358290 (+ 2006us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:02:11.358774 (+ 484us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1215 21:02:11.359431 (+ 657us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:02:11.359615 (+ 184us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:02:11.361977 (+ 2362us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1215 21:02:11.361995 (+ 18us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1215 21:02:11.361998 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1215 21:02:11.362024 (+ 26us) server_negotiation.cc:923] calling sasl_server_start()
1215 21:02:11.363238 (+ 1214us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1215 21:02:11.363673 (+ 435us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1215 21:02:11.363676 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1215 21:02:11.363677 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1215 21:02:11.363724 (+ 47us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1215 21:02:11.363945 (+ 221us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1215 21:02:11.363947 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1215 21:02:11.363948 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1215 21:02:11.364145 (+ 197us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1215 21:02:11.364219 (+ 74us) server_negotiation.cc:1036] Waiting for connection context
1215 21:02:11.364681 (+ 462us) server_negotiation.cc:300] Negotiation successful
1215 21:02:11.364783 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20251215 21:02:11.365557 32591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:02:11.354552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.113.62:37619 (local address 127.28.113.3:58599)
1215 21:02:11.354998 (+ 446us) negotiation.cc:107] Waiting for socket to connect
1215 21:02:11.355042 (+ 44us) client_negotiation.cc:174] Beginning negotiation
1215 21:02:11.355945 (+ 903us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1215 21:02:11.356450 (+ 505us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1215 21:02:11.356461 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
1215 21:02:11.356971 (+ 510us) client_negotiation.cc:189] Negotiated authn=SASL
1215 21:02:11.358153 (+ 1182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:02:11.358170 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:02:11.358890 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1215 21:02:11.358893 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1215 21:02:11.359308 (+ 415us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:02:11.359316 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:02:11.359501 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:02:11.360162 (+ 661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1215 21:02:11.360189 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
1215 21:02:11.361840 (+ 1651us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1215 21:02:11.363332 (+ 1492us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1215 21:02:11.363336 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1215 21:02:11.363346 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
1215 21:02:11.363584 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1215 21:02:11.363793 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1215 21:02:11.363795 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1215 21:02:11.363796 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1215 21:02:11.363878 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1215 21:02:11.364232 (+ 354us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1215 21:02:11.364237 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1215 21:02:11.364465 (+ 228us) client_negotiation.cc:769] Sending connection context
1215 21:02:11.364638 (+ 173us) client_negotiation.cc:240] Negotiation successful
1215 21:02:11.364841 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":97,"threads_started":1}
I20251215 21:02:11.366513 32589 heartbeater.cc:344] Connected to a master server at 127.28.113.62:37619
I20251215 21:02:11.366734 32589 heartbeater.cc:461] Registering TS with master...
I20251215 21:02:11.367218 32589 heartbeater.cc:507] Master 127.28.113.62:37619 requested a full tablet report, sending...
I20251215 21:02:11.368163 32123 ts_manager.cc:194] Registered new tserver with Master: b3cfc98882704df692868cd89b2dd1ee (127.28.113.3:33827)
I20251215 21:02:11.368726 32123 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.113.3@KRBTEST.COM'} at 127.28.113.3:58599
I20251215 21:02:11.372941 29124 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
W20251215 21:02:11.496222 32454 negotiation.cc:337] Failed RPC negotiation. Trace:
1215 21:02:11.493098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51430 (local address 127.28.113.62:37619)
1215 21:02:11.493238 (+ 140us) server_negotiation.cc:207] Beginning negotiation
1215 21:02:11.493241 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1215 21:02:11.494086 (+ 845us) server_negotiation.cc:408] Connection header received
1215 21:02:11.494893 (+ 807us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1215 21:02:11.494896 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1215 21:02:11.494944 (+ 48us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1215 21:02:11.495017 (+ 73us) server_negotiation.cc:227] Negotiated authn=SASL
1215 21:02:11.496095 (+ 1078us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51430: BlockingRecv error: recv got EOF from 127.0.0.1:51430 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
W20251215 21:02:11.497661 32454 negotiation.cc:337] Failed RPC negotiation. Trace:
1215 21:02:11.496797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51438 (local address 127.28.113.62:37619)
1215 21:02:11.496935 (+ 138us) server_negotiation.cc:207] Beginning negotiation
1215 21:02:11.496938 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1215 21:02:11.496969 (+ 31us) server_negotiation.cc:408] Connection header received
1215 21:02:11.497056 (+ 87us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1215 21:02:11.497059 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1215 21:02:11.497105 (+ 46us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1215 21:02:11.497198 (+ 93us) server_negotiation.cc:227] Negotiated authn=SASL
1215 21:02:11.497575 (+ 377us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51438: BlockingRecv error: recv got EOF from 127.0.0.1:51438 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
W20251215 21:02:11.498718 32454 negotiation.cc:337] Failed RPC negotiation. Trace:
1215 21:02:11.497877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51452 (local address 127.28.113.62:37619)
1215 21:02:11.497983 (+ 106us) server_negotiation.cc:207] Beginning negotiation
1215 21:02:11.497986 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1215 21:02:11.497998 (+ 12us) server_negotiation.cc:408] Connection header received
1215 21:02:11.498080 (+ 82us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1215 21:02:11.498084 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1215 21:02:11.498137 (+ 53us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1215 21:02:11.498197 (+ 60us) server_negotiation.cc:227] Negotiated authn=SASL
1215 21:02:11.498523 (+ 326us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51452: BlockingRecv error: recv got EOF from 127.0.0.1:51452 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
W20251215 21:02:11.502010 32454 negotiation.cc:337] Failed RPC negotiation. Trace:
1215 21:02:11.501275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51468 (local address 127.28.113.62:37619)
1215 21:02:11.501386 (+ 111us) server_negotiation.cc:207] Beginning negotiation
1215 21:02:11.501389 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1215 21:02:11.501452 (+ 63us) server_negotiation.cc:408] Connection header received
1215 21:02:11.501545 (+ 93us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1215 21:02:11.501547 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1215 21:02:11.501580 (+ 33us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1215 21:02:11.501631 (+ 51us) server_negotiation.cc:227] Negotiated authn=SASL
1215 21:02:11.501928 (+ 297us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51468: BlockingRecv error: recv got EOF from 127.0.0.1:51468 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
I20251215 21:02:11.559402 29124 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu with pid 32187
I20251215 21:02:11.564358 29124 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu with pid 32323
I20251215 21:02:11.569083 29124 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu with pid 32460
I20251215 21:02:11.573320 29124 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu with pid 32091
2025-12-15T21:02:11Z chronyd exiting
[ OK ] SecurityITest.TestRebalanceReportUnauthorized (3325 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1765832506145157-29124-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Dec 15 21:02:11 dist-test-slave-bjsf krb5kdc[32612](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Dec 15 21:02:11 dist-test-slave-bjsf krb5kdc[32612](info): set up 2 sockets
Dec 15 21:02:11 dist-test-slave-bjsf krb5kdc[32612](info): commencing operation
krb5kdc: starting...
W20251215 21:02:13.408039 29124 mini_kdc.cc:121] Time spent starting KDC: real 1.815s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1765832506145157-29124-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1765832506145157-29124-0/krb5kdc/test-user.keytab.
Dec 15 21:02:13 dist-test-slave-bjsf krb5kdc[32612](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765832533, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-12-15T21:02:13Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-12-15T21:02:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.113.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.113.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.113.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1765832506145157-29124-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.28.113.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1765832506145157-29124-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.28.113.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.28.113.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.28.113.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1765832506145157-29124-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.28.113.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1765832506145157-29124-0/krb5kdc/kudu.keytab.
I20251215 21:02:13.530599 29124 external_mini_cluster.cc:1366] Running /tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu
/tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1765832506145157-29124-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1765832506145157-29124-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1765832506145157-29124-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1765832506145157-29124-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.113.62:46813
--webserver_interface=127.28.113.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1765832506145157-29124-0/krb5kdc/kudu.keytab
--principal=kudu/127.28.113.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.28.113.20:46277
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.113.62:46813
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1765832506145157-29124-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1765832506145157-29124-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1765832506145157-29124-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251215 21:02:13.630800 32628 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251215 21:02:13.631017 32628 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251215 21:02:13.631088 32628 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251215 21:02:13.634212 32628 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251215 21:02:13.634274 32628 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251215 21:02:13.634303 32628 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251215 21:02:13.634328 32628 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251215 21:02:13.634359 32628 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251215 21:02:13.638411 32628 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.113.20:46277
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1765832506145157-29124-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1765832506145157-29124-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.113.62:46813
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1765832506145157-29124-0/krb5kdc/kudu.keytab
--principal=kudu/127.28.113.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.113.62:46813
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1765832506145157-29124-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.113.62
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32628
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1765832506145157-29124-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision dc741732bb6a40213e11a13869fad06a49a209a3
build type DEBUG
built by None at 15 Dec 2025 20:43:14 UTC on 24a791456cd2
build id 9530
I20251215 21:02:13.639401 32628 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251215 21:02:13.640214 32628 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251215 21:02:13.645308 32633 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251215 21:02:13.645375 32634 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251215 21:02:13.645311 32636 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251215 21:02:13.645515 32628 server_base.cc:1047] running on GCE node
I20251215 21:02:13.646049 32628 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251215 21:02:13.646869 32628 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251215 21:02:13.648010 32628 hybrid_clock.cc:648] HybridClock initialized: now 1765832533647991 us; error 32 us; skew 500 ppm
Dec 15 21:02:13 dist-test-slave-bjsf krb5kdc[32612](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765832533, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.113.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251215 21:02:13.650267 32628 init.cc:377] Logged in from keytab as kudu/127.28.113.62@KRBTEST.COM (short username kudu)
I20251215 21:02:13.651250 32628 webserver.cc:492] Webserver started at http://127.28.113.62:40679/ using document root <none> and password file <none>
I20251215 21:02:13.651746 32628 fs_manager.cc:362] Metadata directory not provided
I20251215 21:02:13.651795 32628 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251215 21:02:13.652017 32628 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251215 21:02:13.653595 32628 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1765832506145157-29124-0/minicluster-data/master-0/data/instance:
uuid: "2b610cf9f3ed4d41abd2650300339691"
format_stamp: "Formatted at 2025-12-15 21:02:13 on dist-test-slave-bjsf"
server_key: "bdac993dcde00a86dec32b0e70ef9807"
server_key_iv: "efa7646348d3cb2e9b4b1d603253c6da"
server_key_version: "encryptionkey@0"
I20251215 21:02:13.654037 32628 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1765832506145157-29124-0/minicluster-data/master-0/wal/instance:
uuid: "2b610cf9f3ed4d41abd2650300339691"
format_stamp: "Formatted at 2025-12-15 21:02:13 on dist-test-slave-bjsf"
server_key: "bdac993dcde00a86dec32b0e70ef9807"
server_key_iv: "efa7646348d3cb2e9b4b1d603253c6da"
server_key_version: "encryptionkey@0"
I20251215 21:02:13.657131 32628 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.002s
I20251215 21:02:13.659019 32643 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251215 21:02:13.659921 32628 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20251215 21:02:13.660045 32628 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1765832506145157-29124-0/minicluster-data/master-0/data,/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1765832506145157-29124-0/minicluster-data/master-0/wal
uuid: "2b610cf9f3ed4d41abd2650300339691"
format_stamp: "Formatted at 2025-12-15 21:02:13 on dist-test-slave-bjsf"
server_key: "bdac993dcde00a86dec32b0e70ef9807"
server_key_iv: "efa7646348d3cb2e9b4b1d603253c6da"
server_key_version: "encryptionkey@0"
I20251215 21:02:13.660121 32628 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1765832506145157-29124-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1765832506145157-29124-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1765832506145157-29124-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251215 21:02:13.692404 32628 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251215 21:02:13.694921 32628 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251215 21:02:13.695104 32628 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251215 21:02:13.701853 32695 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.28.113.62:46813 every 8 connection(s)
I20251215 21:02:13.701861 32628 rpc_server.cc:307] RPC server started. Bound to: 127.28.113.62:46813
I20251215 21:02:13.702828 32628 server_base.cc:1179] Dumped server information to /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1765832506145157-29124-0/minicluster-data/master-0/data/info.pb
I20251215 21:02:13.705098 29124 external_mini_cluster.cc:1428] Started /tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu as pid 32628
I20251215 21:02:13.705207 29124 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1765832506145157-29124-0/minicluster-data/master-0/wal/instance
I20251215 21:02:13.705449 29124 external_mini_cluster.cc:1468] Setting key 9786b317e7ca20acf4e901245ac5b22d
I20251215 21:02:13.705525 32696 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 21:02:13.710816 32696 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2b610cf9f3ed4d41abd2650300339691: Bootstrap starting.
Dec 15 21:02:13 dist-test-slave-bjsf krb5kdc[32612](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765832533, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.113.62@KRBTEST.COM
I20251215 21:02:13.713122 32696 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2b610cf9f3ed4d41abd2650300339691: Neither blocks nor log segments found. Creating new log.
I20251215 21:02:13.713873 32696 log.cc:826] T 00000000000000000000000000000000 P 2b610cf9f3ed4d41abd2650300339691: Log is configured to *not* fsync() on all Append() calls
I20251215 21:02:13.715720 32696 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2b610cf9f3ed4d41abd2650300339691: No bootstrap required, opened a new log
I20251215 21:02:13.717154 32699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:02:13.706570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51088 (local address 127.28.113.62:46813)
1215 21:02:13.706979 (+ 409us) server_negotiation.cc:207] Beginning negotiation
1215 21:02:13.706988 (+ 9us) server_negotiation.cc:400] Waiting for connection header
1215 21:02:13.707021 (+ 33us) server_negotiation.cc:408] Connection header received
1215 21:02:13.707797 (+ 776us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1215 21:02:13.707815 (+ 18us) server_negotiation.cc:462] Received NEGOTIATE request from client
1215 21:02:13.708128 (+ 313us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1215 21:02:13.708411 (+ 283us) server_negotiation.cc:227] Negotiated authn=SASL
1215 21:02:13.709266 (+ 855us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:02:13.710045 (+ 779us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1215 21:02:13.710669 (+ 624us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:02:13.710942 (+ 273us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:02:13.713119 (+ 2177us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1215 21:02:13.713136 (+ 17us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1215 21:02:13.713146 (+ 10us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1215 21:02:13.713183 (+ 37us) server_negotiation.cc:923] calling sasl_server_start()
1215 21:02:13.714945 (+ 1762us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1215 21:02:13.715357 (+ 412us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1215 21:02:13.715361 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1215 21:02:13.715365 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
1215 21:02:13.715425 (+ 60us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1215 21:02:13.715628 (+ 203us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1215 21:02:13.715630 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1215 21:02:13.715632 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1215 21:02:13.715950 (+ 318us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1215 21:02:13.716119 (+ 169us) server_negotiation.cc:1036] Waiting for connection context
1215 21:02:13.716364 (+ 245us) server_negotiation.cc:300] Negotiation successful
1215 21:02:13.716596 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":281,"thread_start_us":124,"threads_started":1}
I20251215 21:02:13.718218 32696 raft_consensus.cc:359] T 00000000000000000000000000000000 P 2b610cf9f3ed4d41abd2650300339691 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b610cf9f3ed4d41abd2650300339691" member_type: VOTER last_known_addr { host: "127.28.113.62" port: 46813 } }
I20251215 21:02:13.718386 32696 raft_consensus.cc:385] T 00000000000000000000000000000000 P 2b610cf9f3ed4d41abd2650300339691 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251215 21:02:13.718443 32696 raft_consensus.cc:740] T 00000000000000000000000000000000 P 2b610cf9f3ed4d41abd2650300339691 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b610cf9f3ed4d41abd2650300339691, State: Initialized, Role: FOLLOWER
I20251215 21:02:13.718863 32696 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2b610cf9f3ed4d41abd2650300339691 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b610cf9f3ed4d41abd2650300339691" member_type: VOTER last_known_addr { host: "127.28.113.62" port: 46813 } }
I20251215 21:02:13.719002 32696 raft_consensus.cc:399] T 00000000000000000000000000000000 P 2b610cf9f3ed4d41abd2650300339691 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251215 21:02:13.719069 32696 raft_consensus.cc:493] T 00000000000000000000000000000000 P 2b610cf9f3ed4d41abd2650300339691 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251215 21:02:13.719157 32696 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 2b610cf9f3ed4d41abd2650300339691 [term 0 FOLLOWER]: Advancing to term 1
I20251215 21:02:13.719512 29124 external_mini_cluster.cc:949] 0 TS(s) registered with all masters
I20251215 21:02:13.720011 32696 raft_consensus.cc:515] T 00000000000000000000000000000000 P 2b610cf9f3ed4d41abd2650300339691 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b610cf9f3ed4d41abd2650300339691" member_type: VOTER last_known_addr { host: "127.28.113.62" port: 46813 } }
I20251215 21:02:13.720302 32696 leader_election.cc:304] T 00000000000000000000000000000000 P 2b610cf9f3ed4d41abd2650300339691 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2b610cf9f3ed4d41abd2650300339691; no voters:
I20251215 21:02:13.720531 32696 leader_election.cc:290] T 00000000000000000000000000000000 P 2b610cf9f3ed4d41abd2650300339691 [CANDIDATE]: Term 1 election: Requested vote from peers
I20251215 21:02:13.720644 32701 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 2b610cf9f3ed4d41abd2650300339691 [term 1 FOLLOWER]: Leader election won for term 1
I20251215 21:02:13.720849 32701 raft_consensus.cc:697] T 00000000000000000000000000000000 P 2b610cf9f3ed4d41abd2650300339691 [term 1 LEADER]: Becoming Leader. State: Replica: 2b610cf9f3ed4d41abd2650300339691, State: Running, Role: LEADER
I20251215 21:02:13.721184 32701 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2b610cf9f3ed4d41abd2650300339691 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b610cf9f3ed4d41abd2650300339691" member_type: VOTER last_known_addr { host: "127.28.113.62" port: 46813 } }
I20251215 21:02:13.721616 32696 sys_catalog.cc:565] T 00000000000000000000000000000000 P 2b610cf9f3ed4d41abd2650300339691 [sys.catalog]: configured and running, proceeding with master startup.
I20251215 21:02:13.724519 32702 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b610cf9f3ed4d41abd2650300339691 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2b610cf9f3ed4d41abd2650300339691" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b610cf9f3ed4d41abd2650300339691" member_type: VOTER last_known_addr { host: "127.28.113.62" port: 46813 } } }
I20251215 21:02:13.724622 32702 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b610cf9f3ed4d41abd2650300339691 [sys.catalog]: This master's current role is: LEADER
I20251215 21:02:13.724871 32703 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b610cf9f3ed4d41abd2650300339691 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2b610cf9f3ed4d41abd2650300339691. Latest consensus state: current_term: 1 leader_uuid: "2b610cf9f3ed4d41abd2650300339691" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b610cf9f3ed4d41abd2650300339691" member_type: VOTER last_known_addr { host: "127.28.113.62" port: 46813 } } }
I20251215 21:02:13.724987 32703 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b610cf9f3ed4d41abd2650300339691 [sys.catalog]: This master's current role is: LEADER
W20251215 21:02:13.726882 32717 catalog_manager.cc:1568] T 00000000000000000000000000000000 P 2b610cf9f3ed4d41abd2650300339691: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20251215 21:02:13.726955 32717 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20251215 21:02:13.727028 32718 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251215 21:02:13.727859 32718 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251215 21:02:13.733048 32718 catalog_manager.cc:1357] Generated new cluster ID: 0e263f0e477b467ba8e0c53be5b6587c
I20251215 21:02:13.733105 32718 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251215 21:02:13.735759 32699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:02:13.731301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51092 (local address 127.28.113.62:46813)
1215 21:02:13.731420 (+ 119us) server_negotiation.cc:207] Beginning negotiation
1215 21:02:13.731424 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1215 21:02:13.731539 (+ 115us) server_negotiation.cc:408] Connection header received
1215 21:02:13.731684 (+ 145us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1215 21:02:13.731688 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1215 21:02:13.731744 (+ 56us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1215 21:02:13.731831 (+ 87us) server_negotiation.cc:227] Negotiated authn=SASL
1215 21:02:13.732614 (+ 783us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:02:13.733359 (+ 745us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1215 21:02:13.734172 (+ 813us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:02:13.734373 (+ 201us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:02:13.734809 (+ 436us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1215 21:02:13.734826 (+ 17us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1215 21:02:13.734830 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1215 21:02:13.735333 (+ 503us) server_negotiation.cc:923] calling sasl_server_start()
1215 21:02:13.735366 (+ 33us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1215 21:02:13.735386 (+ 20us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1215 21:02:13.735482 (+ 96us) server_negotiation.cc:1036] Waiting for connection context
1215 21:02:13.735558 (+ 76us) server_negotiation.cc:300] Negotiation successful
1215 21:02:13.735616 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20251215 21:02:13.758126 32718 catalog_manager.cc:1380] Generated new certificate authority record
I20251215 21:02:13.758955 32718 catalog_manager.cc:1514] Loading token signing keys...
I20251215 21:02:13.766155 32718 catalog_manager.cc:6027] T 00000000000000000000000000000000 P 2b610cf9f3ed4d41abd2650300339691: Generated new TSK 0
I20251215 21:02:13.766750 32718 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251215 21:02:13.776955 29124 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu with pid 32628
2025-12-15T21:02:13Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (2202 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765832506145157-29124-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Dec 15 21:02:13 dist-test-slave-bjsf krb5kdc[32729](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Dec 15 21:02:13 dist-test-slave-bjsf krb5kdc[32729](info): set up 2 sockets
Dec 15 21:02:13 dist-test-slave-bjsf krb5kdc[32729](info): commencing operation
krb5kdc: starting...
W20251215 21:02:15.588011 29124 mini_kdc.cc:121] Time spent starting KDC: real 1.793s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765832506145157-29124-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765832506145157-29124-0/krb5kdc/test-user.keytab.
Dec 15 21:02:15 dist-test-slave-bjsf krb5kdc[32729](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765832535, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-12-15T21:02:15Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-12-15T21:02:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.113.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.113.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.113.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765832506145157-29124-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.28.113.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765832506145157-29124-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.28.113.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.28.113.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.28.113.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765832506145157-29124-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.28.113.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765832506145157-29124-0/krb5kdc/kudu.keytab.
I20251215 21:02:15.702687 29124 external_mini_cluster.cc:1366] Running /tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu
/tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765832506145157-29124-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765832506145157-29124-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765832506145157-29124-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765832506145157-29124-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.113.62:44037
--webserver_interface=127.28.113.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765832506145157-29124-0/krb5kdc/kudu.keytab
--principal=kudu/127.28.113.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.28.113.20:43859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.113.62:44037
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765832506145157-29124-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765832506145157-29124-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765832506145157-29124-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251215 21:02:15.798233 32745 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251215 21:02:15.798424 32745 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251215 21:02:15.798492 32745 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251215 21:02:15.801587 32745 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251215 21:02:15.801642 32745 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251215 21:02:15.801671 32745 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251215 21:02:15.801695 32745 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251215 21:02:15.801719 32745 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251215 21:02:15.805763 32745 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.113.20:43859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765832506145157-29124-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765832506145157-29124-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.113.62:44037
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765832506145157-29124-0/krb5kdc/kudu.keytab
--principal=kudu/127.28.113.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.113.62:44037
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765832506145157-29124-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.113.62
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32745
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765832506145157-29124-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision dc741732bb6a40213e11a13869fad06a49a209a3
build type DEBUG
built by None at 15 Dec 2025 20:43:14 UTC on 24a791456cd2
build id 9530
I20251215 21:02:15.806661 32745 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251215 21:02:15.807413 32745 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251215 21:02:15.812314 32753 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251215 21:02:15.812314 32751 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251215 21:02:15.812345 32750 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251215 21:02:15.812469 32745 server_base.cc:1047] running on GCE node
I20251215 21:02:15.812937 32745 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251215 21:02:15.813740 32745 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251215 21:02:15.814881 32745 hybrid_clock.cc:648] HybridClock initialized: now 1765832535814864 us; error 32 us; skew 500 ppm
Dec 15 21:02:15 dist-test-slave-bjsf krb5kdc[32729](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765832535, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.113.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251215 21:02:15.817262 32745 init.cc:377] Logged in from keytab as kudu/127.28.113.62@KRBTEST.COM (short username kudu)
I20251215 21:02:15.818210 32745 webserver.cc:492] Webserver started at http://127.28.113.62:37715/ using document root <none> and password file <none>
I20251215 21:02:15.818697 32745 fs_manager.cc:362] Metadata directory not provided
I20251215 21:02:15.818745 32745 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251215 21:02:15.818913 32745 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251215 21:02:15.820503 32745 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765832506145157-29124-0/minicluster-data/master-0/data/instance:
uuid: "adc2eba2a9e64b4ca9c6bfe15414df03"
format_stamp: "Formatted at 2025-12-15 21:02:15 on dist-test-slave-bjsf"
server_key: "4b3bf9cb5df841e2744cd9a11c566c4a"
server_key_iv: "aa8cc4b38a3780d23a121f6e82d8b05b"
server_key_version: "encryptionkey@0"
I20251215 21:02:15.820945 32745 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765832506145157-29124-0/minicluster-data/master-0/wal/instance:
uuid: "adc2eba2a9e64b4ca9c6bfe15414df03"
format_stamp: "Formatted at 2025-12-15 21:02:15 on dist-test-slave-bjsf"
server_key: "4b3bf9cb5df841e2744cd9a11c566c4a"
server_key_iv: "aa8cc4b38a3780d23a121f6e82d8b05b"
server_key_version: "encryptionkey@0"
I20251215 21:02:15.823928 32745 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20251215 21:02:15.825834 32760 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251215 21:02:15.826714 32745 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20251215 21:02:15.826813 32745 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765832506145157-29124-0/minicluster-data/master-0/data,/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765832506145157-29124-0/minicluster-data/master-0/wal
uuid: "adc2eba2a9e64b4ca9c6bfe15414df03"
format_stamp: "Formatted at 2025-12-15 21:02:15 on dist-test-slave-bjsf"
server_key: "4b3bf9cb5df841e2744cd9a11c566c4a"
server_key_iv: "aa8cc4b38a3780d23a121f6e82d8b05b"
server_key_version: "encryptionkey@0"
I20251215 21:02:15.826895 32745 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765832506145157-29124-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765832506145157-29124-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765832506145157-29124-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251215 21:02:15.840512 32745 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251215 21:02:15.842792 32745 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251215 21:02:15.842952 32745 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251215 21:02:15.849284 344 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.28.113.62:44037 every 8 connection(s)
I20251215 21:02:15.849288 32745 rpc_server.cc:307] RPC server started. Bound to: 127.28.113.62:44037
I20251215 21:02:15.850232 32745 server_base.cc:1179] Dumped server information to /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765832506145157-29124-0/minicluster-data/master-0/data/info.pb
I20251215 21:02:15.852695 345 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 21:02:15.856593 29124 external_mini_cluster.cc:1428] Started /tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu as pid 32745
I20251215 21:02:15.856722 29124 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765832506145157-29124-0/minicluster-data/master-0/wal/instance
I20251215 21:02:15.856930 29124 external_mini_cluster.cc:1468] Setting key 6111d3e177d26bc85e66f38b367c4660
I20251215 21:02:15.857817 345 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P adc2eba2a9e64b4ca9c6bfe15414df03: Bootstrap starting.
I20251215 21:02:15.860127 345 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P adc2eba2a9e64b4ca9c6bfe15414df03: Neither blocks nor log segments found. Creating new log.
I20251215 21:02:15.860857 345 log.cc:826] T 00000000000000000000000000000000 P adc2eba2a9e64b4ca9c6bfe15414df03: Log is configured to *not* fsync() on all Append() calls
I20251215 21:02:15.862548 345 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P adc2eba2a9e64b4ca9c6bfe15414df03: No bootstrap required, opened a new log
Dec 15 21:02:15 dist-test-slave-bjsf krb5kdc[32729](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765832535, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.113.62@KRBTEST.COM
I20251215 21:02:15.865018 345 raft_consensus.cc:359] T 00000000000000000000000000000000 P adc2eba2a9e64b4ca9c6bfe15414df03 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "adc2eba2a9e64b4ca9c6bfe15414df03" member_type: VOTER last_known_addr { host: "127.28.113.62" port: 44037 } }
I20251215 21:02:15.865219 345 raft_consensus.cc:385] T 00000000000000000000000000000000 P adc2eba2a9e64b4ca9c6bfe15414df03 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251215 21:02:15.865279 345 raft_consensus.cc:740] T 00000000000000000000000000000000 P adc2eba2a9e64b4ca9c6bfe15414df03 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: adc2eba2a9e64b4ca9c6bfe15414df03, State: Initialized, Role: FOLLOWER
I20251215 21:02:15.865725 345 consensus_queue.cc:260] T 00000000000000000000000000000000 P adc2eba2a9e64b4ca9c6bfe15414df03 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "adc2eba2a9e64b4ca9c6bfe15414df03" member_type: VOTER last_known_addr { host: "127.28.113.62" port: 44037 } }
I20251215 21:02:15.865866 345 raft_consensus.cc:399] T 00000000000000000000000000000000 P adc2eba2a9e64b4ca9c6bfe15414df03 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251215 21:02:15.865929 345 raft_consensus.cc:493] T 00000000000000000000000000000000 P adc2eba2a9e64b4ca9c6bfe15414df03 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251215 21:02:15.866019 345 raft_consensus.cc:3060] T 00000000000000000000000000000000 P adc2eba2a9e64b4ca9c6bfe15414df03 [term 0 FOLLOWER]: Advancing to term 1
I20251215 21:02:15.866869 345 raft_consensus.cc:515] T 00000000000000000000000000000000 P adc2eba2a9e64b4ca9c6bfe15414df03 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "adc2eba2a9e64b4ca9c6bfe15414df03" member_type: VOTER last_known_addr { host: "127.28.113.62" port: 44037 } }
I20251215 21:02:15.867208 345 leader_election.cc:304] T 00000000000000000000000000000000 P adc2eba2a9e64b4ca9c6bfe15414df03 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: adc2eba2a9e64b4ca9c6bfe15414df03; no voters:
I20251215 21:02:15.867487 345 leader_election.cc:290] T 00000000000000000000000000000000 P adc2eba2a9e64b4ca9c6bfe15414df03 [CANDIDATE]: Term 1 election: Requested vote from peers
I20251215 21:02:15.867795 350 raft_consensus.cc:2804] T 00000000000000000000000000000000 P adc2eba2a9e64b4ca9c6bfe15414df03 [term 1 FOLLOWER]: Leader election won for term 1
I20251215 21:02:15.868036 350 raft_consensus.cc:697] T 00000000000000000000000000000000 P adc2eba2a9e64b4ca9c6bfe15414df03 [term 1 LEADER]: Becoming Leader. State: Replica: adc2eba2a9e64b4ca9c6bfe15414df03, State: Running, Role: LEADER
I20251215 21:02:15.868331 350 consensus_queue.cc:237] T 00000000000000000000000000000000 P adc2eba2a9e64b4ca9c6bfe15414df03 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "adc2eba2a9e64b4ca9c6bfe15414df03" member_type: VOTER last_known_addr { host: "127.28.113.62" port: 44037 } }
I20251215 21:02:15.868654 345 sys_catalog.cc:565] T 00000000000000000000000000000000 P adc2eba2a9e64b4ca9c6bfe15414df03 [sys.catalog]: configured and running, proceeding with master startup.
I20251215 21:02:15.868687 348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:02:15.858048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53058 (local address 127.28.113.62:44037)
1215 21:02:15.858424 (+ 376us) server_negotiation.cc:207] Beginning negotiation
1215 21:02:15.858435 (+ 11us) server_negotiation.cc:400] Waiting for connection header
1215 21:02:15.858469 (+ 34us) server_negotiation.cc:408] Connection header received
1215 21:02:15.859095 (+ 626us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1215 21:02:15.859117 (+ 22us) server_negotiation.cc:462] Received NEGOTIATE request from client
1215 21:02:15.859437 (+ 320us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1215 21:02:15.859742 (+ 305us) server_negotiation.cc:227] Negotiated authn=SASL
1215 21:02:15.860524 (+ 782us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:02:15.861487 (+ 963us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1215 21:02:15.862123 (+ 636us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:02:15.862362 (+ 239us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:02:15.864232 (+ 1870us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1215 21:02:15.864251 (+ 19us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1215 21:02:15.864263 (+ 12us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1215 21:02:15.864298 (+ 35us) server_negotiation.cc:923] calling sasl_server_start()
1215 21:02:15.866151 (+ 1853us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1215 21:02:15.866515 (+ 364us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1215 21:02:15.866520 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1215 21:02:15.866526 (+ 6us) server_negotiation.cc:968] calling sasl_server_step()
1215 21:02:15.866611 (+ 85us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1215 21:02:15.866793 (+ 182us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1215 21:02:15.866797 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1215 21:02:15.866799 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1215 21:02:15.867195 (+ 396us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1215 21:02:15.867416 (+ 221us) server_negotiation.cc:1036] Waiting for connection context
1215 21:02:15.867679 (+ 263us) server_negotiation.cc:300] Negotiation successful
1215 21:02:15.867933 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":244,"thread_start_us":109,"threads_started":1}
I20251215 21:02:15.869422 350 sys_catalog.cc:455] T 00000000000000000000000000000000 P adc2eba2a9e64b4ca9c6bfe15414df03 [sys.catalog]: SysCatalogTable state changed. Reason: New leader adc2eba2a9e64b4ca9c6bfe15414df03. Latest consensus state: current_term: 1 leader_uuid: "adc2eba2a9e64b4ca9c6bfe15414df03" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "adc2eba2a9e64b4ca9c6bfe15414df03" member_type: VOTER last_known_addr { host: "127.28.113.62" port: 44037 } } }
I20251215 21:02:15.869585 350 sys_catalog.cc:458] T 00000000000000000000000000000000 P adc2eba2a9e64b4ca9c6bfe15414df03 [sys.catalog]: This master's current role is: LEADER
I20251215 21:02:15.869732 351 sys_catalog.cc:455] T 00000000000000000000000000000000 P adc2eba2a9e64b4ca9c6bfe15414df03 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "adc2eba2a9e64b4ca9c6bfe15414df03" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "adc2eba2a9e64b4ca9c6bfe15414df03" member_type: VOTER last_known_addr { host: "127.28.113.62" port: 44037 } } }
I20251215 21:02:15.869901 351 sys_catalog.cc:458] T 00000000000000000000000000000000 P adc2eba2a9e64b4ca9c6bfe15414df03 [sys.catalog]: This master's current role is: LEADER
I20251215 21:02:15.870519 359 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251215 21:02:15.873237 359 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251215 21:02:15.878021 359 catalog_manager.cc:1357] Generated new cluster ID: 3461d3e8a55f4447bc232862e5509ded
I20251215 21:02:15.878070 359 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251215 21:02:15.887779 359 catalog_manager.cc:1380] Generated new certificate authority record
I20251215 21:02:15.888712 359 catalog_manager.cc:1514] Loading token signing keys...
I20251215 21:02:15.897403 359 catalog_manager.cc:6027] T 00000000000000000000000000000000 P adc2eba2a9e64b4ca9c6bfe15414df03: Generated new TSK 0
I20251215 21:02:15.897974 359 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.113.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.113.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.113.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765832506145157-29124-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.28.113.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765832506145157-29124-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.28.113.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.28.113.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.28.113.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765832506145157-29124-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.28.113.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765832506145157-29124-0/krb5kdc/kudu.keytab.
I20251215 21:02:15.948865 29124 external_mini_cluster.cc:1366] Running /tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu
/tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765832506145157-29124-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765832506145157-29124-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765832506145157-29124-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765832506145157-29124-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.113.1:0
--local_ip_for_outbound_sockets=127.28.113.1
--webserver_interface=127.28.113.1
--webserver_port=0
--tserver_master_addrs=127.28.113.62:44037
--keytab_file=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765832506145157-29124-0/krb5kdc/kudu.keytab
--principal=kudu/127.28.113.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.28.113.20:43859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765832506145157-29124-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765832506145157-29124-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765832506145157-29124-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251215 21:02:16.074918 373 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251215 21:02:16.075107 373 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251215 21:02:16.075165 373 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251215 21:02:16.078244 373 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251215 21:02:16.078297 373 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251215 21:02:16.078388 373 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.113.1
I20251215 21:02:16.082290 373 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.113.20:43859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765832506145157-29124-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765832506145157-29124-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765832506145157-29124-0/krb5kdc/kudu.keytab
--principal=kudu/127.28.113.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.113.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765832506145157-29124-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.113.1
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.28.113.62:44037
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.373
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.113.1
--log_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765832506145157-29124-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision dc741732bb6a40213e11a13869fad06a49a209a3
build type DEBUG
built by None at 15 Dec 2025 20:43:14 UTC on 24a791456cd2
build id 9530
I20251215 21:02:16.083297 373 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251215 21:02:16.084091 373 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251215 21:02:16.090173 378 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251215 21:02:16.090276 381 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251215 21:02:16.090624 373 server_base.cc:1047] running on GCE node
W20251215 21:02:16.090690 379 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251215 21:02:16.091069 373 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251215 21:02:16.091544 373 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251215 21:02:16.092675 373 hybrid_clock.cc:648] HybridClock initialized: now 1765832536092666 us; error 26 us; skew 500 ppm
Dec 15 21:02:16 dist-test-slave-bjsf krb5kdc[32729](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765832536, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.113.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251215 21:02:16.095022 373 init.cc:377] Logged in from keytab as kudu/127.28.113.1@KRBTEST.COM (short username kudu)
I20251215 21:02:16.095942 373 webserver.cc:492] Webserver started at http://127.28.113.1:45477/ using document root <none> and password file <none>
I20251215 21:02:16.096443 373 fs_manager.cc:362] Metadata directory not provided
I20251215 21:02:16.096490 373 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251215 21:02:16.096648 373 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251215 21:02:16.098188 373 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765832506145157-29124-0/minicluster-data/ts-0/data/instance:
uuid: "1a3ff414e1434501aee1f8d8388c0b69"
format_stamp: "Formatted at 2025-12-15 21:02:16 on dist-test-slave-bjsf"
server_key: "69dfbcbf7d885311e830c5278594c49c"
server_key_iv: "8dfb2b7003a14ee238e076336d4a6bc7"
server_key_version: "encryptionkey@0"
I20251215 21:02:16.098598 373 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765832506145157-29124-0/minicluster-data/ts-0/wal/instance:
uuid: "1a3ff414e1434501aee1f8d8388c0b69"
format_stamp: "Formatted at 2025-12-15 21:02:16 on dist-test-slave-bjsf"
server_key: "69dfbcbf7d885311e830c5278594c49c"
server_key_iv: "8dfb2b7003a14ee238e076336d4a6bc7"
server_key_version: "encryptionkey@0"
I20251215 21:02:16.101575 373 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20251215 21:02:16.103492 388 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251215 21:02:16.104430 373 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20251215 21:02:16.104525 373 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765832506145157-29124-0/minicluster-data/ts-0/data,/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765832506145157-29124-0/minicluster-data/ts-0/wal
uuid: "1a3ff414e1434501aee1f8d8388c0b69"
format_stamp: "Formatted at 2025-12-15 21:02:16 on dist-test-slave-bjsf"
server_key: "69dfbcbf7d885311e830c5278594c49c"
server_key_iv: "8dfb2b7003a14ee238e076336d4a6bc7"
server_key_version: "encryptionkey@0"
I20251215 21:02:16.104600 373 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765832506145157-29124-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765832506145157-29124-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765832506145157-29124-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251215 21:02:16.121217 373 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251215 21:02:16.123457 373 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251215 21:02:16.123608 373 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251215 21:02:16.124097 373 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251215 21:02:16.124889 373 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251215 21:02:16.124944 373 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251215 21:02:16.124995 373 ts_tablet_manager.cc:616] Registered 0 tablets
I20251215 21:02:16.125021 373 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251215 21:02:16.133159 373 rpc_server.cc:307] RPC server started. Bound to: 127.28.113.1:34019
I20251215 21:02:16.133184 501 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.28.113.1:34019 every 8 connection(s)
I20251215 21:02:16.133982 373 server_base.cc:1179] Dumped server information to /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765832506145157-29124-0/minicluster-data/ts-0/data/info.pb
Dec 15 21:02:16 dist-test-slave-bjsf krb5kdc[32729](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765832536, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.113.1@KRBTEST.COM for kudu/127.28.113.62@KRBTEST.COM
I20251215 21:02:16.143657 29124 external_mini_cluster.cc:1428] Started /tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu as pid 373
I20251215 21:02:16.143746 29124 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765832506145157-29124-0/minicluster-data/ts-0/wal/instance
I20251215 21:02:16.144011 29124 external_mini_cluster.cc:1468] Setting key 43f5969557a2793bc21aef0dafbeeeb6
I20251215 21:02:16.144892 348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:02:16.135470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.113.1:56605 (local address 127.28.113.62:44037)
1215 21:02:16.135584 (+ 114us) server_negotiation.cc:207] Beginning negotiation
1215 21:02:16.135588 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1215 21:02:16.136219 (+ 631us) server_negotiation.cc:408] Connection header received
1215 21:02:16.136987 (+ 768us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1215 21:02:16.136991 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1215 21:02:16.137053 (+ 62us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1215 21:02:16.137124 (+ 71us) server_negotiation.cc:227] Negotiated authn=SASL
1215 21:02:16.138746 (+ 1622us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:02:16.139250 (+ 504us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1215 21:02:16.139855 (+ 605us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:02:16.140076 (+ 221us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:02:16.141911 (+ 1835us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1215 21:02:16.141931 (+ 20us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1215 21:02:16.141934 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1215 21:02:16.141959 (+ 25us) server_negotiation.cc:923] calling sasl_server_start()
1215 21:02:16.143101 (+ 1142us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1215 21:02:16.143517 (+ 416us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1215 21:02:16.143520 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1215 21:02:16.143521 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1215 21:02:16.143570 (+ 49us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1215 21:02:16.143853 (+ 283us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1215 21:02:16.143857 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1215 21:02:16.143859 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1215 21:02:16.144061 (+ 202us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1215 21:02:16.144161 (+ 100us) server_negotiation.cc:1036] Waiting for connection context
1215 21:02:16.144651 (+ 490us) server_negotiation.cc:300] Negotiation successful
1215 21:02:16.144776 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20251215 21:02:16.145545 504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:02:16.135674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.113.62:44037 (local address 127.28.113.1:56605)
1215 21:02:16.136097 (+ 423us) negotiation.cc:107] Waiting for socket to connect
1215 21:02:16.136130 (+ 33us) client_negotiation.cc:174] Beginning negotiation
1215 21:02:16.136800 (+ 670us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1215 21:02:16.137282 (+ 482us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1215 21:02:16.137289 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
1215 21:02:16.137625 (+ 336us) client_negotiation.cc:189] Negotiated authn=SASL
1215 21:02:16.138590 (+ 965us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:02:16.138603 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:02:16.139361 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1215 21:02:16.139364 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1215 21:02:16.139742 (+ 378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:02:16.139749 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:02:16.139953 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:02:16.140500 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1215 21:02:16.140521 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
1215 21:02:16.141778 (+ 1257us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1215 21:02:16.143206 (+ 1428us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1215 21:02:16.143212 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1215 21:02:16.143222 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
1215 21:02:16.143428 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1215 21:02:16.143645 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1215 21:02:16.143648 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1215 21:02:16.143650 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1215 21:02:16.143763 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1215 21:02:16.144168 (+ 405us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1215 21:02:16.144175 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1215 21:02:16.144431 (+ 256us) client_negotiation.cc:769] Sending connection context
1215 21:02:16.144607 (+ 176us) client_negotiation.cc:240] Negotiation successful
1215 21:02:16.144852 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":112,"threads_started":1}
I20251215 21:02:16.146577 502 heartbeater.cc:344] Connected to a master server at 127.28.113.62:44037
I20251215 21:02:16.146790 502 heartbeater.cc:461] Registering TS with master...
I20251215 21:02:16.147363 502 heartbeater.cc:507] Master 127.28.113.62:44037 requested a full tablet report, sending...
I20251215 21:02:16.148859 309 ts_manager.cc:194] Registered new tserver with Master: 1a3ff414e1434501aee1f8d8388c0b69 (127.28.113.1:34019)
I20251215 21:02:16.150396 309 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.113.1@KRBTEST.COM'} at 127.28.113.1:56605
WARNING: no policy specified for kudu/127.28.113.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.113.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.113.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765832506145157-29124-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.28.113.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765832506145157-29124-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.28.113.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.28.113.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.28.113.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765832506145157-29124-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.28.113.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765832506145157-29124-0/krb5kdc/kudu.keytab.
I20251215 21:02:16.184303 29124 external_mini_cluster.cc:1366] Running /tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu
/tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765832506145157-29124-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765832506145157-29124-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765832506145157-29124-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765832506145157-29124-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.113.2:0
--local_ip_for_outbound_sockets=127.28.113.2
--webserver_interface=127.28.113.2
--webserver_port=0
--tserver_master_addrs=127.28.113.62:44037
--keytab_file=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765832506145157-29124-0/krb5kdc/kudu.keytab
--principal=kudu/127.28.113.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.28.113.20:43859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765832506145157-29124-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765832506145157-29124-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765832506145157-29124-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251215 21:02:16.283833 509 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251215 21:02:16.284073 509 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251215 21:02:16.284128 509 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251215 21:02:16.287200 509 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251215 21:02:16.287252 509 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251215 21:02:16.287326 509 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.113.2
I20251215 21:02:16.291249 509 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.113.20:43859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765832506145157-29124-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765832506145157-29124-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765832506145157-29124-0/krb5kdc/kudu.keytab
--principal=kudu/127.28.113.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.113.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765832506145157-29124-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.113.2
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.28.113.62:44037
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.509
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.113.2
--log_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765832506145157-29124-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision dc741732bb6a40213e11a13869fad06a49a209a3
build type DEBUG
built by None at 15 Dec 2025 20:43:14 UTC on 24a791456cd2
build id 9530
I20251215 21:02:16.292165 509 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251215 21:02:16.292889 509 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251215 21:02:16.298815 517 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251215 21:02:16.298899 514 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251215 21:02:16.298930 509 server_base.cc:1047] running on GCE node
W20251215 21:02:16.298825 515 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251215 21:02:16.299356 509 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251215 21:02:16.299865 509 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251215 21:02:16.301007 509 hybrid_clock.cc:648] HybridClock initialized: now 1765832536300991 us; error 30 us; skew 500 ppm
Dec 15 21:02:16 dist-test-slave-bjsf krb5kdc[32729](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765832536, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.113.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251215 21:02:16.303360 509 init.cc:377] Logged in from keytab as kudu/127.28.113.2@KRBTEST.COM (short username kudu)
I20251215 21:02:16.304342 509 webserver.cc:492] Webserver started at http://127.28.113.2:45231/ using document root <none> and password file <none>
I20251215 21:02:16.304845 509 fs_manager.cc:362] Metadata directory not provided
I20251215 21:02:16.304893 509 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251215 21:02:16.305054 509 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251215 21:02:16.306609 509 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765832506145157-29124-0/minicluster-data/ts-1/data/instance:
uuid: "8dafdec01080450baf50cda9c7125199"
format_stamp: "Formatted at 2025-12-15 21:02:16 on dist-test-slave-bjsf"
server_key: "ef5fb6866a8b2c218f8a60f423988e9e"
server_key_iv: "1e5c46f002ebb038a60b321fca51d217"
server_key_version: "encryptionkey@0"
I20251215 21:02:16.307039 509 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765832506145157-29124-0/minicluster-data/ts-1/wal/instance:
uuid: "8dafdec01080450baf50cda9c7125199"
format_stamp: "Formatted at 2025-12-15 21:02:16 on dist-test-slave-bjsf"
server_key: "ef5fb6866a8b2c218f8a60f423988e9e"
server_key_iv: "1e5c46f002ebb038a60b321fca51d217"
server_key_version: "encryptionkey@0"
I20251215 21:02:16.310139 509 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20251215 21:02:16.312124 524 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251215 21:02:16.313023 509 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20251215 21:02:16.313119 509 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765832506145157-29124-0/minicluster-data/ts-1/data,/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765832506145157-29124-0/minicluster-data/ts-1/wal
uuid: "8dafdec01080450baf50cda9c7125199"
format_stamp: "Formatted at 2025-12-15 21:02:16 on dist-test-slave-bjsf"
server_key: "ef5fb6866a8b2c218f8a60f423988e9e"
server_key_iv: "1e5c46f002ebb038a60b321fca51d217"
server_key_version: "encryptionkey@0"
I20251215 21:02:16.313197 509 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765832506145157-29124-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765832506145157-29124-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765832506145157-29124-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251215 21:02:16.325949 509 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251215 21:02:16.328413 509 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251215 21:02:16.328562 509 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251215 21:02:16.329042 509 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251215 21:02:16.329824 509 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251215 21:02:16.329869 509 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251215 21:02:16.329917 509 ts_tablet_manager.cc:616] Registered 0 tablets
I20251215 21:02:16.329943 509 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251215 21:02:16.338248 509 rpc_server.cc:307] RPC server started. Bound to: 127.28.113.2:37845
I20251215 21:02:16.338276 637 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.28.113.2:37845 every 8 connection(s)
I20251215 21:02:16.339093 509 server_base.cc:1179] Dumped server information to /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765832506145157-29124-0/minicluster-data/ts-1/data/info.pb
Dec 15 21:02:16 dist-test-slave-bjsf krb5kdc[32729](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765832536, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.113.2@KRBTEST.COM for kudu/127.28.113.62@KRBTEST.COM
I20251215 21:02:16.348974 29124 external_mini_cluster.cc:1428] Started /tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu as pid 509
I20251215 21:02:16.349084 29124 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765832506145157-29124-0/minicluster-data/ts-1/wal/instance
I20251215 21:02:16.349344 29124 external_mini_cluster.cc:1468] Setting key c5759cac40a1060ba5a04ade09b2a4b4
I20251215 21:02:16.350221 348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:02:16.340705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.113.2:48827 (local address 127.28.113.62:44037)
1215 21:02:16.340853 (+ 148us) server_negotiation.cc:207] Beginning negotiation
1215 21:02:16.340858 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1215 21:02:16.341477 (+ 619us) server_negotiation.cc:408] Connection header received
1215 21:02:16.342218 (+ 741us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1215 21:02:16.342221 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1215 21:02:16.342267 (+ 46us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1215 21:02:16.342337 (+ 70us) server_negotiation.cc:227] Negotiated authn=SASL
1215 21:02:16.343936 (+ 1599us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:02:16.344455 (+ 519us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1215 21:02:16.345104 (+ 649us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:02:16.345236 (+ 132us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:02:16.347138 (+ 1902us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1215 21:02:16.347156 (+ 18us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1215 21:02:16.347158 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1215 21:02:16.347182 (+ 24us) server_negotiation.cc:923] calling sasl_server_start()
1215 21:02:16.348484 (+ 1302us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1215 21:02:16.348930 (+ 446us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1215 21:02:16.348933 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1215 21:02:16.348934 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1215 21:02:16.348978 (+ 44us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1215 21:02:16.349276 (+ 298us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1215 21:02:16.349278 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1215 21:02:16.349280 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1215 21:02:16.349441 (+ 161us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1215 21:02:16.349515 (+ 74us) server_negotiation.cc:1036] Waiting for connection context
1215 21:02:16.350008 (+ 493us) server_negotiation.cc:300] Negotiation successful
1215 21:02:16.350107 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20251215 21:02:16.350916 640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:02:16.340954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.113.62:44037 (local address 127.28.113.2:48827)
1215 21:02:16.341352 (+ 398us) negotiation.cc:107] Waiting for socket to connect
1215 21:02:16.341386 (+ 34us) client_negotiation.cc:174] Beginning negotiation
1215 21:02:16.342055 (+ 669us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1215 21:02:16.342479 (+ 424us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1215 21:02:16.342486 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
1215 21:02:16.342841 (+ 355us) client_negotiation.cc:189] Negotiated authn=SASL
1215 21:02:16.343814 (+ 973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:02:16.343827 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:02:16.344549 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1215 21:02:16.344553 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1215 21:02:16.344995 (+ 442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:02:16.345004 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:02:16.345166 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:02:16.345661 (+ 495us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1215 21:02:16.345681 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
1215 21:02:16.347005 (+ 1324us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1215 21:02:16.348574 (+ 1569us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1215 21:02:16.348577 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1215 21:02:16.348584 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
1215 21:02:16.348825 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1215 21:02:16.349062 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1215 21:02:16.349065 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1215 21:02:16.349067 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1215 21:02:16.349179 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1215 21:02:16.349528 (+ 349us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1215 21:02:16.349534 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1215 21:02:16.349783 (+ 249us) client_negotiation.cc:769] Sending connection context
1215 21:02:16.349956 (+ 173us) client_negotiation.cc:240] Negotiation successful
1215 21:02:16.350203 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":102,"threads_started":1}
I20251215 21:02:16.351837 638 heartbeater.cc:344] Connected to a master server at 127.28.113.62:44037
I20251215 21:02:16.352113 638 heartbeater.cc:461] Registering TS with master...
I20251215 21:02:16.352607 638 heartbeater.cc:507] Master 127.28.113.62:44037 requested a full tablet report, sending...
I20251215 21:02:16.353565 309 ts_manager.cc:194] Registered new tserver with Master: 8dafdec01080450baf50cda9c7125199 (127.28.113.2:37845)
I20251215 21:02:16.354127 309 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.113.2@KRBTEST.COM'} at 127.28.113.2:48827
WARNING: no policy specified for kudu/127.28.113.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.113.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.113.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765832506145157-29124-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.28.113.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765832506145157-29124-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.28.113.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.28.113.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.28.113.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765832506145157-29124-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.28.113.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765832506145157-29124-0/krb5kdc/kudu.keytab.
I20251215 21:02:16.387719 29124 external_mini_cluster.cc:1366] Running /tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu
/tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765832506145157-29124-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765832506145157-29124-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765832506145157-29124-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765832506145157-29124-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.113.3:0
--local_ip_for_outbound_sockets=127.28.113.3
--webserver_interface=127.28.113.3
--webserver_port=0
--tserver_master_addrs=127.28.113.62:44037
--keytab_file=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765832506145157-29124-0/krb5kdc/kudu.keytab
--principal=kudu/127.28.113.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.28.113.20:43859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765832506145157-29124-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765832506145157-29124-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765832506145157-29124-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251215 21:02:16.484855 645 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251215 21:02:16.485061 645 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251215 21:02:16.485121 645 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251215 21:02:16.488248 645 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251215 21:02:16.488301 645 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251215 21:02:16.488394 645 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.113.3
I20251215 21:02:16.492523 645 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.113.20:43859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765832506145157-29124-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765832506145157-29124-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765832506145157-29124-0/krb5kdc/kudu.keytab
--principal=kudu/127.28.113.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.113.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765832506145157-29124-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.113.3
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.28.113.62:44037
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.645
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.113.3
--log_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765832506145157-29124-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision dc741732bb6a40213e11a13869fad06a49a209a3
build type DEBUG
built by None at 15 Dec 2025 20:43:14 UTC on 24a791456cd2
build id 9530
I20251215 21:02:16.493465 645 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251215 21:02:16.494252 645 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251215 21:02:16.500355 651 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251215 21:02:16.500360 650 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251215 21:02:16.500360 653 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251215 21:02:16.500571 645 server_base.cc:1047] running on GCE node
I20251215 21:02:16.500943 645 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251215 21:02:16.501435 645 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251215 21:02:16.502573 645 hybrid_clock.cc:648] HybridClock initialized: now 1765832536502548 us; error 41 us; skew 500 ppm
Dec 15 21:02:16 dist-test-slave-bjsf krb5kdc[32729](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765832536, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.113.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251215 21:02:16.504954 645 init.cc:377] Logged in from keytab as kudu/127.28.113.3@KRBTEST.COM (short username kudu)
I20251215 21:02:16.505877 645 webserver.cc:492] Webserver started at http://127.28.113.3:36731/ using document root <none> and password file <none>
I20251215 21:02:16.506345 645 fs_manager.cc:362] Metadata directory not provided
I20251215 21:02:16.506386 645 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251215 21:02:16.506532 645 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251215 21:02:16.508136 645 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765832506145157-29124-0/minicluster-data/ts-2/data/instance:
uuid: "fe1ddb1b1bf14f46afe622f4f220ddd8"
format_stamp: "Formatted at 2025-12-15 21:02:16 on dist-test-slave-bjsf"
server_key: "070b8034605ec6c11d64b25a8c71ab7d"
server_key_iv: "253c7313f95e88ff1d7bcccc68b0d37f"
server_key_version: "encryptionkey@0"
I20251215 21:02:16.508546 645 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765832506145157-29124-0/minicluster-data/ts-2/wal/instance:
uuid: "fe1ddb1b1bf14f46afe622f4f220ddd8"
format_stamp: "Formatted at 2025-12-15 21:02:16 on dist-test-slave-bjsf"
server_key: "070b8034605ec6c11d64b25a8c71ab7d"
server_key_iv: "253c7313f95e88ff1d7bcccc68b0d37f"
server_key_version: "encryptionkey@0"
I20251215 21:02:16.511631 645 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20251215 21:02:16.513643 660 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251215 21:02:16.514554 645 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20251215 21:02:16.514647 645 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765832506145157-29124-0/minicluster-data/ts-2/data,/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765832506145157-29124-0/minicluster-data/ts-2/wal
uuid: "fe1ddb1b1bf14f46afe622f4f220ddd8"
format_stamp: "Formatted at 2025-12-15 21:02:16 on dist-test-slave-bjsf"
server_key: "070b8034605ec6c11d64b25a8c71ab7d"
server_key_iv: "253c7313f95e88ff1d7bcccc68b0d37f"
server_key_version: "encryptionkey@0"
I20251215 21:02:16.514734 645 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765832506145157-29124-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765832506145157-29124-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765832506145157-29124-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251215 21:02:16.532632 645 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251215 21:02:16.535320 645 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251215 21:02:16.535485 645 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251215 21:02:16.536031 645 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251215 21:02:16.536844 645 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251215 21:02:16.536888 645 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251215 21:02:16.536936 645 ts_tablet_manager.cc:616] Registered 0 tablets
I20251215 21:02:16.536963 645 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251215 21:02:16.545194 645 rpc_server.cc:307] RPC server started. Bound to: 127.28.113.3:36971
I20251215 21:02:16.545235 773 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.28.113.3:36971 every 8 connection(s)
I20251215 21:02:16.546056 645 server_base.cc:1179] Dumped server information to /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765832506145157-29124-0/minicluster-data/ts-2/data/info.pb
I20251215 21:02:16.551891 29124 external_mini_cluster.cc:1428] Started /tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu as pid 645
I20251215 21:02:16.552014 29124 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765832506145157-29124-0/minicluster-data/ts-2/wal/instance
I20251215 21:02:16.552259 29124 external_mini_cluster.cc:1468] Setting key 2d21aa1e4a74eceb374e9870a65b8157
Dec 15 21:02:16 dist-test-slave-bjsf krb5kdc[32729](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765832536, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.113.3@KRBTEST.COM for kudu/127.28.113.62@KRBTEST.COM
I20251215 21:02:16.557499 348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:02:16.547664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.113.3:57527 (local address 127.28.113.62:44037)
1215 21:02:16.547807 (+ 143us) server_negotiation.cc:207] Beginning negotiation
1215 21:02:16.547811 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1215 21:02:16.548463 (+ 652us) server_negotiation.cc:408] Connection header received
1215 21:02:16.549273 (+ 810us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1215 21:02:16.549276 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1215 21:02:16.549324 (+ 48us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1215 21:02:16.549385 (+ 61us) server_negotiation.cc:227] Negotiated authn=SASL
1215 21:02:16.551044 (+ 1659us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:02:16.551525 (+ 481us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1215 21:02:16.552163 (+ 638us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:02:16.552341 (+ 178us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:02:16.554652 (+ 2311us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1215 21:02:16.554666 (+ 14us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1215 21:02:16.554668 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1215 21:02:16.554697 (+ 29us) server_negotiation.cc:923] calling sasl_server_start()
1215 21:02:16.555807 (+ 1110us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1215 21:02:16.556268 (+ 461us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1215 21:02:16.556270 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1215 21:02:16.556272 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1215 21:02:16.556313 (+ 41us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1215 21:02:16.556573 (+ 260us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1215 21:02:16.556576 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1215 21:02:16.556577 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1215 21:02:16.556720 (+ 143us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1215 21:02:16.556797 (+ 77us) server_negotiation.cc:1036] Waiting for connection context
1215 21:02:16.557246 (+ 449us) server_negotiation.cc:300] Negotiation successful
1215 21:02:16.557357 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20251215 21:02:16.558045 776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:02:16.547905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.113.62:44037 (local address 127.28.113.3:57527)
1215 21:02:16.548339 (+ 434us) negotiation.cc:107] Waiting for socket to connect
1215 21:02:16.548374 (+ 35us) client_negotiation.cc:174] Beginning negotiation
1215 21:02:16.549086 (+ 712us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1215 21:02:16.549518 (+ 432us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1215 21:02:16.549525 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
1215 21:02:16.549912 (+ 387us) client_negotiation.cc:189] Negotiated authn=SASL
1215 21:02:16.550925 (+ 1013us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:02:16.550939 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:02:16.551625 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1215 21:02:16.551628 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1215 21:02:16.552052 (+ 424us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:02:16.552061 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:02:16.552238 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:02:16.552834 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1215 21:02:16.552865 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
1215 21:02:16.554525 (+ 1660us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1215 21:02:16.555905 (+ 1380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1215 21:02:16.555910 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1215 21:02:16.555920 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
1215 21:02:16.556174 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1215 21:02:16.556385 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1215 21:02:16.556388 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1215 21:02:16.556389 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1215 21:02:16.556477 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1215 21:02:16.556811 (+ 334us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1215 21:02:16.556818 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1215 21:02:16.557025 (+ 207us) client_negotiation.cc:769] Sending connection context
1215 21:02:16.557183 (+ 158us) client_negotiation.cc:240] Negotiation successful
1215 21:02:16.557411 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":97,"threads_started":1}
I20251215 21:02:16.558907 774 heartbeater.cc:344] Connected to a master server at 127.28.113.62:44037
I20251215 21:02:16.559114 774 heartbeater.cc:461] Registering TS with master...
I20251215 21:02:16.559557 774 heartbeater.cc:507] Master 127.28.113.62:44037 requested a full tablet report, sending...
I20251215 21:02:16.560431 309 ts_manager.cc:194] Registered new tserver with Master: fe1ddb1b1bf14f46afe622f4f220ddd8 (127.28.113.3:36971)
I20251215 21:02:16.560937 309 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.113.3@KRBTEST.COM'} at 127.28.113.3:57527
I20251215 21:02:16.565644 29124 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
Dec 15 21:02:16 dist-test-slave-bjsf krb5kdc[32729](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765832536, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Dec 15 21:02:16 dist-test-slave-bjsf krb5kdc[32729](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765832536, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.28.113.62@KRBTEST.COM
I20251215 21:02:16.586556 348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:02:16.579795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53060 (local address 127.28.113.62:44037)
1215 21:02:16.579945 (+ 150us) server_negotiation.cc:207] Beginning negotiation
1215 21:02:16.579950 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1215 21:02:16.580047 (+ 97us) server_negotiation.cc:408] Connection header received
1215 21:02:16.580186 (+ 139us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1215 21:02:16.580189 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1215 21:02:16.580230 (+ 41us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1215 21:02:16.580291 (+ 61us) server_negotiation.cc:227] Negotiated authn=SASL
1215 21:02:16.581050 (+ 759us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:02:16.581520 (+ 470us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1215 21:02:16.582143 (+ 623us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:02:16.582321 (+ 178us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:02:16.584157 (+ 1836us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1215 21:02:16.584171 (+ 14us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1215 21:02:16.584173 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1215 21:02:16.584196 (+ 23us) server_negotiation.cc:923] calling sasl_server_start()
1215 21:02:16.585283 (+ 1087us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1215 21:02:16.585615 (+ 332us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1215 21:02:16.585619 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1215 21:02:16.585620 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1215 21:02:16.585660 (+ 40us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1215 21:02:16.585849 (+ 189us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1215 21:02:16.585853 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1215 21:02:16.585855 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1215 21:02:16.586013 (+ 158us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1215 21:02:16.586126 (+ 113us) server_negotiation.cc:1036] Waiting for connection context
1215 21:02:16.586312 (+ 186us) server_negotiation.cc:300] Negotiation successful
1215 21:02:16.586416 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
W20251215 21:02:16.586896 309 server_base.cc:1129] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:53060
I20251215 21:02:16.587659 29124 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu with pid 373
I20251215 21:02:16.592710 29124 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu with pid 509
I20251215 21:02:16.597611 29124 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu with pid 645
I20251215 21:02:16.604039 29124 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu with pid 32745
2025-12-15T21:02:16Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (2828 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765832506145157-29124-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Dec 15 21:02:16 dist-test-slave-bjsf krb5kdc[786](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Dec 15 21:02:16 dist-test-slave-bjsf krb5kdc[786](info): set up 2 sockets
Dec 15 21:02:16 dist-test-slave-bjsf krb5kdc[786](info): commencing operation
krb5kdc: starting...
W20251215 21:02:18.419874 29124 mini_kdc.cc:121] Time spent starting KDC: real 1.796s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765832506145157-29124-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765832506145157-29124-0/krb5kdc/test-user.keytab.
Dec 15 21:02:18 dist-test-slave-bjsf krb5kdc[786](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765832538, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-12-15T21:02:18Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-12-15T21:02:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.113.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.113.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.113.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765832506145157-29124-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.28.113.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765832506145157-29124-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.28.113.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.28.113.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.28.113.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765832506145157-29124-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.28.113.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765832506145157-29124-0/krb5kdc/kudu.keytab.
I20251215 21:02:18.538460 29124 external_mini_cluster.cc:1366] Running /tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu
/tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765832506145157-29124-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765832506145157-29124-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765832506145157-29124-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765832506145157-29124-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.113.62:44239
--webserver_interface=127.28.113.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765832506145157-29124-0/krb5kdc/kudu.keytab
--principal=kudu/127.28.113.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.28.113.20:32789
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.113.62:44239
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765832506145157-29124-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765832506145157-29124-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765832506145157-29124-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251215 21:02:18.639837 802 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251215 21:02:18.640107 802 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251215 21:02:18.640165 802 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251215 21:02:18.643281 802 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251215 21:02:18.643332 802 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251215 21:02:18.643353 802 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251215 21:02:18.643378 802 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251215 21:02:18.643392 802 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251215 21:02:18.647431 802 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.113.20:32789
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765832506145157-29124-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765832506145157-29124-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.113.62:44239
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765832506145157-29124-0/krb5kdc/kudu.keytab
--principal=kudu/127.28.113.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.113.62:44239
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765832506145157-29124-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.113.62
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.802
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765832506145157-29124-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision dc741732bb6a40213e11a13869fad06a49a209a3
build type DEBUG
built by None at 15 Dec 2025 20:43:14 UTC on 24a791456cd2
build id 9530
I20251215 21:02:18.648396 802 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251215 21:02:18.649144 802 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251215 21:02:18.654145 810 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251215 21:02:18.654281 802 server_base.cc:1047] running on GCE node
W20251215 21:02:18.654151 808 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251215 21:02:18.654218 807 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251215 21:02:18.654824 802 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251215 21:02:18.655635 802 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251215 21:02:18.656783 802 hybrid_clock.cc:648] HybridClock initialized: now 1765832538656774 us; error 27 us; skew 500 ppm
Dec 15 21:02:18 dist-test-slave-bjsf krb5kdc[786](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765832538, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.113.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251215 21:02:18.659073 802 init.cc:377] Logged in from keytab as kudu/127.28.113.62@KRBTEST.COM (short username kudu)
I20251215 21:02:18.660007 802 webserver.cc:492] Webserver started at http://127.28.113.62:45429/ using document root <none> and password file <none>
I20251215 21:02:18.660475 802 fs_manager.cc:362] Metadata directory not provided
I20251215 21:02:18.660517 802 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251215 21:02:18.660667 802 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251215 21:02:18.662212 802 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765832506145157-29124-0/minicluster-data/master-0/data/instance:
uuid: "1c940ed81e014d40ab1ea764c7d5333c"
format_stamp: "Formatted at 2025-12-15 21:02:18 on dist-test-slave-bjsf"
server_key: "672e3533a61235a0d19e82311271ccbd"
server_key_iv: "cb56c511c1e564beeea6cd1eb2ce2fcb"
server_key_version: "encryptionkey@0"
I20251215 21:02:18.662638 802 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765832506145157-29124-0/minicluster-data/master-0/wal/instance:
uuid: "1c940ed81e014d40ab1ea764c7d5333c"
format_stamp: "Formatted at 2025-12-15 21:02:18 on dist-test-slave-bjsf"
server_key: "672e3533a61235a0d19e82311271ccbd"
server_key_iv: "cb56c511c1e564beeea6cd1eb2ce2fcb"
server_key_version: "encryptionkey@0"
I20251215 21:02:18.665738 802 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20251215 21:02:18.667713 817 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251215 21:02:18.668690 802 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20251215 21:02:18.668781 802 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765832506145157-29124-0/minicluster-data/master-0/data,/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765832506145157-29124-0/minicluster-data/master-0/wal
uuid: "1c940ed81e014d40ab1ea764c7d5333c"
format_stamp: "Formatted at 2025-12-15 21:02:18 on dist-test-slave-bjsf"
server_key: "672e3533a61235a0d19e82311271ccbd"
server_key_iv: "cb56c511c1e564beeea6cd1eb2ce2fcb"
server_key_version: "encryptionkey@0"
I20251215 21:02:18.668865 802 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765832506145157-29124-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765832506145157-29124-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765832506145157-29124-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251215 21:02:18.684435 802 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251215 21:02:18.689839 802 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251215 21:02:18.689983 802 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251215 21:02:18.696297 869 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.28.113.62:44239 every 8 connection(s)
I20251215 21:02:18.696300 802 rpc_server.cc:307] RPC server started. Bound to: 127.28.113.62:44239
I20251215 21:02:18.697199 802 server_base.cc:1179] Dumped server information to /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765832506145157-29124-0/minicluster-data/master-0/data/info.pb
I20251215 21:02:18.699607 870 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 21:02:18.702540 29124 external_mini_cluster.cc:1428] Started /tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu as pid 802
I20251215 21:02:18.702654 29124 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765832506145157-29124-0/minicluster-data/master-0/wal/instance
I20251215 21:02:18.702878 29124 external_mini_cluster.cc:1468] Setting key 4d041f198c381f8afbb4a81b385be697
I20251215 21:02:18.704797 870 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1c940ed81e014d40ab1ea764c7d5333c: Bootstrap starting.
I20251215 21:02:18.706629 870 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1c940ed81e014d40ab1ea764c7d5333c: Neither blocks nor log segments found. Creating new log.
I20251215 21:02:18.707242 870 log.cc:826] T 00000000000000000000000000000000 P 1c940ed81e014d40ab1ea764c7d5333c: Log is configured to *not* fsync() on all Append() calls
I20251215 21:02:18.708796 870 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1c940ed81e014d40ab1ea764c7d5333c: No bootstrap required, opened a new log
Dec 15 21:02:18 dist-test-slave-bjsf krb5kdc[786](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765832538, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.113.62@KRBTEST.COM
I20251215 21:02:18.711177 870 raft_consensus.cc:359] T 00000000000000000000000000000000 P 1c940ed81e014d40ab1ea764c7d5333c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c940ed81e014d40ab1ea764c7d5333c" member_type: VOTER last_known_addr { host: "127.28.113.62" port: 44239 } }
I20251215 21:02:18.711335 870 raft_consensus.cc:385] T 00000000000000000000000000000000 P 1c940ed81e014d40ab1ea764c7d5333c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251215 21:02:18.711416 870 raft_consensus.cc:740] T 00000000000000000000000000000000 P 1c940ed81e014d40ab1ea764c7d5333c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c940ed81e014d40ab1ea764c7d5333c, State: Initialized, Role: FOLLOWER
I20251215 21:02:18.711827 870 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1c940ed81e014d40ab1ea764c7d5333c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c940ed81e014d40ab1ea764c7d5333c" member_type: VOTER last_known_addr { host: "127.28.113.62" port: 44239 } }
I20251215 21:02:18.711980 870 raft_consensus.cc:399] T 00000000000000000000000000000000 P 1c940ed81e014d40ab1ea764c7d5333c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251215 21:02:18.712085 870 raft_consensus.cc:493] T 00000000000000000000000000000000 P 1c940ed81e014d40ab1ea764c7d5333c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251215 21:02:18.712162 870 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 1c940ed81e014d40ab1ea764c7d5333c [term 0 FOLLOWER]: Advancing to term 1
I20251215 21:02:18.712981 870 raft_consensus.cc:515] T 00000000000000000000000000000000 P 1c940ed81e014d40ab1ea764c7d5333c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c940ed81e014d40ab1ea764c7d5333c" member_type: VOTER last_known_addr { host: "127.28.113.62" port: 44239 } }
I20251215 21:02:18.713280 870 leader_election.cc:304] T 00000000000000000000000000000000 P 1c940ed81e014d40ab1ea764c7d5333c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1c940ed81e014d40ab1ea764c7d5333c; no voters:
I20251215 21:02:18.713502 870 leader_election.cc:290] T 00000000000000000000000000000000 P 1c940ed81e014d40ab1ea764c7d5333c [CANDIDATE]: Term 1 election: Requested vote from peers
I20251215 21:02:18.713603 875 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 1c940ed81e014d40ab1ea764c7d5333c [term 1 FOLLOWER]: Leader election won for term 1
I20251215 21:02:18.713791 873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:02:18.704036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49220 (local address 127.28.113.62:44239)
1215 21:02:18.704392 (+ 356us) server_negotiation.cc:207] Beginning negotiation
1215 21:02:18.704401 (+ 9us) server_negotiation.cc:400] Waiting for connection header
1215 21:02:18.704431 (+ 30us) server_negotiation.cc:408] Connection header received
1215 21:02:18.704929 (+ 498us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1215 21:02:18.704946 (+ 17us) server_negotiation.cc:462] Received NEGOTIATE request from client
1215 21:02:18.705207 (+ 261us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1215 21:02:18.705468 (+ 261us) server_negotiation.cc:227] Negotiated authn=SASL
1215 21:02:18.706289 (+ 821us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:02:18.707024 (+ 735us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1215 21:02:18.707657 (+ 633us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:02:18.707906 (+ 249us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:02:18.709757 (+ 1851us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1215 21:02:18.709773 (+ 16us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1215 21:02:18.709781 (+ 8us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1215 21:02:18.709811 (+ 30us) server_negotiation.cc:923] calling sasl_server_start()
1215 21:02:18.711311 (+ 1500us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1215 21:02:18.711726 (+ 415us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1215 21:02:18.711730 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1215 21:02:18.711734 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
1215 21:02:18.711797 (+ 63us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1215 21:02:18.712047 (+ 250us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1215 21:02:18.712051 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1215 21:02:18.712053 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1215 21:02:18.712384 (+ 331us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1215 21:02:18.712540 (+ 156us) server_negotiation.cc:1036] Waiting for connection context
1215 21:02:18.712769 (+ 229us) server_negotiation.cc:300] Negotiation successful
1215 21:02:18.712969 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":234,"thread_start_us":115,"threads_started":1}
I20251215 21:02:18.713798 875 raft_consensus.cc:697] T 00000000000000000000000000000000 P 1c940ed81e014d40ab1ea764c7d5333c [term 1 LEADER]: Becoming Leader. State: Replica: 1c940ed81e014d40ab1ea764c7d5333c, State: Running, Role: LEADER
I20251215 21:02:18.714291 875 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1c940ed81e014d40ab1ea764c7d5333c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c940ed81e014d40ab1ea764c7d5333c" member_type: VOTER last_known_addr { host: "127.28.113.62" port: 44239 } }
I20251215 21:02:18.714434 870 sys_catalog.cc:565] T 00000000000000000000000000000000 P 1c940ed81e014d40ab1ea764c7d5333c [sys.catalog]: configured and running, proceeding with master startup.
I20251215 21:02:18.716207 876 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1c940ed81e014d40ab1ea764c7d5333c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1c940ed81e014d40ab1ea764c7d5333c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c940ed81e014d40ab1ea764c7d5333c" member_type: VOTER last_known_addr { host: "127.28.113.62" port: 44239 } } }
I20251215 21:02:18.716464 876 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1c940ed81e014d40ab1ea764c7d5333c [sys.catalog]: This master's current role is: LEADER
I20251215 21:02:18.716775 877 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1c940ed81e014d40ab1ea764c7d5333c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1c940ed81e014d40ab1ea764c7d5333c. Latest consensus state: current_term: 1 leader_uuid: "1c940ed81e014d40ab1ea764c7d5333c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c940ed81e014d40ab1ea764c7d5333c" member_type: VOTER last_known_addr { host: "127.28.113.62" port: 44239 } } }
I20251215 21:02:18.716884 877 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1c940ed81e014d40ab1ea764c7d5333c [sys.catalog]: This master's current role is: LEADER
W20251215 21:02:18.718641 889 catalog_manager.cc:1568] T 00000000000000000000000000000000 P 1c940ed81e014d40ab1ea764c7d5333c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20251215 21:02:18.718703 889 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20251215 21:02:18.718767 891 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251215 21:02:18.719419 891 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251215 21:02:18.723879 891 catalog_manager.cc:1357] Generated new cluster ID: 893027dd737243c9a1e3ca66782d5d3b
I20251215 21:02:18.723937 891 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251215 21:02:18.743055 891 catalog_manager.cc:1380] Generated new certificate authority record
I20251215 21:02:18.744189 891 catalog_manager.cc:1514] Loading token signing keys...
I20251215 21:02:18.759202 891 catalog_manager.cc:6027] T 00000000000000000000000000000000 P 1c940ed81e014d40ab1ea764c7d5333c: Generated new TSK 0
I20251215 21:02:18.759791 891 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.113.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.113.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.113.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765832506145157-29124-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.28.113.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765832506145157-29124-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.28.113.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.28.113.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.28.113.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765832506145157-29124-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.28.113.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765832506145157-29124-0/krb5kdc/kudu.keytab.
I20251215 21:02:18.804900 29124 external_mini_cluster.cc:1366] Running /tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu
/tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765832506145157-29124-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765832506145157-29124-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765832506145157-29124-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765832506145157-29124-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.113.1:0
--local_ip_for_outbound_sockets=127.28.113.1
--webserver_interface=127.28.113.1
--webserver_port=0
--tserver_master_addrs=127.28.113.62:44239
--keytab_file=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765832506145157-29124-0/krb5kdc/kudu.keytab
--principal=kudu/127.28.113.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.28.113.20:32789
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765832506145157-29124-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765832506145157-29124-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765832506145157-29124-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251215 21:02:18.902001 898 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251215 21:02:18.902163 898 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251215 21:02:18.902222 898 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251215 21:02:18.905313 898 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251215 21:02:18.905362 898 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251215 21:02:18.905462 898 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.113.1
I20251215 21:02:18.909368 898 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.113.20:32789
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765832506145157-29124-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765832506145157-29124-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765832506145157-29124-0/krb5kdc/kudu.keytab
--principal=kudu/127.28.113.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.113.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765832506145157-29124-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.113.1
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.28.113.62:44239
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.898
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.113.1
--log_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765832506145157-29124-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision dc741732bb6a40213e11a13869fad06a49a209a3
build type DEBUG
built by None at 15 Dec 2025 20:43:14 UTC on 24a791456cd2
build id 9530
I20251215 21:02:18.910269 898 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251215 21:02:18.911005 898 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251215 21:02:18.916589 906 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251215 21:02:18.916589 903 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251215 21:02:18.916627 904 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251215 21:02:18.916849 898 server_base.cc:1047] running on GCE node
I20251215 21:02:18.917155 898 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251215 21:02:18.917604 898 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251215 21:02:18.918733 898 hybrid_clock.cc:648] HybridClock initialized: now 1765832538918720 us; error 29 us; skew 500 ppm
Dec 15 21:02:18 dist-test-slave-bjsf krb5kdc[786](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765832538, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.113.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251215 21:02:18.920984 898 init.cc:377] Logged in from keytab as kudu/127.28.113.1@KRBTEST.COM (short username kudu)
I20251215 21:02:18.921825 898 webserver.cc:492] Webserver started at http://127.28.113.1:34013/ using document root <none> and password file <none>
I20251215 21:02:18.922290 898 fs_manager.cc:362] Metadata directory not provided
I20251215 21:02:18.922333 898 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251215 21:02:18.922503 898 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251215 21:02:18.924039 898 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765832506145157-29124-0/minicluster-data/ts-0/data/instance:
uuid: "f3f6b1b3d4c04c6faa72556352c6844a"
format_stamp: "Formatted at 2025-12-15 21:02:18 on dist-test-slave-bjsf"
server_key: "23e1b8e80c8a545ed49ae382d356bebc"
server_key_iv: "5a1a4796520de589b48f14980ba687f0"
server_key_version: "encryptionkey@0"
I20251215 21:02:18.924440 898 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765832506145157-29124-0/minicluster-data/ts-0/wal/instance:
uuid: "f3f6b1b3d4c04c6faa72556352c6844a"
format_stamp: "Formatted at 2025-12-15 21:02:18 on dist-test-slave-bjsf"
server_key: "23e1b8e80c8a545ed49ae382d356bebc"
server_key_iv: "5a1a4796520de589b48f14980ba687f0"
server_key_version: "encryptionkey@0"
I20251215 21:02:18.927254 898 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20251215 21:02:18.929083 913 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251215 21:02:18.929917 898 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20251215 21:02:18.930007 898 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765832506145157-29124-0/minicluster-data/ts-0/data,/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765832506145157-29124-0/minicluster-data/ts-0/wal
uuid: "f3f6b1b3d4c04c6faa72556352c6844a"
format_stamp: "Formatted at 2025-12-15 21:02:18 on dist-test-slave-bjsf"
server_key: "23e1b8e80c8a545ed49ae382d356bebc"
server_key_iv: "5a1a4796520de589b48f14980ba687f0"
server_key_version: "encryptionkey@0"
I20251215 21:02:18.930083 898 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765832506145157-29124-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765832506145157-29124-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765832506145157-29124-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251215 21:02:18.954032 898 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251215 21:02:18.956328 898 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251215 21:02:18.956476 898 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251215 21:02:18.956933 898 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251215 21:02:18.957690 898 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251215 21:02:18.957732 898 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251215 21:02:18.957772 898 ts_tablet_manager.cc:616] Registered 0 tablets
I20251215 21:02:18.957799 898 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251215 21:02:18.965572 898 rpc_server.cc:307] RPC server started. Bound to: 127.28.113.1:39309
I20251215 21:02:18.965603 1026 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.28.113.1:39309 every 8 connection(s)
I20251215 21:02:18.966387 898 server_base.cc:1179] Dumped server information to /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765832506145157-29124-0/minicluster-data/ts-0/data/info.pb
I20251215 21:02:18.968969 29124 external_mini_cluster.cc:1428] Started /tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu as pid 898
I20251215 21:02:18.969065 29124 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765832506145157-29124-0/minicluster-data/ts-0/wal/instance
I20251215 21:02:18.969277 29124 external_mini_cluster.cc:1468] Setting key 09cb92c226a07e74feb0c9a8f97c9496
Dec 15 21:02:18 dist-test-slave-bjsf krb5kdc[786](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765832538, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.113.1@KRBTEST.COM for kudu/127.28.113.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.28.113.2@KRBTEST.COM; defaulting to no policy
I20251215 21:02:18.978443 873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:02:18.967910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.113.1:56827 (local address 127.28.113.62:44239)
1215 21:02:18.968060 (+ 150us) server_negotiation.cc:207] Beginning negotiation
1215 21:02:18.968063 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1215 21:02:18.968638 (+ 575us) server_negotiation.cc:408] Connection header received
1215 21:02:18.969455 (+ 817us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1215 21:02:18.969459 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1215 21:02:18.969517 (+ 58us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1215 21:02:18.969616 (+ 99us) server_negotiation.cc:227] Negotiated authn=SASL
1215 21:02:18.971363 (+ 1747us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:02:18.972092 (+ 729us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1215 21:02:18.972772 (+ 680us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:02:18.972953 (+ 181us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:02:18.975144 (+ 2191us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1215 21:02:18.975166 (+ 22us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1215 21:02:18.975170 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1215 21:02:18.975198 (+ 28us) server_negotiation.cc:923] calling sasl_server_start()
1215 21:02:18.976743 (+ 1545us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1215 21:02:18.977167 (+ 424us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1215 21:02:18.977171 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1215 21:02:18.977173 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1215 21:02:18.977225 (+ 52us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1215 21:02:18.977454 (+ 229us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1215 21:02:18.977458 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1215 21:02:18.977460 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1215 21:02:18.977651 (+ 191us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1215 21:02:18.977756 (+ 105us) server_negotiation.cc:1036] Waiting for connection context
1215 21:02:18.978155 (+ 399us) server_negotiation.cc:300] Negotiation successful
1215 21:02:18.978279 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20251215 21:02:18.978909 1029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:02:18.968123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.113.62:44239 (local address 127.28.113.1:56827)
1215 21:02:18.968508 (+ 385us) negotiation.cc:107] Waiting for socket to connect
1215 21:02:18.968541 (+ 33us) client_negotiation.cc:174] Beginning negotiation
1215 21:02:18.969260 (+ 719us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1215 21:02:18.969783 (+ 523us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1215 21:02:18.969791 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
1215 21:02:18.970242 (+ 451us) client_negotiation.cc:189] Negotiated authn=SASL
1215 21:02:18.971217 (+ 975us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:02:18.971230 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:02:18.972233 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1215 21:02:18.972236 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1215 21:02:18.972660 (+ 424us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:02:18.972668 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:02:18.972878 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:02:18.973442 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1215 21:02:18.973463 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
1215 21:02:18.974999 (+ 1536us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1215 21:02:18.976850 (+ 1851us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1215 21:02:18.976855 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1215 21:02:18.976862 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
1215 21:02:18.977076 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1215 21:02:18.977305 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1215 21:02:18.977307 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1215 21:02:18.977309 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1215 21:02:18.977378 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1215 21:02:18.977740 (+ 362us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1215 21:02:18.977744 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1215 21:02:18.977929 (+ 185us) client_negotiation.cc:769] Sending connection context
1215 21:02:18.978089 (+ 160us) client_negotiation.cc:240] Negotiation successful
1215 21:02:18.978291 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":94,"threads_started":1}
I20251215 21:02:18.979929 1027 heartbeater.cc:344] Connected to a master server at 127.28.113.62:44239
I20251215 21:02:18.980223 1027 heartbeater.cc:461] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.113.2@KRBTEST.COM" created.
I20251215 21:02:18.980782 1027 heartbeater.cc:507] Master 127.28.113.62:44239 requested a full tablet report, sending...
I20251215 21:02:18.982091 834 ts_manager.cc:194] Registered new tserver with Master: f3f6b1b3d4c04c6faa72556352c6844a (127.28.113.1:39309)
I20251215 21:02:18.983160 834 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.113.1@KRBTEST.COM'} at 127.28.113.1:56827
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.113.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765832506145157-29124-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.28.113.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765832506145157-29124-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.28.113.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.28.113.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.28.113.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765832506145157-29124-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.28.113.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765832506145157-29124-0/krb5kdc/kudu.keytab.
I20251215 21:02:19.009135 29124 external_mini_cluster.cc:1366] Running /tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu
/tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765832506145157-29124-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765832506145157-29124-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765832506145157-29124-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765832506145157-29124-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.113.2:0
--local_ip_for_outbound_sockets=127.28.113.2
--webserver_interface=127.28.113.2
--webserver_port=0
--tserver_master_addrs=127.28.113.62:44239
--keytab_file=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765832506145157-29124-0/krb5kdc/kudu.keytab
--principal=kudu/127.28.113.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.28.113.20:32789
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765832506145157-29124-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765832506145157-29124-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765832506145157-29124-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251215 21:02:19.109179 1034 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251215 21:02:19.109375 1034 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251215 21:02:19.109448 1034 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251215 21:02:19.112546 1034 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251215 21:02:19.112600 1034 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251215 21:02:19.112687 1034 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.113.2
I20251215 21:02:19.116786 1034 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.113.20:32789
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765832506145157-29124-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765832506145157-29124-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765832506145157-29124-0/krb5kdc/kudu.keytab
--principal=kudu/127.28.113.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.113.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765832506145157-29124-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.113.2
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.28.113.62:44239
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1034
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.113.2
--log_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765832506145157-29124-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision dc741732bb6a40213e11a13869fad06a49a209a3
build type DEBUG
built by None at 15 Dec 2025 20:43:14 UTC on 24a791456cd2
build id 9530
I20251215 21:02:19.117712 1034 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251215 21:02:19.118449 1034 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251215 21:02:19.124504 1040 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251215 21:02:19.124506 1042 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251215 21:02:19.124667 1039 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251215 21:02:19.124588 1034 server_base.cc:1047] running on GCE node
I20251215 21:02:19.125006 1034 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251215 21:02:19.125483 1034 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251215 21:02:19.126641 1034 hybrid_clock.cc:648] HybridClock initialized: now 1765832539126618 us; error 35 us; skew 500 ppm
Dec 15 21:02:19 dist-test-slave-bjsf krb5kdc[786](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765832539, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.113.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251215 21:02:19.128999 1034 init.cc:377] Logged in from keytab as kudu/127.28.113.2@KRBTEST.COM (short username kudu)
I20251215 21:02:19.129863 1034 webserver.cc:492] Webserver started at http://127.28.113.2:45395/ using document root <none> and password file <none>
I20251215 21:02:19.130318 1034 fs_manager.cc:362] Metadata directory not provided
I20251215 21:02:19.130363 1034 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251215 21:02:19.130496 1034 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251215 21:02:19.132030 1034 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765832506145157-29124-0/minicluster-data/ts-1/data/instance:
uuid: "352f178432704842b58e49df2d822366"
format_stamp: "Formatted at 2025-12-15 21:02:19 on dist-test-slave-bjsf"
server_key: "374cd7f0d0d564098e9508178319f40f"
server_key_iv: "2875d57430b25e6d098aca3972dbdc79"
server_key_version: "encryptionkey@0"
I20251215 21:02:19.132417 1034 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765832506145157-29124-0/minicluster-data/ts-1/wal/instance:
uuid: "352f178432704842b58e49df2d822366"
format_stamp: "Formatted at 2025-12-15 21:02:19 on dist-test-slave-bjsf"
server_key: "374cd7f0d0d564098e9508178319f40f"
server_key_iv: "2875d57430b25e6d098aca3972dbdc79"
server_key_version: "encryptionkey@0"
I20251215 21:02:19.135432 1034 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20251215 21:02:19.137315 1049 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251215 21:02:19.138181 1034 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20251215 21:02:19.138265 1034 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765832506145157-29124-0/minicluster-data/ts-1/data,/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765832506145157-29124-0/minicluster-data/ts-1/wal
uuid: "352f178432704842b58e49df2d822366"
format_stamp: "Formatted at 2025-12-15 21:02:19 on dist-test-slave-bjsf"
server_key: "374cd7f0d0d564098e9508178319f40f"
server_key_iv: "2875d57430b25e6d098aca3972dbdc79"
server_key_version: "encryptionkey@0"
I20251215 21:02:19.138342 1034 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765832506145157-29124-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765832506145157-29124-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765832506145157-29124-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251215 21:02:19.165445 1034 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251215 21:02:19.168115 1034 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251215 21:02:19.168265 1034 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251215 21:02:19.168751 1034 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251215 21:02:19.169557 1034 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251215 21:02:19.169601 1034 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251215 21:02:19.169637 1034 ts_tablet_manager.cc:616] Registered 0 tablets
I20251215 21:02:19.169664 1034 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251215 21:02:19.178046 1034 rpc_server.cc:307] RPC server started. Bound to: 127.28.113.2:37291
I20251215 21:02:19.178088 1163 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.28.113.2:37291 every 8 connection(s)
I20251215 21:02:19.178884 1034 server_base.cc:1179] Dumped server information to /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765832506145157-29124-0/minicluster-data/ts-1/data/info.pb
I20251215 21:02:19.183677 29124 external_mini_cluster.cc:1428] Started /tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu as pid 1034
I20251215 21:02:19.183792 29124 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765832506145157-29124-0/minicluster-data/ts-1/wal/instance
I20251215 21:02:19.184078 29124 external_mini_cluster.cc:1468] Setting key 1d66fddafaff4e23a4bf223da933de25
Dec 15 21:02:19 dist-test-slave-bjsf krb5kdc[786](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765832539, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.113.2@KRBTEST.COM for kudu/127.28.113.62@KRBTEST.COM
I20251215 21:02:19.190802 873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:02:19.180521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.113.2:35793 (local address 127.28.113.62:44239)
1215 21:02:19.180695 (+ 174us) server_negotiation.cc:207] Beginning negotiation
1215 21:02:19.180699 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1215 21:02:19.181276 (+ 577us) server_negotiation.cc:408] Connection header received
1215 21:02:19.182085 (+ 809us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1215 21:02:19.182088 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1215 21:02:19.182147 (+ 59us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1215 21:02:19.182216 (+ 69us) server_negotiation.cc:227] Negotiated authn=SASL
1215 21:02:19.183899 (+ 1683us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:02:19.184507 (+ 608us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1215 21:02:19.185200 (+ 693us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:02:19.185364 (+ 164us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:02:19.187576 (+ 2212us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1215 21:02:19.187592 (+ 16us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1215 21:02:19.187596 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1215 21:02:19.187622 (+ 26us) server_negotiation.cc:923] calling sasl_server_start()
1215 21:02:19.189089 (+ 1467us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1215 21:02:19.189514 (+ 425us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1215 21:02:19.189518 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1215 21:02:19.189520 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1215 21:02:19.189571 (+ 51us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1215 21:02:19.189808 (+ 237us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1215 21:02:19.189811 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1215 21:02:19.189814 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
1215 21:02:19.190003 (+ 189us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1215 21:02:19.190104 (+ 101us) server_negotiation.cc:1036] Waiting for connection context
1215 21:02:19.190533 (+ 429us) server_negotiation.cc:300] Negotiation successful
1215 21:02:19.190648 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20251215 21:02:19.191414 1166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:02:19.180759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.113.62:44239 (local address 127.28.113.2:35793)
1215 21:02:19.181149 (+ 390us) negotiation.cc:107] Waiting for socket to connect
1215 21:02:19.181183 (+ 34us) client_negotiation.cc:174] Beginning negotiation
1215 21:02:19.181896 (+ 713us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1215 21:02:19.182377 (+ 481us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1215 21:02:19.182384 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
1215 21:02:19.182759 (+ 375us) client_negotiation.cc:189] Negotiated authn=SASL
1215 21:02:19.183745 (+ 986us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:02:19.183764 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:02:19.184654 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1215 21:02:19.184658 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1215 21:02:19.185077 (+ 419us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:02:19.185087 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:02:19.185279 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:02:19.185827 (+ 548us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1215 21:02:19.185857 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
1215 21:02:19.187431 (+ 1574us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1215 21:02:19.189196 (+ 1765us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1215 21:02:19.189201 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1215 21:02:19.189209 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
1215 21:02:19.189419 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1215 21:02:19.189656 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1215 21:02:19.189659 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1215 21:02:19.189660 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1215 21:02:19.189735 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1215 21:02:19.190092 (+ 357us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1215 21:02:19.190098 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1215 21:02:19.190300 (+ 202us) client_negotiation.cc:769] Sending connection context
1215 21:02:19.190472 (+ 172us) client_negotiation.cc:240] Negotiation successful
1215 21:02:19.190670 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":102,"threads_started":1}
I20251215 21:02:19.192418 1164 heartbeater.cc:344] Connected to a master server at 127.28.113.62:44239
WARNING: no policy specified for kudu/127.28.113.3@KRBTEST.COM; defaulting to no policy
I20251215 21:02:19.192674 1164 heartbeater.cc:461] Registering TS with master...
I20251215 21:02:19.193166 1164 heartbeater.cc:507] Master 127.28.113.62:44239 requested a full tablet report, sending...
I20251215 21:02:19.194118 834 ts_manager.cc:194] Registered new tserver with Master: 352f178432704842b58e49df2d822366 (127.28.113.2:37291)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.113.3@KRBTEST.COM" created.
I20251215 21:02:19.194718 834 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.113.2@KRBTEST.COM'} at 127.28.113.2:35793
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.113.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765832506145157-29124-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.28.113.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765832506145157-29124-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.28.113.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.28.113.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.28.113.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765832506145157-29124-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.28.113.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765832506145157-29124-0/krb5kdc/kudu.keytab.
I20251215 21:02:19.224402 29124 external_mini_cluster.cc:1366] Running /tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu
/tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765832506145157-29124-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765832506145157-29124-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765832506145157-29124-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765832506145157-29124-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.113.3:0
--local_ip_for_outbound_sockets=127.28.113.3
--webserver_interface=127.28.113.3
--webserver_port=0
--tserver_master_addrs=127.28.113.62:44239
--keytab_file=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765832506145157-29124-0/krb5kdc/kudu.keytab
--principal=kudu/127.28.113.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.28.113.20:32789
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765832506145157-29124-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765832506145157-29124-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765832506145157-29124-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251215 21:02:19.324666 1171 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251215 21:02:19.324873 1171 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251215 21:02:19.324946 1171 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251215 21:02:19.328094 1171 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251215 21:02:19.328150 1171 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251215 21:02:19.328240 1171 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.113.3
I20251215 21:02:19.332190 1171 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.113.20:32789
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765832506145157-29124-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765832506145157-29124-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765832506145157-29124-0/krb5kdc/kudu.keytab
--principal=kudu/127.28.113.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.113.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765832506145157-29124-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.113.3
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.28.113.62:44239
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1171
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.113.3
--log_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765832506145157-29124-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision dc741732bb6a40213e11a13869fad06a49a209a3
build type DEBUG
built by None at 15 Dec 2025 20:43:14 UTC on 24a791456cd2
build id 9530
I20251215 21:02:19.333145 1171 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251215 21:02:19.333947 1171 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251215 21:02:19.339999 1176 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251215 21:02:19.340057 1177 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251215 21:02:19.339998 1179 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251215 21:02:19.340227 1171 server_base.cc:1047] running on GCE node
I20251215 21:02:19.340631 1171 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251215 21:02:19.341114 1171 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251215 21:02:19.342247 1171 hybrid_clock.cc:648] HybridClock initialized: now 1765832539342236 us; error 24 us; skew 500 ppm
Dec 15 21:02:19 dist-test-slave-bjsf krb5kdc[786](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765832539, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.113.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251215 21:02:19.344590 1171 init.cc:377] Logged in from keytab as kudu/127.28.113.3@KRBTEST.COM (short username kudu)
I20251215 21:02:19.345485 1171 webserver.cc:492] Webserver started at http://127.28.113.3:46069/ using document root <none> and password file <none>
I20251215 21:02:19.345964 1171 fs_manager.cc:362] Metadata directory not provided
I20251215 21:02:19.346007 1171 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251215 21:02:19.346145 1171 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251215 21:02:19.347658 1171 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765832506145157-29124-0/minicluster-data/ts-2/data/instance:
uuid: "c44c9bd3289b400eb4076a55cda12e13"
format_stamp: "Formatted at 2025-12-15 21:02:19 on dist-test-slave-bjsf"
server_key: "40493b205fab74e6a7d7063825228b08"
server_key_iv: "7d254d3bc4b7417dff13062c6112665e"
server_key_version: "encryptionkey@0"
I20251215 21:02:19.348090 1171 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765832506145157-29124-0/minicluster-data/ts-2/wal/instance:
uuid: "c44c9bd3289b400eb4076a55cda12e13"
format_stamp: "Formatted at 2025-12-15 21:02:19 on dist-test-slave-bjsf"
server_key: "40493b205fab74e6a7d7063825228b08"
server_key_iv: "7d254d3bc4b7417dff13062c6112665e"
server_key_version: "encryptionkey@0"
I20251215 21:02:19.351033 1171 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20251215 21:02:19.352959 1186 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251215 21:02:19.353833 1171 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20251215 21:02:19.353914 1171 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765832506145157-29124-0/minicluster-data/ts-2/data,/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765832506145157-29124-0/minicluster-data/ts-2/wal
uuid: "c44c9bd3289b400eb4076a55cda12e13"
format_stamp: "Formatted at 2025-12-15 21:02:19 on dist-test-slave-bjsf"
server_key: "40493b205fab74e6a7d7063825228b08"
server_key_iv: "7d254d3bc4b7417dff13062c6112665e"
server_key_version: "encryptionkey@0"
I20251215 21:02:19.353987 1171 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765832506145157-29124-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765832506145157-29124-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765832506145157-29124-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251215 21:02:19.364562 1171 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251215 21:02:19.366812 1171 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251215 21:02:19.366948 1171 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251215 21:02:19.367370 1171 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251215 21:02:19.368163 1171 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251215 21:02:19.368207 1171 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251215 21:02:19.368243 1171 ts_tablet_manager.cc:616] Registered 0 tablets
I20251215 21:02:19.368263 1171 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251215 21:02:19.376665 1171 rpc_server.cc:307] RPC server started. Bound to: 127.28.113.3:44771
I20251215 21:02:19.376701 1299 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.28.113.3:44771 every 8 connection(s)
I20251215 21:02:19.377511 1171 server_base.cc:1179] Dumped server information to /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765832506145157-29124-0/minicluster-data/ts-2/data/info.pb
I20251215 21:02:19.379041 29124 external_mini_cluster.cc:1428] Started /tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu as pid 1171
I20251215 21:02:19.379139 29124 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765832506145157-29124-0/minicluster-data/ts-2/wal/instance
I20251215 21:02:19.379364 29124 external_mini_cluster.cc:1468] Setting key 6a63110a75815ecc8dfd2c120f08a122
Dec 15 21:02:19 dist-test-slave-bjsf krb5kdc[786](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765832539, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.113.3@KRBTEST.COM for kudu/127.28.113.62@KRBTEST.COM
I20251215 21:02:19.388928 873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:02:19.379863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.113.3:50457 (local address 127.28.113.62:44239)
1215 21:02:19.380072 (+ 209us) server_negotiation.cc:207] Beginning negotiation
1215 21:02:19.380077 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1215 21:02:19.380626 (+ 549us) server_negotiation.cc:408] Connection header received
1215 21:02:19.381559 (+ 933us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1215 21:02:19.381562 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1215 21:02:19.381608 (+ 46us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1215 21:02:19.381670 (+ 62us) server_negotiation.cc:227] Negotiated authn=SASL
1215 21:02:19.383160 (+ 1490us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:02:19.383637 (+ 477us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1215 21:02:19.384224 (+ 587us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:02:19.384349 (+ 125us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:02:19.386256 (+ 1907us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1215 21:02:19.386271 (+ 15us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1215 21:02:19.386273 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1215 21:02:19.386296 (+ 23us) server_negotiation.cc:923] calling sasl_server_start()
1215 21:02:19.387479 (+ 1183us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1215 21:02:19.387875 (+ 396us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1215 21:02:19.387877 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1215 21:02:19.387879 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1215 21:02:19.387912 (+ 33us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1215 21:02:19.388143 (+ 231us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1215 21:02:19.388146 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1215 21:02:19.388147 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1215 21:02:19.388277 (+ 130us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1215 21:02:19.388343 (+ 66us) server_negotiation.cc:1036] Waiting for connection context
1215 21:02:19.388730 (+ 387us) server_negotiation.cc:300] Negotiation successful
1215 21:02:19.388815 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20251215 21:02:19.389489 1302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:02:19.379460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.113.62:44239 (local address 127.28.113.3:50457)
1215 21:02:19.380498 (+ 1038us) negotiation.cc:107] Waiting for socket to connect
1215 21:02:19.380541 (+ 43us) client_negotiation.cc:174] Beginning negotiation
1215 21:02:19.381387 (+ 846us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1215 21:02:19.381798 (+ 411us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1215 21:02:19.381805 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
1215 21:02:19.382154 (+ 349us) client_negotiation.cc:189] Negotiated authn=SASL
1215 21:02:19.383054 (+ 900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:02:19.383068 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:02:19.383729 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1215 21:02:19.383732 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1215 21:02:19.384146 (+ 414us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:02:19.384153 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:02:19.384305 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:02:19.384834 (+ 529us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1215 21:02:19.384856 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
1215 21:02:19.386137 (+ 1281us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1215 21:02:19.387573 (+ 1436us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1215 21:02:19.387577 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1215 21:02:19.387585 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
1215 21:02:19.387803 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1215 21:02:19.387993 (+ 190us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1215 21:02:19.387996 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1215 21:02:19.387998 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1215 21:02:19.388075 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1215 21:02:19.388342 (+ 267us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1215 21:02:19.388347 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1215 21:02:19.388532 (+ 185us) client_negotiation.cc:769] Sending connection context
1215 21:02:19.388697 (+ 165us) client_negotiation.cc:240] Negotiation successful
1215 21:02:19.388888 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":820,"thread_start_us":106,"threads_started":1}
I20251215 21:02:19.390390 1300 heartbeater.cc:344] Connected to a master server at 127.28.113.62:44239
I20251215 21:02:19.390594 1300 heartbeater.cc:461] Registering TS with master...
I20251215 21:02:19.391049 1300 heartbeater.cc:507] Master 127.28.113.62:44239 requested a full tablet report, sending...
I20251215 21:02:19.391978 834 ts_manager.cc:194] Registered new tserver with Master: c44c9bd3289b400eb4076a55cda12e13 (127.28.113.3:44771)
I20251215 21:02:19.392596 834 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.113.3@KRBTEST.COM'} at 127.28.113.3:50457
I20251215 21:02:19.392830 29124 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
Dec 15 21:02:19 dist-test-slave-bjsf krb5kdc[786](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765832539, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Dec 15 21:02:19 dist-test-slave-bjsf krb5kdc[786](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765832539, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.113.1@KRBTEST.COM
I20251215 21:02:19.414659 1307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:02:19.406792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41074 (local address 127.28.113.1:39309)
1215 21:02:19.407100 (+ 308us) server_negotiation.cc:207] Beginning negotiation
1215 21:02:19.407106 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1215 21:02:19.407128 (+ 22us) server_negotiation.cc:408] Connection header received
1215 21:02:19.407180 (+ 52us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1215 21:02:19.407184 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1215 21:02:19.407344 (+ 160us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1215 21:02:19.407454 (+ 110us) server_negotiation.cc:227] Negotiated authn=SASL
1215 21:02:19.408227 (+ 773us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:02:19.408942 (+ 715us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1215 21:02:19.409501 (+ 559us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:02:19.409669 (+ 168us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:02:19.411519 (+ 1850us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1215 21:02:19.411541 (+ 22us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1215 21:02:19.411553 (+ 12us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1215 21:02:19.411574 (+ 21us) server_negotiation.cc:923] calling sasl_server_start()
1215 21:02:19.413253 (+ 1679us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1215 21:02:19.413713 (+ 460us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1215 21:02:19.413719 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1215 21:02:19.413721 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1215 21:02:19.413776 (+ 55us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1215 21:02:19.413929 (+ 153us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1215 21:02:19.413931 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1215 21:02:19.413933 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1215 21:02:19.414106 (+ 173us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1215 21:02:19.414254 (+ 148us) server_negotiation.cc:1036] Waiting for connection context
1215 21:02:19.414359 (+ 105us) server_negotiation.cc:300] Negotiation successful
1215 21:02:19.414480 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":212,"thread_start_us":115,"threads_started":1}
Dec 15 21:02:19 dist-test-slave-bjsf krb5kdc[786](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765832539, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.113.62@KRBTEST.COM
I20251215 21:02:19.423570 873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:02:19.417220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49234 (local address 127.28.113.62:44239)
1215 21:02:19.417328 (+ 108us) server_negotiation.cc:207] Beginning negotiation
1215 21:02:19.417331 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1215 21:02:19.417345 (+ 14us) server_negotiation.cc:408] Connection header received
1215 21:02:19.417419 (+ 74us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1215 21:02:19.417422 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1215 21:02:19.417463 (+ 41us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1215 21:02:19.417525 (+ 62us) server_negotiation.cc:227] Negotiated authn=SASL
1215 21:02:19.418149 (+ 624us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:02:19.418619 (+ 470us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1215 21:02:19.419136 (+ 517us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:02:19.419252 (+ 116us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:02:19.421216 (+ 1964us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1215 21:02:19.421236 (+ 20us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1215 21:02:19.421238 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1215 21:02:19.421265 (+ 27us) server_negotiation.cc:923] calling sasl_server_start()
1215 21:02:19.422444 (+ 1179us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1215 21:02:19.422769 (+ 325us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1215 21:02:19.422772 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1215 21:02:19.422773 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1215 21:02:19.422814 (+ 41us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1215 21:02:19.422969 (+ 155us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1215 21:02:19.422971 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1215 21:02:19.422972 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1215 21:02:19.423138 (+ 166us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1215 21:02:19.423241 (+ 103us) server_negotiation.cc:1036] Waiting for connection context
1215 21:02:19.423358 (+ 117us) server_negotiation.cc:300] Negotiation successful
1215 21:02:19.423451 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
W20251215 21:02:19.424018 834 server_base.cc:1129] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49234
I20251215 21:02:19.424571 29124 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu with pid 898
I20251215 21:02:19.429657 29124 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu with pid 1034
I20251215 21:02:19.434149 29124 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu with pid 1171
I20251215 21:02:19.438598 29124 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu with pid 802
2025-12-15T21:02:19Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (2834 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765832506145157-29124-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Dec 15 21:02:19 dist-test-slave-bjsf krb5kdc[1312](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Dec 15 21:02:19 dist-test-slave-bjsf krb5kdc[1312](info): set up 2 sockets
Dec 15 21:02:19 dist-test-slave-bjsf krb5kdc[1312](info): commencing operation
krb5kdc: starting...
W20251215 21:02:21.276203 29124 mini_kdc.cc:121] Time spent starting KDC: real 1.818s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765832506145157-29124-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765832506145157-29124-0/krb5kdc/test-user.keytab.
Dec 15 21:02:21 dist-test-slave-bjsf krb5kdc[1312](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765832541, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-12-15T21:02:21Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-12-15T21:02:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.113.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.113.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.113.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765832506145157-29124-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.28.113.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765832506145157-29124-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.28.113.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.28.113.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.28.113.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765832506145157-29124-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.28.113.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765832506145157-29124-0/krb5kdc/kudu.keytab.
I20251215 21:02:21.398417 29124 external_mini_cluster.cc:1366] Running /tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu
/tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765832506145157-29124-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765832506145157-29124-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765832506145157-29124-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765832506145157-29124-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.113.62:33771
--webserver_interface=127.28.113.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765832506145157-29124-0/krb5kdc/kudu.keytab
--principal=kudu/127.28.113.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.28.113.20:36365
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.113.62:33771
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765832506145157-29124-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765832506145157-29124-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765832506145157-29124-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251215 21:02:21.502795 1328 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251215 21:02:21.503015 1328 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251215 21:02:21.503075 1328 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251215 21:02:21.506268 1328 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251215 21:02:21.506333 1328 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251215 21:02:21.506356 1328 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251215 21:02:21.506372 1328 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251215 21:02:21.506392 1328 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251215 21:02:21.510589 1328 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.113.20:36365
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765832506145157-29124-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765832506145157-29124-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.113.62:33771
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765832506145157-29124-0/krb5kdc/kudu.keytab
--principal=kudu/127.28.113.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.113.62:33771
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765832506145157-29124-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.113.62
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1328
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765832506145157-29124-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision dc741732bb6a40213e11a13869fad06a49a209a3
build type DEBUG
built by None at 15 Dec 2025 20:43:14 UTC on 24a791456cd2
build id 9530
I20251215 21:02:21.511600 1328 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251215 21:02:21.512439 1328 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251215 21:02:21.517777 1334 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251215 21:02:21.517846 1333 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251215 21:02:21.517859 1328 server_base.cc:1047] running on GCE node
W20251215 21:02:21.517777 1336 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251215 21:02:21.518528 1328 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251215 21:02:21.519449 1328 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251215 21:02:21.520593 1328 hybrid_clock.cc:648] HybridClock initialized: now 1765832541520571 us; error 34 us; skew 500 ppm
Dec 15 21:02:21 dist-test-slave-bjsf krb5kdc[1312](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765832541, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.113.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251215 21:02:21.523047 1328 init.cc:377] Logged in from keytab as kudu/127.28.113.62@KRBTEST.COM (short username kudu)
I20251215 21:02:21.523697 1328 fs_manager.cc:362] Metadata directory not provided
I20251215 21:02:21.523749 1328 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251215 21:02:21.523910 1328 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251215 21:02:21.525532 1328 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765832506145157-29124-0/minicluster-data/master-0/data/instance:
uuid: "fd6bc7e478b64e228ee5f45174d6b417"
format_stamp: "Formatted at 2025-12-15 21:02:21 on dist-test-slave-bjsf"
server_key: "37dd20d8a13e0cc36827892a3f1bdbe3"
server_key_iv: "1ac1251611b830e5384da4320bc86c69"
server_key_version: "encryptionkey@0"
I20251215 21:02:21.525975 1328 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765832506145157-29124-0/minicluster-data/master-0/wal/instance:
uuid: "fd6bc7e478b64e228ee5f45174d6b417"
format_stamp: "Formatted at 2025-12-15 21:02:21 on dist-test-slave-bjsf"
server_key: "37dd20d8a13e0cc36827892a3f1bdbe3"
server_key_iv: "1ac1251611b830e5384da4320bc86c69"
server_key_version: "encryptionkey@0"
I20251215 21:02:21.529131 1328 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20251215 21:02:21.531350 1342 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251215 21:02:21.532478 1328 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20251215 21:02:21.532593 1328 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765832506145157-29124-0/minicluster-data/master-0/data,/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765832506145157-29124-0/minicluster-data/master-0/wal
uuid: "fd6bc7e478b64e228ee5f45174d6b417"
format_stamp: "Formatted at 2025-12-15 21:02:21 on dist-test-slave-bjsf"
server_key: "37dd20d8a13e0cc36827892a3f1bdbe3"
server_key_iv: "1ac1251611b830e5384da4320bc86c69"
server_key_version: "encryptionkey@0"
I20251215 21:02:21.532707 1328 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765832506145157-29124-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765832506145157-29124-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765832506145157-29124-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251215 21:02:21.556509 1328 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251215 21:02:21.559660 1328 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251215 21:02:21.559847 1328 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251215 21:02:21.567292 1328 rpc_server.cc:307] RPC server started. Bound to: 127.28.113.62:33771
I20251215 21:02:21.567277 1394 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.28.113.62:33771 every 8 connection(s)
I20251215 21:02:21.567999 1328 server_base.cc:1179] Dumped server information to /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765832506145157-29124-0/minicluster-data/master-0/data/info.pb
I20251215 21:02:21.570775 1395 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 21:02:21.573839 29124 external_mini_cluster.cc:1428] Started /tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu as pid 1328
I20251215 21:02:21.573974 29124 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765832506145157-29124-0/minicluster-data/master-0/wal/instance
I20251215 21:02:21.574213 29124 external_mini_cluster.cc:1468] Setting key 1df70af28b1426e9420da3001531f1c9
I20251215 21:02:21.576855 1395 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fd6bc7e478b64e228ee5f45174d6b417: Bootstrap starting.
I20251215 21:02:21.579034 1395 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fd6bc7e478b64e228ee5f45174d6b417: Neither blocks nor log segments found. Creating new log.
I20251215 21:02:21.579733 1395 log.cc:826] T 00000000000000000000000000000000 P fd6bc7e478b64e228ee5f45174d6b417: Log is configured to *not* fsync() on all Append() calls
Dec 15 21:02:21 dist-test-slave-bjsf krb5kdc[1312](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765832541, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.113.62@KRBTEST.COM
I20251215 21:02:21.581815 1395 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fd6bc7e478b64e228ee5f45174d6b417: No bootstrap required, opened a new log
I20251215 21:02:21.584384 1395 raft_consensus.cc:359] T 00000000000000000000000000000000 P fd6bc7e478b64e228ee5f45174d6b417 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd6bc7e478b64e228ee5f45174d6b417" member_type: VOTER last_known_addr { host: "127.28.113.62" port: 33771 } }
I20251215 21:02:21.584587 1395 raft_consensus.cc:385] T 00000000000000000000000000000000 P fd6bc7e478b64e228ee5f45174d6b417 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251215 21:02:21.584656 1395 raft_consensus.cc:740] T 00000000000000000000000000000000 P fd6bc7e478b64e228ee5f45174d6b417 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd6bc7e478b64e228ee5f45174d6b417, State: Initialized, Role: FOLLOWER
I20251215 21:02:21.585150 1395 consensus_queue.cc:260] T 00000000000000000000000000000000 P fd6bc7e478b64e228ee5f45174d6b417 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd6bc7e478b64e228ee5f45174d6b417" member_type: VOTER last_known_addr { host: "127.28.113.62" port: 33771 } }
I20251215 21:02:21.585301 1395 raft_consensus.cc:399] T 00000000000000000000000000000000 P fd6bc7e478b64e228ee5f45174d6b417 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251215 21:02:21.585361 1395 raft_consensus.cc:493] T 00000000000000000000000000000000 P fd6bc7e478b64e228ee5f45174d6b417 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251215 21:02:21.585443 1395 raft_consensus.cc:3060] T 00000000000000000000000000000000 P fd6bc7e478b64e228ee5f45174d6b417 [term 0 FOLLOWER]: Advancing to term 1
I20251215 21:02:21.586366 1395 raft_consensus.cc:515] T 00000000000000000000000000000000 P fd6bc7e478b64e228ee5f45174d6b417 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd6bc7e478b64e228ee5f45174d6b417" member_type: VOTER last_known_addr { host: "127.28.113.62" port: 33771 } }
I20251215 21:02:21.586717 1395 leader_election.cc:304] T 00000000000000000000000000000000 P fd6bc7e478b64e228ee5f45174d6b417 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fd6bc7e478b64e228ee5f45174d6b417; no voters:
I20251215 21:02:21.586999 1395 leader_election.cc:290] T 00000000000000000000000000000000 P fd6bc7e478b64e228ee5f45174d6b417 [CANDIDATE]: Term 1 election: Requested vote from peers
I20251215 21:02:21.587098 1400 raft_consensus.cc:2804] T 00000000000000000000000000000000 P fd6bc7e478b64e228ee5f45174d6b417 [term 1 FOLLOWER]: Leader election won for term 1
I20251215 21:02:21.587251 1400 raft_consensus.cc:697] T 00000000000000000000000000000000 P fd6bc7e478b64e228ee5f45174d6b417 [term 1 LEADER]: Becoming Leader. State: Replica: fd6bc7e478b64e228ee5f45174d6b417, State: Running, Role: LEADER
I20251215 21:02:21.587275 1398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:02:21.575532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53664 (local address 127.28.113.62:33771)
1215 21:02:21.576030 (+ 498us) server_negotiation.cc:207] Beginning negotiation
1215 21:02:21.576040 (+ 10us) server_negotiation.cc:400] Waiting for connection header
1215 21:02:21.576075 (+ 35us) server_negotiation.cc:408] Connection header received
1215 21:02:21.576590 (+ 515us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1215 21:02:21.576605 (+ 15us) server_negotiation.cc:462] Received NEGOTIATE request from client
1215 21:02:21.576912 (+ 307us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1215 21:02:21.577230 (+ 318us) server_negotiation.cc:227] Negotiated authn=SASL
1215 21:02:21.578136 (+ 906us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:02:21.579007 (+ 871us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1215 21:02:21.579654 (+ 647us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:02:21.579938 (+ 284us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:02:21.582171 (+ 2233us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1215 21:02:21.582191 (+ 20us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1215 21:02:21.582203 (+ 12us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1215 21:02:21.582239 (+ 36us) server_negotiation.cc:923] calling sasl_server_start()
1215 21:02:21.583945 (+ 1706us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1215 21:02:21.584391 (+ 446us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1215 21:02:21.584396 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1215 21:02:21.584401 (+ 5us) server_negotiation.cc:968] calling sasl_server_step()
1215 21:02:21.584479 (+ 78us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1215 21:02:21.584713 (+ 234us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1215 21:02:21.584717 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1215 21:02:21.584719 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1215 21:02:21.585193 (+ 474us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1215 21:02:21.585403 (+ 210us) server_negotiation.cc:1036] Waiting for connection context
1215 21:02:21.586069 (+ 666us) server_negotiation.cc:300] Negotiation successful
1215 21:02:21.586298 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":313,"thread_start_us":159,"threads_started":1}
I20251215 21:02:21.588165 1400 consensus_queue.cc:237] T 00000000000000000000000000000000 P fd6bc7e478b64e228ee5f45174d6b417 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd6bc7e478b64e228ee5f45174d6b417" member_type: VOTER last_known_addr { host: "127.28.113.62" port: 33771 } }
I20251215 21:02:21.588207 1395 sys_catalog.cc:565] T 00000000000000000000000000000000 P fd6bc7e478b64e228ee5f45174d6b417 [sys.catalog]: configured and running, proceeding with master startup.
I20251215 21:02:21.589249 1400 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd6bc7e478b64e228ee5f45174d6b417 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fd6bc7e478b64e228ee5f45174d6b417. Latest consensus state: current_term: 1 leader_uuid: "fd6bc7e478b64e228ee5f45174d6b417" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd6bc7e478b64e228ee5f45174d6b417" member_type: VOTER last_known_addr { host: "127.28.113.62" port: 33771 } } }
I20251215 21:02:21.589375 1400 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd6bc7e478b64e228ee5f45174d6b417 [sys.catalog]: This master's current role is: LEADER
I20251215 21:02:21.589869 1409 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251215 21:02:21.589985 1401 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd6bc7e478b64e228ee5f45174d6b417 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fd6bc7e478b64e228ee5f45174d6b417" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd6bc7e478b64e228ee5f45174d6b417" member_type: VOTER last_known_addr { host: "127.28.113.62" port: 33771 } } }
I20251215 21:02:21.590086 1401 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd6bc7e478b64e228ee5f45174d6b417 [sys.catalog]: This master's current role is: LEADER
I20251215 21:02:21.592891 1409 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251215 21:02:21.598098 1409 catalog_manager.cc:1357] Generated new cluster ID: 70c21bd3a90b45f5a5ba316fbae47afe
I20251215 21:02:21.598194 1409 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251215 21:02:21.617055 1409 catalog_manager.cc:1380] Generated new certificate authority record
I20251215 21:02:21.618017 1409 catalog_manager.cc:1514] Loading token signing keys...
I20251215 21:02:21.629885 1409 catalog_manager.cc:6027] T 00000000000000000000000000000000 P fd6bc7e478b64e228ee5f45174d6b417: Generated new TSK 0
I20251215 21:02:21.630654 1409 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.113.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.113.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.113.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765832506145157-29124-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.28.113.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765832506145157-29124-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.28.113.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.28.113.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.28.113.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765832506145157-29124-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.28.113.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765832506145157-29124-0/krb5kdc/kudu.keytab.
I20251215 21:02:21.674710 29124 external_mini_cluster.cc:1366] Running /tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu
/tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765832506145157-29124-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765832506145157-29124-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765832506145157-29124-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765832506145157-29124-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.113.1:0
--local_ip_for_outbound_sockets=127.28.113.1
--webserver_interface=127.28.113.1
--webserver_port=0
--tserver_master_addrs=127.28.113.62:33771
--keytab_file=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765832506145157-29124-0/krb5kdc/kudu.keytab
--principal=kudu/127.28.113.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.28.113.20:36365
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765832506145157-29124-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765832506145157-29124-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765832506145157-29124-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251215 21:02:21.782629 1423 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251215 21:02:21.782866 1423 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251215 21:02:21.782948 1423 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251215 21:02:21.786160 1423 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251215 21:02:21.786235 1423 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251215 21:02:21.786346 1423 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.113.1
I20251215 21:02:21.790606 1423 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.113.20:36365
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765832506145157-29124-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765832506145157-29124-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765832506145157-29124-0/krb5kdc/kudu.keytab
--principal=kudu/127.28.113.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.113.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765832506145157-29124-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.113.1
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.28.113.62:33771
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1423
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.113.1
--log_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765832506145157-29124-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision dc741732bb6a40213e11a13869fad06a49a209a3
build type DEBUG
built by None at 15 Dec 2025 20:43:14 UTC on 24a791456cd2
build id 9530
I20251215 21:02:21.791654 1423 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251215 21:02:21.792502 1423 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251215 21:02:21.798394 1431 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251215 21:02:21.798395 1428 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251215 21:02:21.798390 1429 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251215 21:02:21.798686 1423 server_base.cc:1047] running on GCE node
I20251215 21:02:21.799031 1423 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251215 21:02:21.799539 1423 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251215 21:02:21.800679 1423 hybrid_clock.cc:648] HybridClock initialized: now 1765832541800669 us; error 27 us; skew 500 ppm
Dec 15 21:02:21 dist-test-slave-bjsf krb5kdc[1312](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765832541, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.113.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251215 21:02:21.803364 1423 init.cc:377] Logged in from keytab as kudu/127.28.113.1@KRBTEST.COM (short username kudu)
I20251215 21:02:21.804086 1423 fs_manager.cc:362] Metadata directory not provided
I20251215 21:02:21.804150 1423 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251215 21:02:21.804332 1423 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251215 21:02:21.805986 1423 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765832506145157-29124-0/minicluster-data/ts-0/data/instance:
uuid: "fd27a28d3a7e461a9bf9b94258edc38a"
format_stamp: "Formatted at 2025-12-15 21:02:21 on dist-test-slave-bjsf"
server_key: "51b6bae19aeba799a4900699584349b6"
server_key_iv: "f4046090b92615f49ad6304db23f8bd2"
server_key_version: "encryptionkey@0"
I20251215 21:02:21.806430 1423 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765832506145157-29124-0/minicluster-data/ts-0/wal/instance:
uuid: "fd27a28d3a7e461a9bf9b94258edc38a"
format_stamp: "Formatted at 2025-12-15 21:02:21 on dist-test-slave-bjsf"
server_key: "51b6bae19aeba799a4900699584349b6"
server_key_iv: "f4046090b92615f49ad6304db23f8bd2"
server_key_version: "encryptionkey@0"
I20251215 21:02:21.809692 1423 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20251215 21:02:21.811851 1437 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251215 21:02:21.812845 1423 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20251215 21:02:21.812960 1423 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765832506145157-29124-0/minicluster-data/ts-0/data,/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765832506145157-29124-0/minicluster-data/ts-0/wal
uuid: "fd27a28d3a7e461a9bf9b94258edc38a"
format_stamp: "Formatted at 2025-12-15 21:02:21 on dist-test-slave-bjsf"
server_key: "51b6bae19aeba799a4900699584349b6"
server_key_iv: "f4046090b92615f49ad6304db23f8bd2"
server_key_version: "encryptionkey@0"
I20251215 21:02:21.813047 1423 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765832506145157-29124-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765832506145157-29124-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765832506145157-29124-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251215 21:02:21.828310 1423 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251215 21:02:21.831223 1423 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251215 21:02:21.831393 1423 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251215 21:02:21.831926 1423 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251215 21:02:21.832805 1423 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251215 21:02:21.832854 1423 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251215 21:02:21.832922 1423 ts_tablet_manager.cc:616] Registered 0 tablets
I20251215 21:02:21.832947 1423 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251215 21:02:21.841526 1423 rpc_server.cc:307] RPC server started. Bound to: 127.28.113.1:35243
I20251215 21:02:21.841554 1550 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.28.113.1:35243 every 8 connection(s)
I20251215 21:02:21.842051 1423 server_base.cc:1179] Dumped server information to /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765832506145157-29124-0/minicluster-data/ts-0/data/info.pb
Dec 15 21:02:21 dist-test-slave-bjsf krb5kdc[1312](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765832541, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.113.1@KRBTEST.COM for kudu/127.28.113.62@KRBTEST.COM
I20251215 21:02:21.850369 29124 external_mini_cluster.cc:1428] Started /tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu as pid 1423
I20251215 21:02:21.850479 29124 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765832506145157-29124-0/minicluster-data/ts-0/wal/instance
I20251215 21:02:21.850700 29124 external_mini_cluster.cc:1468] Setting key 7b9c90cbb0c18db38eba2cb37269639c
I20251215 21:02:21.854038 1398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:02:21.843643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.113.1:34935 (local address 127.28.113.62:33771)
1215 21:02:21.843785 (+ 142us) server_negotiation.cc:207] Beginning negotiation
1215 21:02:21.843789 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1215 21:02:21.844476 (+ 687us) server_negotiation.cc:408] Connection header received
1215 21:02:21.845279 (+ 803us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1215 21:02:21.845282 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1215 21:02:21.845330 (+ 48us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1215 21:02:21.845403 (+ 73us) server_negotiation.cc:227] Negotiated authn=SASL
1215 21:02:21.847011 (+ 1608us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:02:21.847520 (+ 509us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1215 21:02:21.848167 (+ 647us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:02:21.848300 (+ 133us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:02:21.850386 (+ 2086us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1215 21:02:21.850407 (+ 21us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1215 21:02:21.850409 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1215 21:02:21.850434 (+ 25us) server_negotiation.cc:923] calling sasl_server_start()
1215 21:02:21.852264 (+ 1830us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1215 21:02:21.852723 (+ 459us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1215 21:02:21.852726 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1215 21:02:21.852727 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1215 21:02:21.852770 (+ 43us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1215 21:02:21.853028 (+ 258us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1215 21:02:21.853031 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1215 21:02:21.853033 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1215 21:02:21.853181 (+ 148us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1215 21:02:21.853265 (+ 84us) server_negotiation.cc:1036] Waiting for connection context
1215 21:02:21.853749 (+ 484us) server_negotiation.cc:300] Negotiation successful
1215 21:02:21.853877 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20251215 21:02:21.854705 1553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:02:21.843900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.113.62:33771 (local address 127.28.113.1:34935)
1215 21:02:21.844335 (+ 435us) negotiation.cc:107] Waiting for socket to connect
1215 21:02:21.844370 (+ 35us) client_negotiation.cc:174] Beginning negotiation
1215 21:02:21.845103 (+ 733us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1215 21:02:21.845538 (+ 435us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1215 21:02:21.845545 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
1215 21:02:21.845940 (+ 395us) client_negotiation.cc:189] Negotiated authn=SASL
1215 21:02:21.846842 (+ 902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:02:21.846856 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:02:21.847617 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1215 21:02:21.847619 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1215 21:02:21.848074 (+ 455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:02:21.848082 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:02:21.848245 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:02:21.848772 (+ 527us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1215 21:02:21.848796 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
1215 21:02:21.850229 (+ 1433us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1215 21:02:21.852378 (+ 2149us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1215 21:02:21.852384 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1215 21:02:21.852397 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
1215 21:02:21.852629 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1215 21:02:21.852847 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1215 21:02:21.852850 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1215 21:02:21.852852 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1215 21:02:21.852945 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1215 21:02:21.853279 (+ 334us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1215 21:02:21.853286 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1215 21:02:21.853514 (+ 228us) client_negotiation.cc:769] Sending connection context
1215 21:02:21.853700 (+ 186us) client_negotiation.cc:240] Negotiation successful
1215 21:02:21.853982 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":103,"threads_started":1}
I20251215 21:02:21.855794 1551 heartbeater.cc:344] Connected to a master server at 127.28.113.62:33771
I20251215 21:02:21.856074 1551 heartbeater.cc:461] Registering TS with master...
I20251215 21:02:21.856546 1551 heartbeater.cc:507] Master 127.28.113.62:33771 requested a full tablet report, sending...
I20251215 21:02:21.857908 1355 ts_manager.cc:194] Registered new tserver with Master: fd27a28d3a7e461a9bf9b94258edc38a (127.28.113.1:35243)
WARNING: no policy specified for kudu/127.28.113.2@KRBTEST.COM; defaulting to no policy
I20251215 21:02:21.859035 1355 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.113.1@KRBTEST.COM'} at 127.28.113.1:34935
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.113.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.113.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765832506145157-29124-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.28.113.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765832506145157-29124-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.28.113.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.28.113.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.28.113.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765832506145157-29124-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.28.113.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765832506145157-29124-0/krb5kdc/kudu.keytab.
I20251215 21:02:21.891254 29124 external_mini_cluster.cc:1366] Running /tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu
/tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765832506145157-29124-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765832506145157-29124-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765832506145157-29124-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765832506145157-29124-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.113.2:0
--local_ip_for_outbound_sockets=127.28.113.2
--webserver_interface=127.28.113.2
--webserver_port=0
--tserver_master_addrs=127.28.113.62:33771
--keytab_file=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765832506145157-29124-0/krb5kdc/kudu.keytab
--principal=kudu/127.28.113.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.28.113.20:36365
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765832506145157-29124-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765832506145157-29124-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765832506145157-29124-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251215 21:02:21.993616 1558 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251215 21:02:21.993839 1558 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251215 21:02:21.993927 1558 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251215 21:02:21.997112 1558 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251215 21:02:21.997177 1558 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251215 21:02:21.997285 1558 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.113.2
I20251215 21:02:22.001390 1558 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.113.20:36365
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765832506145157-29124-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765832506145157-29124-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765832506145157-29124-0/krb5kdc/kudu.keytab
--principal=kudu/127.28.113.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.113.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765832506145157-29124-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.113.2
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.28.113.62:33771
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1558
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.113.2
--log_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765832506145157-29124-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision dc741732bb6a40213e11a13869fad06a49a209a3
build type DEBUG
built by None at 15 Dec 2025 20:43:14 UTC on 24a791456cd2
build id 9530
I20251215 21:02:22.002398 1558 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251215 21:02:22.003211 1558 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251215 21:02:22.009110 1566 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251215 21:02:22.009207 1558 server_base.cc:1047] running on GCE node
W20251215 21:02:22.009068 1563 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251215 21:02:22.009068 1564 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251215 21:02:22.009697 1558 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251215 21:02:22.010188 1558 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251215 21:02:22.011329 1558 hybrid_clock.cc:648] HybridClock initialized: now 1765832542011317 us; error 28 us; skew 500 ppm
Dec 15 21:02:22 dist-test-slave-bjsf krb5kdc[1312](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765832542, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.113.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251215 21:02:22.013777 1558 init.cc:377] Logged in from keytab as kudu/127.28.113.2@KRBTEST.COM (short username kudu)
I20251215 21:02:22.014465 1558 fs_manager.cc:362] Metadata directory not provided
I20251215 21:02:22.014525 1558 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251215 21:02:22.014693 1558 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251215 21:02:22.016381 1558 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765832506145157-29124-0/minicluster-data/ts-1/data/instance:
uuid: "4f489d2d8890467db8677e1650bbdcff"
format_stamp: "Formatted at 2025-12-15 21:02:22 on dist-test-slave-bjsf"
server_key: "ac270e71bb45abac0891b48754b6e9c6"
server_key_iv: "3ae8b4d3052139d32dfdcfa466894f3f"
server_key_version: "encryptionkey@0"
I20251215 21:02:22.016817 1558 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765832506145157-29124-0/minicluster-data/ts-1/wal/instance:
uuid: "4f489d2d8890467db8677e1650bbdcff"
format_stamp: "Formatted at 2025-12-15 21:02:22 on dist-test-slave-bjsf"
server_key: "ac270e71bb45abac0891b48754b6e9c6"
server_key_iv: "3ae8b4d3052139d32dfdcfa466894f3f"
server_key_version: "encryptionkey@0"
I20251215 21:02:22.020037 1558 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20251215 21:02:22.022182 1572 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251215 21:02:22.023183 1558 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20251215 21:02:22.023285 1558 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765832506145157-29124-0/minicluster-data/ts-1/data,/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765832506145157-29124-0/minicluster-data/ts-1/wal
uuid: "4f489d2d8890467db8677e1650bbdcff"
format_stamp: "Formatted at 2025-12-15 21:02:22 on dist-test-slave-bjsf"
server_key: "ac270e71bb45abac0891b48754b6e9c6"
server_key_iv: "3ae8b4d3052139d32dfdcfa466894f3f"
server_key_version: "encryptionkey@0"
I20251215 21:02:22.023370 1558 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765832506145157-29124-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765832506145157-29124-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765832506145157-29124-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251215 21:02:22.045368 1558 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251215 21:02:22.047932 1558 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251215 21:02:22.048122 1558 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251215 21:02:22.048642 1558 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251215 21:02:22.049481 1558 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251215 21:02:22.049530 1558 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251215 21:02:22.049580 1558 ts_tablet_manager.cc:616] Registered 0 tablets
I20251215 21:02:22.049604 1558 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251215 21:02:22.058043 1558 rpc_server.cc:307] RPC server started. Bound to: 127.28.113.2:33527
I20251215 21:02:22.058070 1685 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.28.113.2:33527 every 8 connection(s)
I20251215 21:02:22.058591 1558 server_base.cc:1179] Dumped server information to /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765832506145157-29124-0/minicluster-data/ts-1/data/info.pb
I20251215 21:02:22.066304 29124 external_mini_cluster.cc:1428] Started /tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu as pid 1558
Dec 15 21:02:22 dist-test-slave-bjsf krb5kdc[1312](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765832542, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.113.2@KRBTEST.COM for kudu/127.28.113.62@KRBTEST.COM
I20251215 21:02:22.066422 29124 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765832506145157-29124-0/minicluster-data/ts-1/wal/instance
I20251215 21:02:22.066642 29124 external_mini_cluster.cc:1468] Setting key 860d245b916f818622bb9ead7e9cc3ec
I20251215 21:02:22.070235 1398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:02:22.060147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.113.2:47693 (local address 127.28.113.62:33771)
1215 21:02:22.060274 (+ 127us) server_negotiation.cc:207] Beginning negotiation
1215 21:02:22.060278 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1215 21:02:22.060911 (+ 633us) server_negotiation.cc:408] Connection header received
1215 21:02:22.061740 (+ 829us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1215 21:02:22.061744 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1215 21:02:22.061804 (+ 60us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1215 21:02:22.061877 (+ 73us) server_negotiation.cc:227] Negotiated authn=SASL
1215 21:02:22.063516 (+ 1639us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:02:22.064054 (+ 538us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1215 21:02:22.064688 (+ 634us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:02:22.064858 (+ 170us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:02:22.066935 (+ 2077us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1215 21:02:22.066956 (+ 21us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1215 21:02:22.066958 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1215 21:02:22.066986 (+ 28us) server_negotiation.cc:923] calling sasl_server_start()
1215 21:02:22.068437 (+ 1451us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1215 21:02:22.068909 (+ 472us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1215 21:02:22.068912 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1215 21:02:22.068914 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1215 21:02:22.068957 (+ 43us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1215 21:02:22.069205 (+ 248us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1215 21:02:22.069209 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1215 21:02:22.069211 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1215 21:02:22.069360 (+ 149us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1215 21:02:22.069445 (+ 85us) server_negotiation.cc:1036] Waiting for connection context
1215 21:02:22.069951 (+ 506us) server_negotiation.cc:300] Negotiation successful
1215 21:02:22.070072 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20251215 21:02:22.070986 1688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:02:22.060387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.113.62:33771 (local address 127.28.113.2:47693)
1215 21:02:22.060789 (+ 402us) negotiation.cc:107] Waiting for socket to connect
1215 21:02:22.060822 (+ 33us) client_negotiation.cc:174] Beginning negotiation
1215 21:02:22.061532 (+ 710us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1215 21:02:22.062049 (+ 517us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1215 21:02:22.062057 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
1215 21:02:22.062434 (+ 377us) client_negotiation.cc:189] Negotiated authn=SASL
1215 21:02:22.063360 (+ 926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:02:22.063375 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:02:22.064168 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1215 21:02:22.064171 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1215 21:02:22.064574 (+ 403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:02:22.064580 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:02:22.064779 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:02:22.065363 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1215 21:02:22.065387 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
1215 21:02:22.066789 (+ 1402us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1215 21:02:22.068554 (+ 1765us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1215 21:02:22.068560 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1215 21:02:22.068574 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
1215 21:02:22.068809 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1215 21:02:22.069034 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1215 21:02:22.069037 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1215 21:02:22.069039 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1215 21:02:22.069125 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1215 21:02:22.069455 (+ 330us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1215 21:02:22.069463 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1215 21:02:22.069710 (+ 247us) client_negotiation.cc:769] Sending connection context
1215 21:02:22.069898 (+ 188us) client_negotiation.cc:240] Negotiation successful
1215 21:02:22.070173 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":93,"threads_started":1}
I20251215 21:02:22.072078 1686 heartbeater.cc:344] Connected to a master server at 127.28.113.62:33771
I20251215 21:02:22.072319 1686 heartbeater.cc:461] Registering TS with master...
I20251215 21:02:22.072790 1686 heartbeater.cc:507] Master 127.28.113.62:33771 requested a full tablet report, sending...
I20251215 21:02:22.073814 1355 ts_manager.cc:194] Registered new tserver with Master: 4f489d2d8890467db8677e1650bbdcff (127.28.113.2:33527)
I20251215 21:02:22.074559 1355 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.113.2@KRBTEST.COM'} at 127.28.113.2:47693
WARNING: no policy specified for kudu/127.28.113.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.113.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.113.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765832506145157-29124-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.28.113.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765832506145157-29124-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.28.113.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.28.113.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.28.113.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765832506145157-29124-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.28.113.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765832506145157-29124-0/krb5kdc/kudu.keytab.
I20251215 21:02:22.108902 29124 external_mini_cluster.cc:1366] Running /tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu
/tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765832506145157-29124-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765832506145157-29124-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765832506145157-29124-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765832506145157-29124-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.113.3:0
--local_ip_for_outbound_sockets=127.28.113.3
--webserver_interface=127.28.113.3
--webserver_port=0
--tserver_master_addrs=127.28.113.62:33771
--keytab_file=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765832506145157-29124-0/krb5kdc/kudu.keytab
--principal=kudu/127.28.113.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.28.113.20:36365
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765832506145157-29124-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765832506145157-29124-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765832506145157-29124-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251215 21:02:22.210223 1693 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251215 21:02:22.210425 1693 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251215 21:02:22.210480 1693 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251215 21:02:22.213641 1693 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251215 21:02:22.213701 1693 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251215 21:02:22.213778 1693 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.113.3
I20251215 21:02:22.217880 1693 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.113.20:36365
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765832506145157-29124-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765832506145157-29124-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765832506145157-29124-0/krb5kdc/kudu.keytab
--principal=kudu/127.28.113.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.113.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765832506145157-29124-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.113.3
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.28.113.62:33771
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1693
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.113.3
--log_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765832506145157-29124-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision dc741732bb6a40213e11a13869fad06a49a209a3
build type DEBUG
built by None at 15 Dec 2025 20:43:14 UTC on 24a791456cd2
build id 9530
I20251215 21:02:22.218861 1693 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251215 21:02:22.219632 1693 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251215 21:02:22.225525 1698 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251215 21:02:22.225521 1701 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251215 21:02:22.225564 1699 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251215 21:02:22.225809 1693 server_base.cc:1047] running on GCE node
I20251215 21:02:22.226156 1693 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251215 21:02:22.226648 1693 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251215 21:02:22.227823 1693 hybrid_clock.cc:648] HybridClock initialized: now 1765832542227801 us; error 32 us; skew 500 ppm
Dec 15 21:02:22 dist-test-slave-bjsf krb5kdc[1312](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765832542, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.113.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251215 21:02:22.230319 1693 init.cc:377] Logged in from keytab as kudu/127.28.113.3@KRBTEST.COM (short username kudu)
I20251215 21:02:22.230996 1693 fs_manager.cc:362] Metadata directory not provided
I20251215 21:02:22.231055 1693 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251215 21:02:22.231221 1693 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251215 21:02:22.232894 1693 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765832506145157-29124-0/minicluster-data/ts-2/data/instance:
uuid: "ea1619b8a1b741b7bb84dd4fdc76a5d9"
format_stamp: "Formatted at 2025-12-15 21:02:22 on dist-test-slave-bjsf"
server_key: "14c7b1eb7e1888455d113ec5ec295dae"
server_key_iv: "a193fc9abadbdcbc68552f61b20eee10"
server_key_version: "encryptionkey@0"
I20251215 21:02:22.233330 1693 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765832506145157-29124-0/minicluster-data/ts-2/wal/instance:
uuid: "ea1619b8a1b741b7bb84dd4fdc76a5d9"
format_stamp: "Formatted at 2025-12-15 21:02:22 on dist-test-slave-bjsf"
server_key: "14c7b1eb7e1888455d113ec5ec295dae"
server_key_iv: "a193fc9abadbdcbc68552f61b20eee10"
server_key_version: "encryptionkey@0"
I20251215 21:02:22.236415 1693 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20251215 21:02:22.238451 1707 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251215 21:02:22.239382 1693 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20251215 21:02:22.239481 1693 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765832506145157-29124-0/minicluster-data/ts-2/data,/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765832506145157-29124-0/minicluster-data/ts-2/wal
uuid: "ea1619b8a1b741b7bb84dd4fdc76a5d9"
format_stamp: "Formatted at 2025-12-15 21:02:22 on dist-test-slave-bjsf"
server_key: "14c7b1eb7e1888455d113ec5ec295dae"
server_key_iv: "a193fc9abadbdcbc68552f61b20eee10"
server_key_version: "encryptionkey@0"
I20251215 21:02:22.239560 1693 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765832506145157-29124-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765832506145157-29124-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765832506145157-29124-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251215 21:02:22.252948 1693 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251215 21:02:22.255342 1693 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251215 21:02:22.255504 1693 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251215 21:02:22.256053 1693 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251215 21:02:22.256883 1693 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251215 21:02:22.256929 1693 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251215 21:02:22.256979 1693 ts_tablet_manager.cc:616] Registered 0 tablets
I20251215 21:02:22.257009 1693 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251215 21:02:22.265303 1693 rpc_server.cc:307] RPC server started. Bound to: 127.28.113.3:41093
I20251215 21:02:22.265329 1820 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.28.113.3:41093 every 8 connection(s)
I20251215 21:02:22.265794 1693 server_base.cc:1179] Dumped server information to /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765832506145157-29124-0/minicluster-data/ts-2/data/info.pb
Dec 15 21:02:22 dist-test-slave-bjsf krb5kdc[1312](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765832542, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.113.3@KRBTEST.COM for kudu/127.28.113.62@KRBTEST.COM
I20251215 21:02:22.273952 29124 external_mini_cluster.cc:1428] Started /tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu as pid 1693
I20251215 21:02:22.274057 29124 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765832506145157-29124-0/minicluster-data/ts-2/wal/instance
I20251215 21:02:22.274323 29124 external_mini_cluster.cc:1468] Setting key 3eed9bc15432a26f773b14efc6037784
I20251215 21:02:22.277091 1398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:02:22.267237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.113.3:43063 (local address 127.28.113.62:33771)
1215 21:02:22.267402 (+ 165us) server_negotiation.cc:207] Beginning negotiation
1215 21:02:22.267408 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1215 21:02:22.268072 (+ 664us) server_negotiation.cc:408] Connection header received
1215 21:02:22.268843 (+ 771us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1215 21:02:22.268846 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1215 21:02:22.268898 (+ 52us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1215 21:02:22.268972 (+ 74us) server_negotiation.cc:227] Negotiated authn=SASL
1215 21:02:22.270545 (+ 1573us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:02:22.271073 (+ 528us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1215 21:02:22.271687 (+ 614us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:02:22.271825 (+ 138us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:02:22.273777 (+ 1952us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1215 21:02:22.273797 (+ 20us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1215 21:02:22.273800 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1215 21:02:22.273826 (+ 26us) server_negotiation.cc:923] calling sasl_server_start()
1215 21:02:22.275277 (+ 1451us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1215 21:02:22.275769 (+ 492us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1215 21:02:22.275773 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1215 21:02:22.275775 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1215 21:02:22.275826 (+ 51us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1215 21:02:22.276108 (+ 282us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1215 21:02:22.276111 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1215 21:02:22.276113 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1215 21:02:22.276307 (+ 194us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1215 21:02:22.276399 (+ 92us) server_negotiation.cc:1036] Waiting for connection context
1215 21:02:22.276873 (+ 474us) server_negotiation.cc:300] Negotiation successful
1215 21:02:22.276970 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20251215 21:02:22.277653 1823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:02:22.267497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.113.62:33771 (local address 127.28.113.3:43063)
1215 21:02:22.267915 (+ 418us) negotiation.cc:107] Waiting for socket to connect
1215 21:02:22.267948 (+ 33us) client_negotiation.cc:174] Beginning negotiation
1215 21:02:22.268664 (+ 716us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1215 21:02:22.269104 (+ 440us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1215 21:02:22.269111 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
1215 21:02:22.269478 (+ 367us) client_negotiation.cc:189] Negotiated authn=SASL
1215 21:02:22.270387 (+ 909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:02:22.270402 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:02:22.271176 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1215 21:02:22.271178 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1215 21:02:22.271574 (+ 396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:02:22.271580 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:02:22.271736 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:02:22.272279 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1215 21:02:22.272301 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
1215 21:02:22.273623 (+ 1322us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1215 21:02:22.275387 (+ 1764us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1215 21:02:22.275393 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1215 21:02:22.275402 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
1215 21:02:22.275682 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1215 21:02:22.275909 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1215 21:02:22.275911 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1215 21:02:22.275912 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1215 21:02:22.276032 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1215 21:02:22.276419 (+ 387us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1215 21:02:22.276426 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1215 21:02:22.276644 (+ 218us) client_negotiation.cc:769] Sending connection context
1215 21:02:22.276809 (+ 165us) client_negotiation.cc:240] Negotiation successful
1215 21:02:22.277025 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":100,"threads_started":1}
I20251215 21:02:22.278550 1821 heartbeater.cc:344] Connected to a master server at 127.28.113.62:33771
I20251215 21:02:22.278762 1821 heartbeater.cc:461] Registering TS with master...
I20251215 21:02:22.279174 1821 heartbeater.cc:507] Master 127.28.113.62:33771 requested a full tablet report, sending...
I20251215 21:02:22.280144 1355 ts_manager.cc:194] Registered new tserver with Master: ea1619b8a1b741b7bb84dd4fdc76a5d9 (127.28.113.3:41093)
I20251215 21:02:22.280678 1355 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.113.3@KRBTEST.COM'} at 127.28.113.3:43063
I20251215 21:02:22.287848 29124 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20251215 21:02:22.295331 1398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:02:22.289539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53674 (local address 127.28.113.62:33771)
1215 21:02:22.289659 (+ 120us) server_negotiation.cc:207] Beginning negotiation
1215 21:02:22.289662 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1215 21:02:22.289825 (+ 163us) server_negotiation.cc:408] Connection header received
1215 21:02:22.289935 (+ 110us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1215 21:02:22.289937 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1215 21:02:22.289988 (+ 51us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1215 21:02:22.290052 (+ 64us) server_negotiation.cc:227] Negotiated authn=SASL
1215 21:02:22.290840 (+ 788us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:02:22.291319 (+ 479us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1215 21:02:22.291898 (+ 579us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:02:22.292060 (+ 162us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:02:22.293028 (+ 968us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1215 21:02:22.293042 (+ 14us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1215 21:02:22.293045 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1215 21:02:22.293066 (+ 21us) server_negotiation.cc:923] calling sasl_server_start()
1215 21:02:22.294307 (+ 1241us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1215 21:02:22.294581 (+ 274us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1215 21:02:22.294583 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1215 21:02:22.294585 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1215 21:02:22.294618 (+ 33us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1215 21:02:22.294760 (+ 142us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1215 21:02:22.294762 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1215 21:02:22.294763 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1215 21:02:22.294911 (+ 148us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1215 21:02:22.295011 (+ 100us) server_negotiation.cc:1036] Waiting for connection context
1215 21:02:22.295131 (+ 120us) server_negotiation.cc:300] Negotiation successful
1215 21:02:22.295212 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20251215 21:02:22.297581 1355 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53674:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251215 21:02:22.299865 1355 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251215 21:02:22.311043 1832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:02:22.307076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.113.3:41093 (local address 127.0.0.1:53838)
1215 21:02:22.307391 (+ 315us) negotiation.cc:107] Waiting for socket to connect
1215 21:02:22.307428 (+ 37us) client_negotiation.cc:174] Beginning negotiation
1215 21:02:22.307547 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1215 21:02:22.308532 (+ 985us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1215 21:02:22.308538 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
1215 21:02:22.308560 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1215 21:02:22.308815 (+ 255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:02:22.308821 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:02:22.309818 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1215 21:02:22.309822 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1215 21:02:22.310563 (+ 741us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:02:22.310569 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:02:22.310692 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:02:22.310751 (+ 59us) client_negotiation.cc:769] Sending connection context
1215 21:02:22.310836 (+ 85us) client_negotiation.cc:240] Negotiation successful
1215 21:02:22.310911 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":71,"threads_started":1}
I20251215 21:02:22.311269 1834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:02:22.307592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.113.1:35243 (local address 127.0.0.1:52396)
1215 21:02:22.308144 (+ 552us) negotiation.cc:107] Waiting for socket to connect
1215 21:02:22.308157 (+ 13us) client_negotiation.cc:174] Beginning negotiation
1215 21:02:22.308228 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1215 21:02:22.308811 (+ 583us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1215 21:02:22.308815 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
1215 21:02:22.308831 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1215 21:02:22.309036 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:02:22.309042 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:02:22.310017 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1215 21:02:22.310021 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1215 21:02:22.310923 (+ 902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:02:22.310934 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:02:22.311031 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:02:22.311048 (+ 17us) client_negotiation.cc:769] Sending connection context
1215 21:02:22.311094 (+ 46us) client_negotiation.cc:240] Negotiation successful
1215 21:02:22.311151 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":484,"thread_start_us":74,"threads_started":1}
I20251215 21:02:22.311604 1833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:02:22.307179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53838 (local address 127.28.113.3:41093)
1215 21:02:22.307816 (+ 637us) server_negotiation.cc:207] Beginning negotiation
1215 21:02:22.307823 (+ 7us) server_negotiation.cc:400] Waiting for connection header
1215 21:02:22.307846 (+ 23us) server_negotiation.cc:408] Connection header received
1215 21:02:22.307940 (+ 94us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1215 21:02:22.307945 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1215 21:02:22.308151 (+ 206us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1215 21:02:22.308944 (+ 793us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1215 21:02:22.309052 (+ 108us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:02:22.309721 (+ 669us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1215 21:02:22.310704 (+ 983us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:02:22.311230 (+ 526us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:02:22.311340 (+ 110us) server_negotiation.cc:1036] Waiting for connection context
1215 21:02:22.311426 (+ 86us) server_negotiation.cc:300] Negotiation successful
1215 21:02:22.311507 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":525,"thread_start_us":144,"threads_started":1}
I20251215 21:02:22.312057 1835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:02:22.307800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52396 (local address 127.28.113.1:35243)
1215 21:02:22.308448 (+ 648us) server_negotiation.cc:207] Beginning negotiation
1215 21:02:22.308453 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1215 21:02:22.308474 (+ 21us) server_negotiation.cc:408] Connection header received
1215 21:02:22.308549 (+ 75us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1215 21:02:22.308552 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1215 21:02:22.308695 (+ 143us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1215 21:02:22.308814 (+ 119us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1215 21:02:22.309134 (+ 320us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:02:22.309900 (+ 766us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1215 21:02:22.311099 (+ 1199us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:02:22.311679 (+ 580us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:02:22.311778 (+ 99us) server_negotiation.cc:1036] Waiting for connection context
1215 21:02:22.311834 (+ 56us) server_negotiation.cc:300] Negotiation successful
1215 21:02:22.311906 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":552,"thread_start_us":105,"threads_started":1}
I20251215 21:02:22.314505 1836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:02:22.307862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.113.2:33527 (local address 127.0.0.1:36258)
1215 21:02:22.308375 (+ 513us) negotiation.cc:107] Waiting for socket to connect
1215 21:02:22.308388 (+ 13us) client_negotiation.cc:174] Beginning negotiation
1215 21:02:22.308453 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1215 21:02:22.310321 (+ 1868us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1215 21:02:22.310325 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
1215 21:02:22.310344 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1215 21:02:22.310562 (+ 218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:02:22.310569 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:02:22.313326 (+ 2757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1215 21:02:22.313330 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1215 21:02:22.314168 (+ 838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:02:22.314178 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:02:22.314288 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:02:22.314303 (+ 15us) client_negotiation.cc:769] Sending connection context
1215 21:02:22.314345 (+ 42us) client_negotiation.cc:240] Negotiation successful
1215 21:02:22.314396 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":456,"thread_start_us":65,"threads_started":1}
I20251215 21:02:22.314404 1755 tablet_service.cc:1505] Processing CreateTablet for tablet 50980bcbdf6d4e0aa449ef0fed4fabd6 (DEFAULT_TABLE table=test-table [id=dcbe1f9781a742e49e2ebd1e9d815ef4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251215 21:02:22.314560 1485 tablet_service.cc:1505] Processing CreateTablet for tablet 50980bcbdf6d4e0aa449ef0fed4fabd6 (DEFAULT_TABLE table=test-table [id=dcbe1f9781a742e49e2ebd1e9d815ef4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251215 21:02:22.315200 1837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:02:22.308163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36258 (local address 127.28.113.2:33527)
1215 21:02:22.309948 (+ 1785us) server_negotiation.cc:207] Beginning negotiation
1215 21:02:22.309954 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1215 21:02:22.309975 (+ 21us) server_negotiation.cc:408] Connection header received
1215 21:02:22.310049 (+ 74us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1215 21:02:22.310054 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1215 21:02:22.310195 (+ 141us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1215 21:02:22.310318 (+ 123us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1215 21:02:22.312329 (+ 2011us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:02:22.313200 (+ 871us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1215 21:02:22.314337 (+ 1137us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:02:22.314838 (+ 501us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:02:22.314946 (+ 108us) server_negotiation.cc:1036] Waiting for connection context
1215 21:02:22.315005 (+ 59us) server_negotiation.cc:300] Negotiation successful
1215 21:02:22.315079 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1689,"thread_start_us":102,"threads_started":1}
I20251215 21:02:22.315333 1755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 50980bcbdf6d4e0aa449ef0fed4fabd6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 21:02:22.315477 1485 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 50980bcbdf6d4e0aa449ef0fed4fabd6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 21:02:22.317605 1620 tablet_service.cc:1505] Processing CreateTablet for tablet 50980bcbdf6d4e0aa449ef0fed4fabd6 (DEFAULT_TABLE table=test-table [id=dcbe1f9781a742e49e2ebd1e9d815ef4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251215 21:02:22.318444 1620 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 50980bcbdf6d4e0aa449ef0fed4fabd6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 21:02:22.320736 1838 tablet_bootstrap.cc:492] T 50980bcbdf6d4e0aa449ef0fed4fabd6 P ea1619b8a1b741b7bb84dd4fdc76a5d9: Bootstrap starting.
I20251215 21:02:22.322594 1838 tablet_bootstrap.cc:654] T 50980bcbdf6d4e0aa449ef0fed4fabd6 P ea1619b8a1b741b7bb84dd4fdc76a5d9: Neither blocks nor log segments found. Creating new log.
I20251215 21:02:22.323230 1838 log.cc:826] T 50980bcbdf6d4e0aa449ef0fed4fabd6 P ea1619b8a1b741b7bb84dd4fdc76a5d9: Log is configured to *not* fsync() on all Append() calls
I20251215 21:02:22.323635 1840 tablet_bootstrap.cc:492] T 50980bcbdf6d4e0aa449ef0fed4fabd6 P 4f489d2d8890467db8677e1650bbdcff: Bootstrap starting.
I20251215 21:02:22.324205 1839 tablet_bootstrap.cc:492] T 50980bcbdf6d4e0aa449ef0fed4fabd6 P fd27a28d3a7e461a9bf9b94258edc38a: Bootstrap starting.
I20251215 21:02:22.325508 1840 tablet_bootstrap.cc:654] T 50980bcbdf6d4e0aa449ef0fed4fabd6 P 4f489d2d8890467db8677e1650bbdcff: Neither blocks nor log segments found. Creating new log.
I20251215 21:02:22.325990 1839 tablet_bootstrap.cc:654] T 50980bcbdf6d4e0aa449ef0fed4fabd6 P fd27a28d3a7e461a9bf9b94258edc38a: Neither blocks nor log segments found. Creating new log.
I20251215 21:02:22.326252 1840 log.cc:826] T 50980bcbdf6d4e0aa449ef0fed4fabd6 P 4f489d2d8890467db8677e1650bbdcff: Log is configured to *not* fsync() on all Append() calls
I20251215 21:02:22.326674 1839 log.cc:826] T 50980bcbdf6d4e0aa449ef0fed4fabd6 P fd27a28d3a7e461a9bf9b94258edc38a: Log is configured to *not* fsync() on all Append() calls
I20251215 21:02:22.328027 1840 tablet_bootstrap.cc:492] T 50980bcbdf6d4e0aa449ef0fed4fabd6 P 4f489d2d8890467db8677e1650bbdcff: No bootstrap required, opened a new log
I20251215 21:02:22.328258 1840 ts_tablet_manager.cc:1403] T 50980bcbdf6d4e0aa449ef0fed4fabd6 P 4f489d2d8890467db8677e1650bbdcff: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20251215 21:02:22.328346 1838 tablet_bootstrap.cc:492] T 50980bcbdf6d4e0aa449ef0fed4fabd6 P ea1619b8a1b741b7bb84dd4fdc76a5d9: No bootstrap required, opened a new log
I20251215 21:02:22.328567 1838 ts_tablet_manager.cc:1403] T 50980bcbdf6d4e0aa449ef0fed4fabd6 P ea1619b8a1b741b7bb84dd4fdc76a5d9: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.000s
I20251215 21:02:22.328647 1839 tablet_bootstrap.cc:492] T 50980bcbdf6d4e0aa449ef0fed4fabd6 P fd27a28d3a7e461a9bf9b94258edc38a: No bootstrap required, opened a new log
I20251215 21:02:22.328848 1839 ts_tablet_manager.cc:1403] T 50980bcbdf6d4e0aa449ef0fed4fabd6 P fd27a28d3a7e461a9bf9b94258edc38a: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.003s
I20251215 21:02:22.331259 1838 raft_consensus.cc:359] T 50980bcbdf6d4e0aa449ef0fed4fabd6 P ea1619b8a1b741b7bb84dd4fdc76a5d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea1619b8a1b741b7bb84dd4fdc76a5d9" member_type: VOTER last_known_addr { host: "127.28.113.3" port: 41093 } } peers { permanent_uuid: "fd27a28d3a7e461a9bf9b94258edc38a" member_type: VOTER last_known_addr { host: "127.28.113.1" port: 35243 } } peers { permanent_uuid: "4f489d2d8890467db8677e1650bbdcff" member_type: VOTER last_known_addr { host: "127.28.113.2" port: 33527 } }
I20251215 21:02:22.331436 1838 raft_consensus.cc:385] T 50980bcbdf6d4e0aa449ef0fed4fabd6 P ea1619b8a1b741b7bb84dd4fdc76a5d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251215 21:02:22.331511 1838 raft_consensus.cc:740] T 50980bcbdf6d4e0aa449ef0fed4fabd6 P ea1619b8a1b741b7bb84dd4fdc76a5d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea1619b8a1b741b7bb84dd4fdc76a5d9, State: Initialized, Role: FOLLOWER
I20251215 21:02:22.331914 1838 consensus_queue.cc:260] T 50980bcbdf6d4e0aa449ef0fed4fabd6 P ea1619b8a1b741b7bb84dd4fdc76a5d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea1619b8a1b741b7bb84dd4fdc76a5d9" member_type: VOTER last_known_addr { host: "127.28.113.3" port: 41093 } } peers { permanent_uuid: "fd27a28d3a7e461a9bf9b94258edc38a" member_type: VOTER last_known_addr { host: "127.28.113.1" port: 35243 } } peers { permanent_uuid: "4f489d2d8890467db8677e1650bbdcff" member_type: VOTER last_known_addr { host: "127.28.113.2" port: 33527 } }
I20251215 21:02:22.331858 1840 raft_consensus.cc:359] T 50980bcbdf6d4e0aa449ef0fed4fabd6 P 4f489d2d8890467db8677e1650bbdcff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea1619b8a1b741b7bb84dd4fdc76a5d9" member_type: VOTER last_known_addr { host: "127.28.113.3" port: 41093 } } peers { permanent_uuid: "fd27a28d3a7e461a9bf9b94258edc38a" member_type: VOTER last_known_addr { host: "127.28.113.1" port: 35243 } } peers { permanent_uuid: "4f489d2d8890467db8677e1650bbdcff" member_type: VOTER last_known_addr { host: "127.28.113.2" port: 33527 } }
I20251215 21:02:22.332080 1840 raft_consensus.cc:385] T 50980bcbdf6d4e0aa449ef0fed4fabd6 P 4f489d2d8890467db8677e1650bbdcff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251215 21:02:22.332168 1840 raft_consensus.cc:740] T 50980bcbdf6d4e0aa449ef0fed4fabd6 P 4f489d2d8890467db8677e1650bbdcff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f489d2d8890467db8677e1650bbdcff, State: Initialized, Role: FOLLOWER
I20251215 21:02:22.332120 1839 raft_consensus.cc:359] T 50980bcbdf6d4e0aa449ef0fed4fabd6 P fd27a28d3a7e461a9bf9b94258edc38a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea1619b8a1b741b7bb84dd4fdc76a5d9" member_type: VOTER last_known_addr { host: "127.28.113.3" port: 41093 } } peers { permanent_uuid: "fd27a28d3a7e461a9bf9b94258edc38a" member_type: VOTER last_known_addr { host: "127.28.113.1" port: 35243 } } peers { permanent_uuid: "4f489d2d8890467db8677e1650bbdcff" member_type: VOTER last_known_addr { host: "127.28.113.2" port: 33527 } }
I20251215 21:02:22.332305 1839 raft_consensus.cc:385] T 50980bcbdf6d4e0aa449ef0fed4fabd6 P fd27a28d3a7e461a9bf9b94258edc38a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251215 21:02:22.332361 1839 raft_consensus.cc:740] T 50980bcbdf6d4e0aa449ef0fed4fabd6 P fd27a28d3a7e461a9bf9b94258edc38a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd27a28d3a7e461a9bf9b94258edc38a, State: Initialized, Role: FOLLOWER
I20251215 21:02:22.332563 1840 consensus_queue.cc:260] T 50980bcbdf6d4e0aa449ef0fed4fabd6 P 4f489d2d8890467db8677e1650bbdcff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea1619b8a1b741b7bb84dd4fdc76a5d9" member_type: VOTER last_known_addr { host: "127.28.113.3" port: 41093 } } peers { permanent_uuid: "fd27a28d3a7e461a9bf9b94258edc38a" member_type: VOTER last_known_addr { host: "127.28.113.1" port: 35243 } } peers { permanent_uuid: "4f489d2d8890467db8677e1650bbdcff" member_type: VOTER last_known_addr { host: "127.28.113.2" port: 33527 } }
I20251215 21:02:22.332671 1839 consensus_queue.cc:260] T 50980bcbdf6d4e0aa449ef0fed4fabd6 P fd27a28d3a7e461a9bf9b94258edc38a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea1619b8a1b741b7bb84dd4fdc76a5d9" member_type: VOTER last_known_addr { host: "127.28.113.3" port: 41093 } } peers { permanent_uuid: "fd27a28d3a7e461a9bf9b94258edc38a" member_type: VOTER last_known_addr { host: "127.28.113.1" port: 35243 } } peers { permanent_uuid: "4f489d2d8890467db8677e1650bbdcff" member_type: VOTER last_known_addr { host: "127.28.113.2" port: 33527 } }
I20251215 21:02:22.332798 1838 ts_tablet_manager.cc:1434] T 50980bcbdf6d4e0aa449ef0fed4fabd6 P ea1619b8a1b741b7bb84dd4fdc76a5d9: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20251215 21:02:22.333185 1821 heartbeater.cc:499] Master 127.28.113.62:33771 was elected leader, sending a full tablet report...
I20251215 21:02:22.333344 1839 ts_tablet_manager.cc:1434] T 50980bcbdf6d4e0aa449ef0fed4fabd6 P fd27a28d3a7e461a9bf9b94258edc38a: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20251215 21:02:22.333344 1840 ts_tablet_manager.cc:1434] T 50980bcbdf6d4e0aa449ef0fed4fabd6 P 4f489d2d8890467db8677e1650bbdcff: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20251215 21:02:22.333653 1551 heartbeater.cc:499] Master 127.28.113.62:33771 was elected leader, sending a full tablet report...
I20251215 21:02:22.334270 1686 heartbeater.cc:499] Master 127.28.113.62:33771 was elected leader, sending a full tablet report...
W20251215 21:02:22.342911 1552 tablet.cc:2378] T 50980bcbdf6d4e0aa449ef0fed4fabd6 P fd27a28d3a7e461a9bf9b94258edc38a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20251215 21:02:22.397646 1846 raft_consensus.cc:493] T 50980bcbdf6d4e0aa449ef0fed4fabd6 P fd27a28d3a7e461a9bf9b94258edc38a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 21:02:22.397847 1846 raft_consensus.cc:515] T 50980bcbdf6d4e0aa449ef0fed4fabd6 P fd27a28d3a7e461a9bf9b94258edc38a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea1619b8a1b741b7bb84dd4fdc76a5d9" member_type: VOTER last_known_addr { host: "127.28.113.3" port: 41093 } } peers { permanent_uuid: "fd27a28d3a7e461a9bf9b94258edc38a" member_type: VOTER last_known_addr { host: "127.28.113.1" port: 35243 } } peers { permanent_uuid: "4f489d2d8890467db8677e1650bbdcff" member_type: VOTER last_known_addr { host: "127.28.113.2" port: 33527 } }
I20251215 21:02:22.398978 1846 leader_election.cc:290] T 50980bcbdf6d4e0aa449ef0fed4fabd6 P fd27a28d3a7e461a9bf9b94258edc38a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea1619b8a1b741b7bb84dd4fdc76a5d9 (127.28.113.3:41093), 4f489d2d8890467db8677e1650bbdcff (127.28.113.2:33527)
I20251215 21:02:22.402670 1847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:02:22.399190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.113.2:33527 (local address 127.28.113.1:58981)
1215 21:02:22.399515 (+ 325us) negotiation.cc:107] Waiting for socket to connect
1215 21:02:22.399528 (+ 13us) client_negotiation.cc:174] Beginning negotiation
1215 21:02:22.399644 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1215 21:02:22.399896 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1215 21:02:22.399899 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1215 21:02:22.399937 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1215 21:02:22.400283 (+ 346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:02:22.400291 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:02:22.401274 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1215 21:02:22.401278 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1215 21:02:22.402221 (+ 943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:02:22.402230 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:02:22.402346 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:02:22.402369 (+ 23us) client_negotiation.cc:769] Sending connection context
1215 21:02:22.402418 (+ 49us) client_negotiation.cc:240] Negotiation successful
1215 21:02:22.402489 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"spinlock_wait_cycles":7296,"thread_start_us":92,"threads_started":1}
I20251215 21:02:22.402674 1848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:02:22.399190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.113.3:41093 (local address 127.28.113.1:60153)
1215 21:02:22.399554 (+ 364us) negotiation.cc:107] Waiting for socket to connect
1215 21:02:22.399564 (+ 10us) client_negotiation.cc:174] Beginning negotiation
1215 21:02:22.399651 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1215 21:02:22.399896 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1215 21:02:22.399899 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1215 21:02:22.399935 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1215 21:02:22.400283 (+ 348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:02:22.400291 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:02:22.401274 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1215 21:02:22.401278 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1215 21:02:22.402221 (+ 943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:02:22.402230 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:02:22.402347 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:02:22.402369 (+ 22us) client_negotiation.cc:769] Sending connection context
1215 21:02:22.402418 (+ 49us) client_negotiation.cc:240] Negotiation successful
1215 21:02:22.402489 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":297,"mutex_wait_us":47,"spinlock_wait_cycles":35584,"thread_start_us":121,"threads_started":1}
I20251215 21:02:22.403107 1837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:02:22.399308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.113.1:58981 (local address 127.28.113.2:33527)
1215 21:02:22.399441 (+ 133us) server_negotiation.cc:207] Beginning negotiation
1215 21:02:22.399445 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1215 21:02:22.399601 (+ 156us) server_negotiation.cc:408] Connection header received
1215 21:02:22.399751 (+ 150us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1215 21:02:22.399754 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1215 21:02:22.399801 (+ 47us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1215 21:02:22.399893 (+ 92us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1215 21:02:22.400392 (+ 499us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:02:22.401176 (+ 784us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1215 21:02:22.402343 (+ 1167us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:02:22.402852 (+ 509us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:02:22.402895 (+ 43us) server_negotiation.cc:1036] Waiting for connection context
1215 21:02:22.402940 (+ 45us) server_negotiation.cc:300] Negotiation successful
1215 21:02:22.402994 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20251215 21:02:22.403107 1833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:02:22.399287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.113.1:60153 (local address 127.28.113.3:41093)
1215 21:02:22.399422 (+ 135us) server_negotiation.cc:207] Beginning negotiation
1215 21:02:22.399426 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1215 21:02:22.399654 (+ 228us) server_negotiation.cc:408] Connection header received
1215 21:02:22.399736 (+ 82us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1215 21:02:22.399739 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1215 21:02:22.399797 (+ 58us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1215 21:02:22.399893 (+ 96us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1215 21:02:22.400392 (+ 499us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:02:22.401172 (+ 780us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1215 21:02:22.402343 (+ 1171us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:02:22.402852 (+ 509us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:02:22.402890 (+ 38us) server_negotiation.cc:1036] Waiting for connection context
1215 21:02:22.402940 (+ 50us) server_negotiation.cc:300] Negotiation successful
1215 21:02:22.402994 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20251215 21:02:22.403714 1640 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "50980bcbdf6d4e0aa449ef0fed4fabd6" candidate_uuid: "fd27a28d3a7e461a9bf9b94258edc38a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f489d2d8890467db8677e1650bbdcff" is_pre_election: true
I20251215 21:02:22.403724 1775 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "50980bcbdf6d4e0aa449ef0fed4fabd6" candidate_uuid: "fd27a28d3a7e461a9bf9b94258edc38a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea1619b8a1b741b7bb84dd4fdc76a5d9" is_pre_election: true
I20251215 21:02:22.404023 1640 raft_consensus.cc:2468] T 50980bcbdf6d4e0aa449ef0fed4fabd6 P 4f489d2d8890467db8677e1650bbdcff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd27a28d3a7e461a9bf9b94258edc38a in term 0.
I20251215 21:02:22.404023 1775 raft_consensus.cc:2468] T 50980bcbdf6d4e0aa449ef0fed4fabd6 P ea1619b8a1b741b7bb84dd4fdc76a5d9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd27a28d3a7e461a9bf9b94258edc38a in term 0.
I20251215 21:02:22.404456 1439 leader_election.cc:304] T 50980bcbdf6d4e0aa449ef0fed4fabd6 P fd27a28d3a7e461a9bf9b94258edc38a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4f489d2d8890467db8677e1650bbdcff, fd27a28d3a7e461a9bf9b94258edc38a; no voters:
I20251215 21:02:22.404707 1846 raft_consensus.cc:2804] T 50980bcbdf6d4e0aa449ef0fed4fabd6 P fd27a28d3a7e461a9bf9b94258edc38a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251215 21:02:22.404781 1846 raft_consensus.cc:493] T 50980bcbdf6d4e0aa449ef0fed4fabd6 P fd27a28d3a7e461a9bf9b94258edc38a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 21:02:22.404831 1846 raft_consensus.cc:3060] T 50980bcbdf6d4e0aa449ef0fed4fabd6 P fd27a28d3a7e461a9bf9b94258edc38a [term 0 FOLLOWER]: Advancing to term 1
I20251215 21:02:22.405756 1846 raft_consensus.cc:515] T 50980bcbdf6d4e0aa449ef0fed4fabd6 P fd27a28d3a7e461a9bf9b94258edc38a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea1619b8a1b741b7bb84dd4fdc76a5d9" member_type: VOTER last_known_addr { host: "127.28.113.3" port: 41093 } } peers { permanent_uuid: "fd27a28d3a7e461a9bf9b94258edc38a" member_type: VOTER last_known_addr { host: "127.28.113.1" port: 35243 } } peers { permanent_uuid: "4f489d2d8890467db8677e1650bbdcff" member_type: VOTER last_known_addr { host: "127.28.113.2" port: 33527 } }
I20251215 21:02:22.406096 1846 leader_election.cc:290] T 50980bcbdf6d4e0aa449ef0fed4fabd6 P fd27a28d3a7e461a9bf9b94258edc38a [CANDIDATE]: Term 1 election: Requested vote from peers ea1619b8a1b741b7bb84dd4fdc76a5d9 (127.28.113.3:41093), 4f489d2d8890467db8677e1650bbdcff (127.28.113.2:33527)
I20251215 21:02:22.406433 1775 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "50980bcbdf6d4e0aa449ef0fed4fabd6" candidate_uuid: "fd27a28d3a7e461a9bf9b94258edc38a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea1619b8a1b741b7bb84dd4fdc76a5d9"
I20251215 21:02:22.406433 1640 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "50980bcbdf6d4e0aa449ef0fed4fabd6" candidate_uuid: "fd27a28d3a7e461a9bf9b94258edc38a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f489d2d8890467db8677e1650bbdcff"
I20251215 21:02:22.406545 1775 raft_consensus.cc:3060] T 50980bcbdf6d4e0aa449ef0fed4fabd6 P ea1619b8a1b741b7bb84dd4fdc76a5d9 [term 0 FOLLOWER]: Advancing to term 1
I20251215 21:02:22.406545 1640 raft_consensus.cc:3060] T 50980bcbdf6d4e0aa449ef0fed4fabd6 P 4f489d2d8890467db8677e1650bbdcff [term 0 FOLLOWER]: Advancing to term 1
I20251215 21:02:22.407660 1640 raft_consensus.cc:2468] T 50980bcbdf6d4e0aa449ef0fed4fabd6 P 4f489d2d8890467db8677e1650bbdcff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd27a28d3a7e461a9bf9b94258edc38a in term 1.
I20251215 21:02:22.407660 1775 raft_consensus.cc:2468] T 50980bcbdf6d4e0aa449ef0fed4fabd6 P ea1619b8a1b741b7bb84dd4fdc76a5d9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd27a28d3a7e461a9bf9b94258edc38a in term 1.
I20251215 21:02:22.407944 1439 leader_election.cc:304] T 50980bcbdf6d4e0aa449ef0fed4fabd6 P fd27a28d3a7e461a9bf9b94258edc38a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4f489d2d8890467db8677e1650bbdcff, fd27a28d3a7e461a9bf9b94258edc38a; no voters:
I20251215 21:02:22.408134 1846 raft_consensus.cc:2804] T 50980bcbdf6d4e0aa449ef0fed4fabd6 P fd27a28d3a7e461a9bf9b94258edc38a [term 1 FOLLOWER]: Leader election won for term 1
I20251215 21:02:22.408320 1846 raft_consensus.cc:697] T 50980bcbdf6d4e0aa449ef0fed4fabd6 P fd27a28d3a7e461a9bf9b94258edc38a [term 1 LEADER]: Becoming Leader. State: Replica: fd27a28d3a7e461a9bf9b94258edc38a, State: Running, Role: LEADER
I20251215 21:02:22.408622 1846 consensus_queue.cc:237] T 50980bcbdf6d4e0aa449ef0fed4fabd6 P fd27a28d3a7e461a9bf9b94258edc38a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea1619b8a1b741b7bb84dd4fdc76a5d9" member_type: VOTER last_known_addr { host: "127.28.113.3" port: 41093 } } peers { permanent_uuid: "fd27a28d3a7e461a9bf9b94258edc38a" member_type: VOTER last_known_addr { host: "127.28.113.1" port: 35243 } } peers { permanent_uuid: "4f489d2d8890467db8677e1650bbdcff" member_type: VOTER last_known_addr { host: "127.28.113.2" port: 33527 } }
I20251215 21:02:22.411413 1355 catalog_manager.cc:5654] T 50980bcbdf6d4e0aa449ef0fed4fabd6 P fd27a28d3a7e461a9bf9b94258edc38a reported cstate change: term changed from 0 to 1, leader changed from <none> to fd27a28d3a7e461a9bf9b94258edc38a (127.28.113.1). New cstate: current_term: 1 leader_uuid: "fd27a28d3a7e461a9bf9b94258edc38a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea1619b8a1b741b7bb84dd4fdc76a5d9" member_type: VOTER last_known_addr { host: "127.28.113.3" port: 41093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fd27a28d3a7e461a9bf9b94258edc38a" member_type: VOTER last_known_addr { host: "127.28.113.1" port: 35243 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4f489d2d8890467db8677e1650bbdcff" member_type: VOTER last_known_addr { host: "127.28.113.2" port: 33527 } health_report { overall_health: UNKNOWN } } }
I20251215 21:02:22.438258 1835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:02:22.435307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52410 (local address 127.28.113.1:35243)
1215 21:02:22.435457 (+ 150us) server_negotiation.cc:207] Beginning negotiation
1215 21:02:22.435462 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1215 21:02:22.435475 (+ 13us) server_negotiation.cc:408] Connection header received
1215 21:02:22.435554 (+ 79us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1215 21:02:22.435557 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1215 21:02:22.435611 (+ 54us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1215 21:02:22.435692 (+ 81us) server_negotiation.cc:227] Negotiated authn=TOKEN
1215 21:02:22.436042 (+ 350us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:02:22.436533 (+ 491us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1215 21:02:22.437181 (+ 648us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:02:22.437358 (+ 177us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:02:22.437456 (+ 98us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1215 21:02:22.437891 (+ 435us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1215 21:02:22.438000 (+ 109us) server_negotiation.cc:1036] Waiting for connection context
1215 21:02:22.438073 (+ 73us) server_negotiation.cc:300] Negotiation successful
1215 21:02:22.438126 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20251215 21:02:22.443506 1640 raft_consensus.cc:1275] T 50980bcbdf6d4e0aa449ef0fed4fabd6 P 4f489d2d8890467db8677e1650bbdcff [term 1 FOLLOWER]: Refusing update from remote peer fd27a28d3a7e461a9bf9b94258edc38a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251215 21:02:22.443508 1775 raft_consensus.cc:1275] T 50980bcbdf6d4e0aa449ef0fed4fabd6 P ea1619b8a1b741b7bb84dd4fdc76a5d9 [term 1 FOLLOWER]: Refusing update from remote peer fd27a28d3a7e461a9bf9b94258edc38a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251215 21:02:22.444125 1849 consensus_queue.cc:1048] T 50980bcbdf6d4e0aa449ef0fed4fabd6 P fd27a28d3a7e461a9bf9b94258edc38a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea1619b8a1b741b7bb84dd4fdc76a5d9" member_type: VOTER last_known_addr { host: "127.28.113.3" port: 41093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251215 21:02:22.444275 1846 consensus_queue.cc:1048] T 50980bcbdf6d4e0aa449ef0fed4fabd6 P fd27a28d3a7e461a9bf9b94258edc38a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f489d2d8890467db8677e1650bbdcff" member_type: VOTER last_known_addr { host: "127.28.113.2" port: 33527 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251215 21:02:22.451530 1854 mvcc.cc:204] Tried to move back new op lower bound from 7232850093841670144 to 7232850093708324864. Current Snapshot: MvccSnapshot[applied={T|T < 7232850093841670144}]
I20251215 21:02:22.453598 1853 mvcc.cc:204] Tried to move back new op lower bound from 7232850093841670144 to 7232850093708324864. Current Snapshot: MvccSnapshot[applied={T|T < 7232850093841670144}]
I20251215 21:02:22.457370 1352 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53674:
table { table_name: "test-table" } modify_external_catalogs: true
I20251215 21:02:22.457508 1352 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53674:
table { table_name: "test-table" } modify_external_catalogs: true
I20251215 21:02:22.459532 1352 catalog_manager.cc:5941] T 00000000000000000000000000000000 P fd6bc7e478b64e228ee5f45174d6b417: Sending DeleteTablet for 3 replicas of tablet 50980bcbdf6d4e0aa449ef0fed4fabd6
I20251215 21:02:22.460196 1755 tablet_service.cc:1552] Processing DeleteTablet for tablet 50980bcbdf6d4e0aa449ef0fed4fabd6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-15 21:02:22 UTC) from {username='kudu', principal='kudu/127.28.113.62@KRBTEST.COM'} at 127.0.0.1:53838
I20251215 21:02:22.460282 1620 tablet_service.cc:1552] Processing DeleteTablet for tablet 50980bcbdf6d4e0aa449ef0fed4fabd6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-15 21:02:22 UTC) from {username='kudu', principal='kudu/127.28.113.62@KRBTEST.COM'} at 127.0.0.1:36258
I20251215 21:02:22.460393 1485 tablet_service.cc:1552] Processing DeleteTablet for tablet 50980bcbdf6d4e0aa449ef0fed4fabd6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-15 21:02:22 UTC) from {username='kudu', principal='kudu/127.28.113.62@KRBTEST.COM'} at 127.0.0.1:52396
I20251215 21:02:22.460726 1863 tablet_replica.cc:333] T 50980bcbdf6d4e0aa449ef0fed4fabd6 P 4f489d2d8890467db8677e1650bbdcff: stopping tablet replica
I20251215 21:02:22.460753 1862 tablet_replica.cc:333] T 50980bcbdf6d4e0aa449ef0fed4fabd6 P ea1619b8a1b741b7bb84dd4fdc76a5d9: stopping tablet replica
I20251215 21:02:22.460970 1862 raft_consensus.cc:2243] T 50980bcbdf6d4e0aa449ef0fed4fabd6 P ea1619b8a1b741b7bb84dd4fdc76a5d9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251215 21:02:22.460970 1863 raft_consensus.cc:2243] T 50980bcbdf6d4e0aa449ef0fed4fabd6 P 4f489d2d8890467db8677e1650bbdcff [term 1 FOLLOWER]: Raft consensus shutting down.
I20251215 21:02:22.461264 1863 raft_consensus.cc:2272] T 50980bcbdf6d4e0aa449ef0fed4fabd6 P 4f489d2d8890467db8677e1650bbdcff [term 1 FOLLOWER]: Raft consensus is shut down!
I20251215 21:02:22.461323 29124 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu with pid 1423
I20251215 21:02:22.462309 1862 raft_consensus.cc:2272] T 50980bcbdf6d4e0aa449ef0fed4fabd6 P ea1619b8a1b741b7bb84dd4fdc76a5d9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251215 21:02:22.463516 1863 ts_tablet_manager.cc:1916] T 50980bcbdf6d4e0aa449ef0fed4fabd6 P 4f489d2d8890467db8677e1650bbdcff: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251215 21:02:22.463644 1862 ts_tablet_manager.cc:1916] T 50980bcbdf6d4e0aa449ef0fed4fabd6 P ea1619b8a1b741b7bb84dd4fdc76a5d9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251215 21:02:22.465970 1863 ts_tablet_manager.cc:1929] T 50980bcbdf6d4e0aa449ef0fed4fabd6 P 4f489d2d8890467db8677e1650bbdcff: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251215 21:02:22.465970 1862 ts_tablet_manager.cc:1929] T 50980bcbdf6d4e0aa449ef0fed4fabd6 P ea1619b8a1b741b7bb84dd4fdc76a5d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251215 21:02:22.466070 1862 log.cc:1199] T 50980bcbdf6d4e0aa449ef0fed4fabd6 P ea1619b8a1b741b7bb84dd4fdc76a5d9: Deleting WAL directory at /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765832506145157-29124-0/minicluster-data/ts-2/wal/wals/50980bcbdf6d4e0aa449ef0fed4fabd6
I20251215 21:02:22.466070 1863 log.cc:1199] T 50980bcbdf6d4e0aa449ef0fed4fabd6 P 4f489d2d8890467db8677e1650bbdcff: Deleting WAL directory at /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765832506145157-29124-0/minicluster-data/ts-1/wal/wals/50980bcbdf6d4e0aa449ef0fed4fabd6
I20251215 21:02:22.466399 1862 ts_tablet_manager.cc:1950] T 50980bcbdf6d4e0aa449ef0fed4fabd6 P ea1619b8a1b741b7bb84dd4fdc76a5d9: Deleting consensus metadata
I20251215 21:02:22.466399 1863 ts_tablet_manager.cc:1950] T 50980bcbdf6d4e0aa449ef0fed4fabd6 P 4f489d2d8890467db8677e1650bbdcff: Deleting consensus metadata
I20251215 21:02:22.467615 1344 catalog_manager.cc:4985] TS 4f489d2d8890467db8677e1650bbdcff (127.28.113.2:33527): tablet 50980bcbdf6d4e0aa449ef0fed4fabd6 (table test-table [id=dcbe1f9781a742e49e2ebd1e9d815ef4]) successfully deleted
I20251215 21:02:22.467682 1345 catalog_manager.cc:4985] TS ea1619b8a1b741b7bb84dd4fdc76a5d9 (127.28.113.3:41093): tablet 50980bcbdf6d4e0aa449ef0fed4fabd6 (table test-table [id=dcbe1f9781a742e49e2ebd1e9d815ef4]) successfully deleted
W20251215 21:02:22.468194 1344 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.28.113.1:35243 (error 108)
I20251215 21:02:22.468315 29124 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu with pid 1558
I20251215 21:02:22.469426 1836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:02:22.469186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.113.1:35243 (local address 127.0.0.1:52412)
1215 21:02:22.469307 (+ 121us) negotiation.cc:107] Waiting for socket to connect
1215 21:02:22.469359 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.113.1:35243: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20251215 21:02:22.469537 1344 catalog_manager.cc:4712] TS fd27a28d3a7e461a9bf9b94258edc38a (127.28.113.1:35243): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 50980bcbdf6d4e0aa449ef0fed4fabd6: Network error: Client connection negotiation failed: client connection to 127.28.113.1:35243: connect: Connection refused (error 111)
I20251215 21:02:22.473927 29124 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu with pid 1693
I20251215 21:02:22.478241 29124 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu with pid 1328
2025-12-15T21:02:22Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (3040 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2025-12-15T21:02:22Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-12-15T21:02:22Z Disabled control of system clock
I20251215 21:02:22.504976 29124 external_mini_cluster.cc:1366] Running /tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu
/tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1765832506145157-29124-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1765832506145157-29124-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1765832506145157-29124-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1765832506145157-29124-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.113.62:46529
--webserver_interface=127.28.113.62
--webserver_port=0
--builtin_ntp_servers=127.28.113.20:45381
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.113.62:46529
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20251215 21:02:22.611215 1869 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251215 21:02:22.611444 1869 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251215 21:02:22.611490 1869 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251215 21:02:22.614733 1869 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251215 21:02:22.614796 1869 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251215 21:02:22.614811 1869 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251215 21:02:22.614835 1869 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251215 21:02:22.614850 1869 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251215 21:02:22.618633 1869 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.113.20:45381
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1765832506145157-29124-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1765832506145157-29124-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.113.62:46529
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.113.62:46529
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1765832506145157-29124-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.113.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1869
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1765832506145157-29124-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision dc741732bb6a40213e11a13869fad06a49a209a3
build type DEBUG
built by None at 15 Dec 2025 20:43:14 UTC on 24a791456cd2
build id 9530
I20251215 21:02:22.619663 1869 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251215 21:02:22.620774 1869 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251215 21:02:22.626019 1874 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251215 21:02:22.626107 1869 server_base.cc:1047] running on GCE node
W20251215 21:02:22.626030 1877 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251215 21:02:22.626053 1875 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251215 21:02:22.626696 1869 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251215 21:02:22.627524 1869 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251215 21:02:22.628664 1869 hybrid_clock.cc:648] HybridClock initialized: now 1765832542628646 us; error 35 us; skew 500 ppm
I20251215 21:02:22.630362 1869 webserver.cc:492] Webserver started at http://127.28.113.62:46621/ using document root <none> and password file <none>
I20251215 21:02:22.630872 1869 fs_manager.cc:362] Metadata directory not provided
I20251215 21:02:22.630924 1869 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251215 21:02:22.631095 1869 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251215 21:02:22.632701 1869 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1765832506145157-29124-0/minicluster-data/master-0/data/instance:
uuid: "63060321ad3e48e3bdb23dbe3909d483"
format_stamp: "Formatted at 2025-12-15 21:02:22 on dist-test-slave-bjsf"
server_key: "8163742500f8f3f651146d4c10ffe596"
server_key_iv: "a8b4684fc157942aae63cad259f3e611"
server_key_version: "encryptionkey@0"
I20251215 21:02:22.633132 1869 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1765832506145157-29124-0/minicluster-data/master-0/wal/instance:
uuid: "63060321ad3e48e3bdb23dbe3909d483"
format_stamp: "Formatted at 2025-12-15 21:02:22 on dist-test-slave-bjsf"
server_key: "8163742500f8f3f651146d4c10ffe596"
server_key_iv: "a8b4684fc157942aae63cad259f3e611"
server_key_version: "encryptionkey@0"
I20251215 21:02:22.636197 1869 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20251215 21:02:22.638271 1883 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251215 21:02:22.639235 1869 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20251215 21:02:22.639339 1869 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1765832506145157-29124-0/minicluster-data/master-0/data,/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1765832506145157-29124-0/minicluster-data/master-0/wal
uuid: "63060321ad3e48e3bdb23dbe3909d483"
format_stamp: "Formatted at 2025-12-15 21:02:22 on dist-test-slave-bjsf"
server_key: "8163742500f8f3f651146d4c10ffe596"
server_key_iv: "a8b4684fc157942aae63cad259f3e611"
server_key_version: "encryptionkey@0"
I20251215 21:02:22.639420 1869 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1765832506145157-29124-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1765832506145157-29124-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1765832506145157-29124-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251215 21:02:22.643707 1869 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251215 21:02:22.644233 1869 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251215 21:02:22.644366 1869 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251215 21:02:22.650637 1869 rpc_server.cc:307] RPC server started. Bound to: 127.28.113.62:46529
I20251215 21:02:22.650660 1935 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.28.113.62:46529 every 8 connection(s)
I20251215 21:02:22.651532 1869 server_base.cc:1179] Dumped server information to /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1765832506145157-29124-0/minicluster-data/master-0/data/info.pb
I20251215 21:02:22.654160 1936 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 21:02:22.659382 1936 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 63060321ad3e48e3bdb23dbe3909d483: Bootstrap starting.
I20251215 21:02:22.659462 29124 external_mini_cluster.cc:1428] Started /tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu as pid 1869
I20251215 21:02:22.659575 29124 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1765832506145157-29124-0/minicluster-data/master-0/wal/instance
I20251215 21:02:22.659787 29124 external_mini_cluster.cc:1468] Setting key ab495e0f2ad2d9dc7b3e47663ad5cfbc
I20251215 21:02:22.661890 1936 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 63060321ad3e48e3bdb23dbe3909d483: Neither blocks nor log segments found. Creating new log.
I20251215 21:02:22.662755 1936 log.cc:826] T 00000000000000000000000000000000 P 63060321ad3e48e3bdb23dbe3909d483: Log is configured to *not* fsync() on all Append() calls
I20251215 21:02:22.664664 1936 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 63060321ad3e48e3bdb23dbe3909d483: No bootstrap required, opened a new log
I20251215 21:02:22.665002 1939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:02:22.660978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34068 (local address 127.28.113.62:46529)
1215 21:02:22.661653 (+ 675us) server_negotiation.cc:207] Beginning negotiation
1215 21:02:22.661664 (+ 11us) server_negotiation.cc:400] Waiting for connection header
1215 21:02:22.661706 (+ 42us) server_negotiation.cc:408] Connection header received
1215 21:02:22.662363 (+ 657us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1215 21:02:22.662386 (+ 23us) server_negotiation.cc:462] Received NEGOTIATE request from client
1215 21:02:22.662788 (+ 402us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1215 21:02:22.663131 (+ 343us) server_negotiation.cc:227] Negotiated authn=SASL
1215 21:02:22.663889 (+ 758us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1215 21:02:22.663894 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1215 21:02:22.663902 (+ 8us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1215 21:02:22.663912 (+ 10us) server_negotiation.cc:923] calling sasl_server_start()
1215 21:02:22.663950 (+ 38us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1215 21:02:22.663986 (+ 36us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1215 21:02:22.664084 (+ 98us) server_negotiation.cc:1036] Waiting for connection context
1215 21:02:22.664202 (+ 118us) server_negotiation.cc:300] Negotiation successful
1215 21:02:22.664424 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":268,"thread_start_us":110,"threads_started":1}
I20251215 21:02:22.667253 1936 raft_consensus.cc:359] T 00000000000000000000000000000000 P 63060321ad3e48e3bdb23dbe3909d483 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63060321ad3e48e3bdb23dbe3909d483" member_type: VOTER last_known_addr { host: "127.28.113.62" port: 46529 } }
I20251215 21:02:22.667421 1936 raft_consensus.cc:385] T 00000000000000000000000000000000 P 63060321ad3e48e3bdb23dbe3909d483 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251215 21:02:22.667485 1936 raft_consensus.cc:740] T 00000000000000000000000000000000 P 63060321ad3e48e3bdb23dbe3909d483 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63060321ad3e48e3bdb23dbe3909d483, State: Initialized, Role: FOLLOWER
I20251215 21:02:22.667840 1936 consensus_queue.cc:260] T 00000000000000000000000000000000 P 63060321ad3e48e3bdb23dbe3909d483 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63060321ad3e48e3bdb23dbe3909d483" member_type: VOTER last_known_addr { host: "127.28.113.62" port: 46529 } }
I20251215 21:02:22.667950 1936 raft_consensus.cc:399] T 00000000000000000000000000000000 P 63060321ad3e48e3bdb23dbe3909d483 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251215 21:02:22.668032 1936 raft_consensus.cc:493] T 00000000000000000000000000000000 P 63060321ad3e48e3bdb23dbe3909d483 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251215 21:02:22.668118 1936 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 63060321ad3e48e3bdb23dbe3909d483 [term 0 FOLLOWER]: Advancing to term 1
I20251215 21:02:22.668887 1936 raft_consensus.cc:515] T 00000000000000000000000000000000 P 63060321ad3e48e3bdb23dbe3909d483 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63060321ad3e48e3bdb23dbe3909d483" member_type: VOTER last_known_addr { host: "127.28.113.62" port: 46529 } }
I20251215 21:02:22.669186 1936 leader_election.cc:304] T 00000000000000000000000000000000 P 63060321ad3e48e3bdb23dbe3909d483 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 63060321ad3e48e3bdb23dbe3909d483; no voters:
I20251215 21:02:22.669427 1936 leader_election.cc:290] T 00000000000000000000000000000000 P 63060321ad3e48e3bdb23dbe3909d483 [CANDIDATE]: Term 1 election: Requested vote from peers
I20251215 21:02:22.669523 1941 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 63060321ad3e48e3bdb23dbe3909d483 [term 1 FOLLOWER]: Leader election won for term 1
I20251215 21:02:22.669710 1941 raft_consensus.cc:697] T 00000000000000000000000000000000 P 63060321ad3e48e3bdb23dbe3909d483 [term 1 LEADER]: Becoming Leader. State: Replica: 63060321ad3e48e3bdb23dbe3909d483, State: Running, Role: LEADER
I20251215 21:02:22.669982 1941 consensus_queue.cc:237] T 00000000000000000000000000000000 P 63060321ad3e48e3bdb23dbe3909d483 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63060321ad3e48e3bdb23dbe3909d483" member_type: VOTER last_known_addr { host: "127.28.113.62" port: 46529 } }
I20251215 21:02:22.670444 1936 sys_catalog.cc:565] T 00000000000000000000000000000000 P 63060321ad3e48e3bdb23dbe3909d483 [sys.catalog]: configured and running, proceeding with master startup.
I20251215 21:02:22.671546 1942 sys_catalog.cc:455] T 00000000000000000000000000000000 P 63060321ad3e48e3bdb23dbe3909d483 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "63060321ad3e48e3bdb23dbe3909d483" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63060321ad3e48e3bdb23dbe3909d483" member_type: VOTER last_known_addr { host: "127.28.113.62" port: 46529 } } }
I20251215 21:02:22.671653 1942 sys_catalog.cc:458] T 00000000000000000000000000000000 P 63060321ad3e48e3bdb23dbe3909d483 [sys.catalog]: This master's current role is: LEADER
I20251215 21:02:22.672024 1943 sys_catalog.cc:455] T 00000000000000000000000000000000 P 63060321ad3e48e3bdb23dbe3909d483 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 63060321ad3e48e3bdb23dbe3909d483. Latest consensus state: current_term: 1 leader_uuid: "63060321ad3e48e3bdb23dbe3909d483" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63060321ad3e48e3bdb23dbe3909d483" member_type: VOTER last_known_addr { host: "127.28.113.62" port: 46529 } } }
I20251215 21:02:22.672107 1943 sys_catalog.cc:458] T 00000000000000000000000000000000 P 63060321ad3e48e3bdb23dbe3909d483 [sys.catalog]: This master's current role is: LEADER
I20251215 21:02:22.672669 1950 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251215 21:02:22.675437 1950 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251215 21:02:22.680342 1950 catalog_manager.cc:1357] Generated new cluster ID: 0e80575cf9bd480ba3d4f01fe398d5f6
I20251215 21:02:22.680408 1950 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251215 21:02:22.690419 1950 catalog_manager.cc:1380] Generated new certificate authority record
I20251215 21:02:22.690757 1950 catalog_manager.cc:1514] Loading token signing keys...
I20251215 21:02:22.707608 1950 catalog_manager.cc:6027] T 00000000000000000000000000000000 P 63060321ad3e48e3bdb23dbe3909d483: Generated new TSK 0
I20251215 21:02:22.708500 1950 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251215 21:02:22.713558 29124 external_mini_cluster.cc:1366] Running /tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu
/tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1765832506145157-29124-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1765832506145157-29124-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1765832506145157-29124-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1765832506145157-29124-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.113.1:0
--local_ip_for_outbound_sockets=127.28.113.1
--webserver_interface=127.28.113.1
--webserver_port=0
--tserver_master_addrs=127.28.113.62:46529
--builtin_ntp_servers=127.28.113.20:45381
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20251215 21:02:22.813114 1960 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251215 21:02:22.813318 1960 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251215 21:02:22.813367 1960 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251215 21:02:22.816551 1960 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251215 21:02:22.816602 1960 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251215 21:02:22.816679 1960 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.113.1
I20251215 21:02:22.820351 1960 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.113.20:45381
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1765832506145157-29124-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1765832506145157-29124-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.113.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1765832506145157-29124-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.113.1
--webserver_port=0
--tserver_master_addrs=127.28.113.62:46529
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1960
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.113.1
--log_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1765832506145157-29124-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision dc741732bb6a40213e11a13869fad06a49a209a3
build type DEBUG
built by None at 15 Dec 2025 20:43:14 UTC on 24a791456cd2
build id 9530
I20251215 21:02:22.821370 1960 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251215 21:02:22.822451 1960 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251215 21:02:22.829044 1966 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251215 21:02:22.829196 1960 server_base.cc:1047] running on GCE node
W20251215 21:02:22.829032 1965 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251215 21:02:22.829028 1968 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251215 21:02:22.829798 1960 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251215 21:02:22.830307 1960 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251215 21:02:22.831451 1960 hybrid_clock.cc:648] HybridClock initialized: now 1765832542831418 us; error 54 us; skew 500 ppm
I20251215 21:02:22.833102 1960 webserver.cc:492] Webserver started at http://127.28.113.1:45647/ using document root <none> and password file <none>
I20251215 21:02:22.833614 1960 fs_manager.cc:362] Metadata directory not provided
I20251215 21:02:22.833669 1960 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251215 21:02:22.833832 1960 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251215 21:02:22.835402 1960 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1765832506145157-29124-0/minicluster-data/ts-0/data/instance:
uuid: "aba26dcf9c534ea4acf32b81b5b169d9"
format_stamp: "Formatted at 2025-12-15 21:02:22 on dist-test-slave-bjsf"
server_key: "d8cdfcf3d4b9494ee0862c21d2704d71"
server_key_iv: "17bac326e36cb6b43c363a78a186bfdc"
server_key_version: "encryptionkey@0"
I20251215 21:02:22.835819 1960 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1765832506145157-29124-0/minicluster-data/ts-0/wal/instance:
uuid: "aba26dcf9c534ea4acf32b81b5b169d9"
format_stamp: "Formatted at 2025-12-15 21:02:22 on dist-test-slave-bjsf"
server_key: "d8cdfcf3d4b9494ee0862c21d2704d71"
server_key_iv: "17bac326e36cb6b43c363a78a186bfdc"
server_key_version: "encryptionkey@0"
I20251215 21:02:22.839005 1960 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.001s sys 0.001s
I20251215 21:02:22.841022 1974 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251215 21:02:22.841949 1960 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20251215 21:02:22.842044 1960 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1765832506145157-29124-0/minicluster-data/ts-0/data,/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1765832506145157-29124-0/minicluster-data/ts-0/wal
uuid: "aba26dcf9c534ea4acf32b81b5b169d9"
format_stamp: "Formatted at 2025-12-15 21:02:22 on dist-test-slave-bjsf"
server_key: "d8cdfcf3d4b9494ee0862c21d2704d71"
server_key_iv: "17bac326e36cb6b43c363a78a186bfdc"
server_key_version: "encryptionkey@0"
I20251215 21:02:22.842123 1960 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1765832506145157-29124-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1765832506145157-29124-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1765832506145157-29124-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251215 21:02:22.846789 1960 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251215 21:02:22.847338 1960 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251215 21:02:22.847481 1960 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251215 21:02:22.847985 1960 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251215 21:02:22.848876 1960 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251215 21:02:22.848932 1960 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251215 21:02:22.848979 1960 ts_tablet_manager.cc:616] Registered 0 tablets
I20251215 21:02:22.849002 1960 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251215 21:02:22.857016 1960 rpc_server.cc:307] RPC server started. Bound to: 127.28.113.1:34333
I20251215 21:02:22.857049 2087 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.28.113.1:34333 every 8 connection(s)
I20251215 21:02:22.857870 1960 server_base.cc:1179] Dumped server information to /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1765832506145157-29124-0/minicluster-data/ts-0/data/info.pb
I20251215 21:02:22.858222 29124 external_mini_cluster.cc:1428] Started /tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu as pid 1960
I20251215 21:02:22.858337 29124 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1765832506145157-29124-0/minicluster-data/ts-0/wal/instance
I20251215 21:02:22.858608 29124 external_mini_cluster.cc:1468] Setting key f2e7d6d9fe936364caac060bf85a675b
I20251215 21:02:22.860613 29124 external_mini_cluster.cc:1366] Running /tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu
/tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1765832506145157-29124-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1765832506145157-29124-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1765832506145157-29124-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1765832506145157-29124-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.113.2:0
--local_ip_for_outbound_sockets=127.28.113.2
--webserver_interface=127.28.113.2
--webserver_port=0
--tserver_master_addrs=127.28.113.62:46529
--builtin_ntp_servers=127.28.113.20:45381
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20251215 21:02:22.863301 1939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:02:22.859551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.113.1:53941 (local address 127.28.113.62:46529)
1215 21:02:22.859691 (+ 140us) server_negotiation.cc:207] Beginning negotiation
1215 21:02:22.859696 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1215 21:02:22.860774 (+ 1078us) server_negotiation.cc:408] Connection header received
1215 21:02:22.861729 (+ 955us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1215 21:02:22.861733 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1215 21:02:22.861769 (+ 36us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1215 21:02:22.861851 (+ 82us) server_negotiation.cc:227] Negotiated authn=SASL
1215 21:02:22.862933 (+ 1082us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1215 21:02:22.862936 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1215 21:02:22.862938 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1215 21:02:22.862947 (+ 9us) server_negotiation.cc:923] calling sasl_server_start()
1215 21:02:22.862966 (+ 19us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1215 21:02:22.862973 (+ 7us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1215 21:02:22.863035 (+ 62us) server_negotiation.cc:1036] Waiting for connection context
1215 21:02:22.863173 (+ 138us) server_negotiation.cc:300] Negotiation successful
1215 21:02:22.863203 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20251215 21:02:22.864192 2090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:02:22.859885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.113.62:46529 (local address 127.28.113.1:53941)
1215 21:02:22.860640 (+ 755us) negotiation.cc:107] Waiting for socket to connect
1215 21:02:22.860684 (+ 44us) client_negotiation.cc:174] Beginning negotiation
1215 21:02:22.861522 (+ 838us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1215 21:02:22.862016 (+ 494us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1215 21:02:22.862027 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
1215 21:02:22.862119 (+ 92us) client_negotiation.cc:189] Negotiated authn=SASL
1215 21:02:22.862684 (+ 565us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
1215 21:02:22.862686 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
1215 21:02:22.862725 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
1215 21:02:22.862790 (+ 65us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1215 21:02:22.863039 (+ 249us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1215 21:02:22.863046 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1215 21:02:22.863050 (+ 4us) client_negotiation.cc:769] Sending connection context
1215 21:02:22.863239 (+ 189us) client_negotiation.cc:240] Negotiation successful
1215 21:02:22.863436 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":109,"threads_started":1}
I20251215 21:02:22.865247 2088 heartbeater.cc:344] Connected to a master server at 127.28.113.62:46529
I20251215 21:02:22.865490 2088 heartbeater.cc:461] Registering TS with master...
I20251215 21:02:22.865949 2088 heartbeater.cc:507] Master 127.28.113.62:46529 requested a full tablet report, sending...
I20251215 21:02:22.867340 1900 ts_manager.cc:194] Registered new tserver with Master: aba26dcf9c534ea4acf32b81b5b169d9 (127.28.113.1:34333)
W20251215 21:02:22.963934 2091 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251215 21:02:22.964190 2091 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251215 21:02:22.964248 2091 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251215 21:02:22.967437 2091 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251215 21:02:22.967489 2091 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251215 21:02:22.967576 2091 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.113.2
I20251215 21:02:22.971210 2091 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.113.20:45381
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1765832506145157-29124-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1765832506145157-29124-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.113.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1765832506145157-29124-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.113.2
--webserver_port=0
--tserver_master_addrs=127.28.113.62:46529
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2091
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.113.2
--log_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1765832506145157-29124-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision dc741732bb6a40213e11a13869fad06a49a209a3
build type DEBUG
built by None at 15 Dec 2025 20:43:14 UTC on 24a791456cd2
build id 9530
I20251215 21:02:22.972239 2091 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251215 21:02:22.973292 2091 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251215 21:02:22.979307 2096 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251215 21:02:22.979310 2097 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251215 21:02:22.979461 2091 server_base.cc:1047] running on GCE node
W20251215 21:02:22.979300 2099 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251215 21:02:22.979877 2091 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251215 21:02:22.980434 2091 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251215 21:02:22.981575 2091 hybrid_clock.cc:648] HybridClock initialized: now 1765832542981559 us; error 32 us; skew 500 ppm
I20251215 21:02:22.983253 2091 webserver.cc:492] Webserver started at http://127.28.113.2:46109/ using document root <none> and password file <none>
I20251215 21:02:22.983785 2091 fs_manager.cc:362] Metadata directory not provided
I20251215 21:02:22.983840 2091 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251215 21:02:22.984052 2091 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251215 21:02:22.985695 2091 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1765832506145157-29124-0/minicluster-data/ts-1/data/instance:
uuid: "bb55fdf9d26e4b58a34f78c5cb9b58e5"
format_stamp: "Formatted at 2025-12-15 21:02:22 on dist-test-slave-bjsf"
server_key: "ec35687593ed9a6cf6d7d25dfe040656"
server_key_iv: "32441eb4cac9e27adbbaa83dca0f842e"
server_key_version: "encryptionkey@0"
I20251215 21:02:22.986133 2091 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1765832506145157-29124-0/minicluster-data/ts-1/wal/instance:
uuid: "bb55fdf9d26e4b58a34f78c5cb9b58e5"
format_stamp: "Formatted at 2025-12-15 21:02:22 on dist-test-slave-bjsf"
server_key: "ec35687593ed9a6cf6d7d25dfe040656"
server_key_iv: "32441eb4cac9e27adbbaa83dca0f842e"
server_key_version: "encryptionkey@0"
I20251215 21:02:22.989250 2091 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20251215 21:02:22.991263 2105 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251215 21:02:22.992236 2091 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20251215 21:02:22.992336 2091 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1765832506145157-29124-0/minicluster-data/ts-1/data,/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1765832506145157-29124-0/minicluster-data/ts-1/wal
uuid: "bb55fdf9d26e4b58a34f78c5cb9b58e5"
format_stamp: "Formatted at 2025-12-15 21:02:22 on dist-test-slave-bjsf"
server_key: "ec35687593ed9a6cf6d7d25dfe040656"
server_key_iv: "32441eb4cac9e27adbbaa83dca0f842e"
server_key_version: "encryptionkey@0"
I20251215 21:02:22.992416 2091 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1765832506145157-29124-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1765832506145157-29124-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1765832506145157-29124-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251215 21:02:22.996898 2091 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251215 21:02:22.997423 2091 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251215 21:02:22.997565 2091 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251215 21:02:22.998037 2091 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251215 21:02:22.998901 2091 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251215 21:02:22.998948 2091 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251215 21:02:22.998996 2091 ts_tablet_manager.cc:616] Registered 0 tablets
I20251215 21:02:22.999022 2091 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251215 21:02:23.007547 2091 rpc_server.cc:307] RPC server started. Bound to: 127.28.113.2:45811
I20251215 21:02:23.007586 2218 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.28.113.2:45811 every 8 connection(s)
I20251215 21:02:23.008456 2091 server_base.cc:1179] Dumped server information to /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1765832506145157-29124-0/minicluster-data/ts-1/data/info.pb
I20251215 21:02:23.013389 1939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:02:23.010097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.113.2:59885 (local address 127.28.113.62:46529)
1215 21:02:23.010242 (+ 145us) server_negotiation.cc:207] Beginning negotiation
1215 21:02:23.010247 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1215 21:02:23.011088 (+ 841us) server_negotiation.cc:408] Connection header received
1215 21:02:23.011811 (+ 723us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1215 21:02:23.011814 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1215 21:02:23.011851 (+ 37us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1215 21:02:23.011930 (+ 79us) server_negotiation.cc:227] Negotiated authn=SASL
1215 21:02:23.012982 (+ 1052us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1215 21:02:23.012985 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1215 21:02:23.012987 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1215 21:02:23.012997 (+ 10us) server_negotiation.cc:923] calling sasl_server_start()
1215 21:02:23.013015 (+ 18us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1215 21:02:23.013023 (+ 8us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1215 21:02:23.013101 (+ 78us) server_negotiation.cc:1036] Waiting for connection context
1215 21:02:23.013253 (+ 152us) server_negotiation.cc:300] Negotiation successful
1215 21:02:23.013285 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20251215 21:02:23.014067 2221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:02:23.010367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.113.62:46529 (local address 127.28.113.2:59885)
1215 21:02:23.010959 (+ 592us) negotiation.cc:107] Waiting for socket to connect
1215 21:02:23.010993 (+ 34us) client_negotiation.cc:174] Beginning negotiation
1215 21:02:23.011647 (+ 654us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1215 21:02:23.012081 (+ 434us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1215 21:02:23.012089 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
1215 21:02:23.012169 (+ 80us) client_negotiation.cc:189] Negotiated authn=SASL
1215 21:02:23.012746 (+ 577us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
1215 21:02:23.012747 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
1215 21:02:23.012777 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
1215 21:02:23.012828 (+ 51us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1215 21:02:23.013114 (+ 286us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1215 21:02:23.013122 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1215 21:02:23.013126 (+ 4us) client_negotiation.cc:769] Sending connection context
1215 21:02:23.013302 (+ 176us) client_negotiation.cc:240] Negotiation successful
1215 21:02:23.013464 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":91,"threads_started":1}
I20251215 21:02:23.015107 2219 heartbeater.cc:344] Connected to a master server at 127.28.113.62:46529
I20251215 21:02:23.015297 2219 heartbeater.cc:461] Registering TS with master...
I20251215 21:02:23.015653 2219 heartbeater.cc:507] Master 127.28.113.62:46529 requested a full tablet report, sending...
I20251215 21:02:23.015799 29124 external_mini_cluster.cc:1428] Started /tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu as pid 2091
I20251215 21:02:23.015903 29124 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1765832506145157-29124-0/minicluster-data/ts-1/wal/instance
I20251215 21:02:23.016145 29124 external_mini_cluster.cc:1468] Setting key c61f425fb9c7b046dcfdf877d42e2c7c
I20251215 21:02:23.016594 1900 ts_manager.cc:194] Registered new tserver with Master: bb55fdf9d26e4b58a34f78c5cb9b58e5 (127.28.113.2:45811)
I20251215 21:02:23.018008 29124 external_mini_cluster.cc:1366] Running /tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu
/tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1765832506145157-29124-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1765832506145157-29124-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1765832506145157-29124-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1765832506145157-29124-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.113.3:0
--local_ip_for_outbound_sockets=127.28.113.3
--webserver_interface=127.28.113.3
--webserver_port=0
--tserver_master_addrs=127.28.113.62:46529
--builtin_ntp_servers=127.28.113.20:45381
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20251215 21:02:23.121984 2222 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251215 21:02:23.122253 2222 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251215 21:02:23.122342 2222 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251215 21:02:23.126691 2222 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251215 21:02:23.126754 2222 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251215 21:02:23.126854 2222 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.113.3
I20251215 21:02:23.131883 2222 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.113.20:45381
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1765832506145157-29124-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1765832506145157-29124-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.113.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1765832506145157-29124-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.113.3
--webserver_port=0
--tserver_master_addrs=127.28.113.62:46529
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2222
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.113.3
--log_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1765832506145157-29124-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision dc741732bb6a40213e11a13869fad06a49a209a3
build type DEBUG
built by None at 15 Dec 2025 20:43:14 UTC on 24a791456cd2
build id 9530
I20251215 21:02:23.133224 2222 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251215 21:02:23.134649 2222 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251215 21:02:23.142176 2227 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251215 21:02:23.142275 2228 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251215 21:02:23.142176 2230 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251215 21:02:23.142437 2222 server_base.cc:1047] running on GCE node
I20251215 21:02:23.142901 2222 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251215 21:02:23.143540 2222 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251215 21:02:23.144732 2222 hybrid_clock.cc:648] HybridClock initialized: now 1765832543144716 us; error 30 us; skew 500 ppm
I20251215 21:02:23.146806 2222 webserver.cc:492] Webserver started at http://127.28.113.3:34905/ using document root <none> and password file <none>
I20251215 21:02:23.147475 2222 fs_manager.cc:362] Metadata directory not provided
I20251215 21:02:23.147552 2222 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251215 21:02:23.147778 2222 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251215 21:02:23.149945 2222 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1765832506145157-29124-0/minicluster-data/ts-2/data/instance:
uuid: "dd475f46eef34091aa6005dbad4acaea"
format_stamp: "Formatted at 2025-12-15 21:02:23 on dist-test-slave-bjsf"
server_key: "065a5bc9af8b08a890f480e076f47773"
server_key_iv: "5afc415405ec8987fa2161bf413cc21b"
server_key_version: "encryptionkey@0"
I20251215 21:02:23.150493 2222 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1765832506145157-29124-0/minicluster-data/ts-2/wal/instance:
uuid: "dd475f46eef34091aa6005dbad4acaea"
format_stamp: "Formatted at 2025-12-15 21:02:23 on dist-test-slave-bjsf"
server_key: "065a5bc9af8b08a890f480e076f47773"
server_key_iv: "5afc415405ec8987fa2161bf413cc21b"
server_key_version: "encryptionkey@0"
I20251215 21:02:23.154670 2222 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20251215 21:02:23.156632 2236 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251215 21:02:23.157578 2222 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20251215 21:02:23.157701 2222 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1765832506145157-29124-0/minicluster-data/ts-2/data,/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1765832506145157-29124-0/minicluster-data/ts-2/wal
uuid: "dd475f46eef34091aa6005dbad4acaea"
format_stamp: "Formatted at 2025-12-15 21:02:23 on dist-test-slave-bjsf"
server_key: "065a5bc9af8b08a890f480e076f47773"
server_key_iv: "5afc415405ec8987fa2161bf413cc21b"
server_key_version: "encryptionkey@0"
I20251215 21:02:23.157794 2222 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1765832506145157-29124-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1765832506145157-29124-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1765832506145157-29124-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251215 21:02:23.162210 2222 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251215 21:02:23.162742 2222 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251215 21:02:23.162882 2222 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251215 21:02:23.163339 2222 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251215 21:02:23.164235 2222 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251215 21:02:23.164283 2222 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251215 21:02:23.164331 2222 ts_tablet_manager.cc:616] Registered 0 tablets
I20251215 21:02:23.164355 2222 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251215 21:02:23.172719 2222 rpc_server.cc:307] RPC server started. Bound to: 127.28.113.3:41603
I20251215 21:02:23.172760 2349 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.28.113.3:41603 every 8 connection(s)
I20251215 21:02:23.173570 2222 server_base.cc:1179] Dumped server information to /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1765832506145157-29124-0/minicluster-data/ts-2/data/info.pb
I20251215 21:02:23.178273 1939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:02:23.175081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.113.3:59675 (local address 127.28.113.62:46529)
1215 21:02:23.175195 (+ 114us) server_negotiation.cc:207] Beginning negotiation
1215 21:02:23.175199 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1215 21:02:23.176059 (+ 860us) server_negotiation.cc:408] Connection header received
1215 21:02:23.176815 (+ 756us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1215 21:02:23.176819 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1215 21:02:23.176863 (+ 44us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1215 21:02:23.176936 (+ 73us) server_negotiation.cc:227] Negotiated authn=SASL
1215 21:02:23.177855 (+ 919us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1215 21:02:23.177859 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1215 21:02:23.177862 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1215 21:02:23.177874 (+ 12us) server_negotiation.cc:923] calling sasl_server_start()
1215 21:02:23.177894 (+ 20us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1215 21:02:23.177902 (+ 8us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1215 21:02:23.177977 (+ 75us) server_negotiation.cc:1036] Waiting for connection context
1215 21:02:23.178102 (+ 125us) server_negotiation.cc:300] Negotiation successful
1215 21:02:23.178143 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20251215 21:02:23.178977 2352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:02:23.175336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.113.62:46529 (local address 127.28.113.3:59675)
1215 21:02:23.175918 (+ 582us) negotiation.cc:107] Waiting for socket to connect
1215 21:02:23.175951 (+ 33us) client_negotiation.cc:174] Beginning negotiation
1215 21:02:23.176634 (+ 683us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1215 21:02:23.177080 (+ 446us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1215 21:02:23.177088 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
1215 21:02:23.177165 (+ 77us) client_negotiation.cc:189] Negotiated authn=SASL
1215 21:02:23.177656 (+ 491us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
1215 21:02:23.177657 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
1215 21:02:23.177684 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
1215 21:02:23.177732 (+ 48us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1215 21:02:23.177970 (+ 238us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1215 21:02:23.177978 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1215 21:02:23.177983 (+ 5us) client_negotiation.cc:769] Sending connection context
1215 21:02:23.178160 (+ 177us) client_negotiation.cc:240] Negotiation successful
1215 21:02:23.178421 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":90,"threads_started":1}
I20251215 21:02:23.179872 2350 heartbeater.cc:344] Connected to a master server at 127.28.113.62:46529
I20251215 21:02:23.180101 2350 heartbeater.cc:461] Registering TS with master...
I20251215 21:02:23.180459 2350 heartbeater.cc:507] Master 127.28.113.62:46529 requested a full tablet report, sending...
I20251215 21:02:23.181326 1899 ts_manager.cc:194] Registered new tserver with Master: dd475f46eef34091aa6005dbad4acaea (127.28.113.3:41603)
I20251215 21:02:23.182732 29124 external_mini_cluster.cc:1428] Started /tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu as pid 2222
I20251215 21:02:23.182847 29124 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1765832506145157-29124-0/minicluster-data/ts-2/wal/instance
I20251215 21:02:23.183096 29124 external_mini_cluster.cc:1468] Setting key 2c7071e385a12282badeaaca5cde5d59
I20251215 21:02:23.184890 29124 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20251215 21:02:23.188598 1939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:02:23.186937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34084 (local address 127.28.113.62:46529)
1215 21:02:23.187075 (+ 138us) server_negotiation.cc:207] Beginning negotiation
1215 21:02:23.187078 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1215 21:02:23.187171 (+ 93us) server_negotiation.cc:408] Connection header received
1215 21:02:23.187321 (+ 150us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1215 21:02:23.187324 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1215 21:02:23.187364 (+ 40us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1215 21:02:23.187431 (+ 67us) server_negotiation.cc:227] Negotiated authn=SASL
1215 21:02:23.188230 (+ 799us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1215 21:02:23.188233 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1215 21:02:23.188235 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1215 21:02:23.188248 (+ 13us) server_negotiation.cc:923] calling sasl_server_start()
1215 21:02:23.188271 (+ 23us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1215 21:02:23.188280 (+ 9us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1215 21:02:23.188364 (+ 84us) server_negotiation.cc:1036] Waiting for connection context
1215 21:02:23.188415 (+ 51us) server_negotiation.cc:300] Negotiation successful
1215 21:02:23.188447 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20251215 21:02:23.190994 1899 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34084:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251215 21:02:23.193328 1899 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251215 21:02:23.203604 2358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:02:23.200773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.113.2:45811 (local address 127.0.0.1:47558)
1215 21:02:23.201136 (+ 363us) negotiation.cc:107] Waiting for socket to connect
1215 21:02:23.201173 (+ 37us) client_negotiation.cc:174] Beginning negotiation
1215 21:02:23.201274 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1215 21:02:23.201977 (+ 703us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1215 21:02:23.201983 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
1215 21:02:23.202047 (+ 64us) client_negotiation.cc:189] Negotiated authn=SASL
1215 21:02:23.202828 (+ 781us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
1215 21:02:23.202830 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
1215 21:02:23.202866 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
1215 21:02:23.202932 (+ 66us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1215 21:02:23.203275 (+ 343us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1215 21:02:23.203284 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1215 21:02:23.203290 (+ 6us) client_negotiation.cc:769] Sending connection context
1215 21:02:23.203405 (+ 115us) client_negotiation.cc:240] Negotiation successful
1215 21:02:23.203463 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":112,"threads_started":1}
I20251215 21:02:23.203667 2359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:02:23.200922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47558 (local address 127.28.113.2:45811)
1215 21:02:23.201296 (+ 374us) server_negotiation.cc:207] Beginning negotiation
1215 21:02:23.201303 (+ 7us) server_negotiation.cc:400] Waiting for connection header
1215 21:02:23.201327 (+ 24us) server_negotiation.cc:408] Connection header received
1215 21:02:23.201384 (+ 57us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1215 21:02:23.201389 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1215 21:02:23.201798 (+ 409us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1215 21:02:23.201931 (+ 133us) server_negotiation.cc:227] Negotiated authn=SASL
1215 21:02:23.203087 (+ 1156us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1215 21:02:23.203094 (+ 7us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1215 21:02:23.203104 (+ 10us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1215 21:02:23.203117 (+ 13us) server_negotiation.cc:923] calling sasl_server_start()
1215 21:02:23.203172 (+ 55us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1215 21:02:23.203187 (+ 15us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1215 21:02:23.203298 (+ 111us) server_negotiation.cc:1036] Waiting for connection context
1215 21:02:23.203439 (+ 141us) server_negotiation.cc:300] Negotiation successful
1215 21:02:23.203508 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":279,"thread_start_us":114,"threads_started":1}
I20251215 21:02:23.203842 2360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:02:23.201098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.113.3:41603 (local address 127.0.0.1:34766)
1215 21:02:23.201600 (+ 502us) negotiation.cc:107] Waiting for socket to connect
1215 21:02:23.201613 (+ 13us) client_negotiation.cc:174] Beginning negotiation
1215 21:02:23.201675 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1215 21:02:23.202274 (+ 599us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1215 21:02:23.202278 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
1215 21:02:23.202292 (+ 14us) client_negotiation.cc:189] Negotiated authn=SASL
1215 21:02:23.203044 (+ 752us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
1215 21:02:23.203045 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
1215 21:02:23.203057 (+ 12us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
1215 21:02:23.203082 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1215 21:02:23.203645 (+ 563us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1215 21:02:23.203648 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1215 21:02:23.203651 (+ 3us) client_negotiation.cc:769] Sending connection context
1215 21:02:23.203705 (+ 54us) client_negotiation.cc:240] Negotiation successful
1215 21:02:23.203735 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":439,"thread_start_us":61,"threads_started":1}
I20251215 21:02:23.204348 2362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:02:23.201530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34766 (local address 127.28.113.3:41603)
1215 21:02:23.201917 (+ 387us) server_negotiation.cc:207] Beginning negotiation
1215 21:02:23.201923 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1215 21:02:23.201944 (+ 21us) server_negotiation.cc:408] Connection header received
1215 21:02:23.201992 (+ 48us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1215 21:02:23.201997 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1215 21:02:23.202125 (+ 128us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1215 21:02:23.202470 (+ 345us) server_negotiation.cc:227] Negotiated authn=SASL
1215 21:02:23.203462 (+ 992us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1215 21:02:23.203467 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1215 21:02:23.203476 (+ 9us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1215 21:02:23.203487 (+ 11us) server_negotiation.cc:923] calling sasl_server_start()
1215 21:02:23.203532 (+ 45us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1215 21:02:23.203543 (+ 11us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1215 21:02:23.204122 (+ 579us) server_negotiation.cc:1036] Waiting for connection context
1215 21:02:23.204160 (+ 38us) server_negotiation.cc:300] Negotiation successful
1215 21:02:23.204216 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":276,"thread_start_us":153,"threads_started":1}
I20251215 21:02:23.205437 2361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:02:23.201476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.113.1:34333 (local address 127.0.0.1:44620)
1215 21:02:23.202636 (+ 1160us) negotiation.cc:107] Waiting for socket to connect
1215 21:02:23.202648 (+ 12us) client_negotiation.cc:174] Beginning negotiation
1215 21:02:23.202741 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1215 21:02:23.203782 (+ 1041us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1215 21:02:23.203785 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1215 21:02:23.203801 (+ 16us) client_negotiation.cc:189] Negotiated authn=SASL
1215 21:02:23.204868 (+ 1067us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
1215 21:02:23.204870 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
1215 21:02:23.204879 (+ 9us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
1215 21:02:23.204900 (+ 21us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1215 21:02:23.205228 (+ 328us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1215 21:02:23.205232 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1215 21:02:23.205235 (+ 3us) client_negotiation.cc:769] Sending connection context
1215 21:02:23.205294 (+ 59us) client_negotiation.cc:240] Negotiation successful
1215 21:02:23.205325 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1093,"thread_start_us":62,"threads_started":1}
I20251215 21:02:23.206024 2363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:02:23.201834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44620 (local address 127.28.113.1:34333)
1215 21:02:23.202442 (+ 608us) server_negotiation.cc:207] Beginning negotiation
1215 21:02:23.202449 (+ 7us) server_negotiation.cc:400] Waiting for connection header
1215 21:02:23.203334 (+ 885us) server_negotiation.cc:408] Connection header received
1215 21:02:23.203395 (+ 61us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1215 21:02:23.203400 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1215 21:02:23.203536 (+ 136us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1215 21:02:23.204099 (+ 563us) server_negotiation.cc:227] Negotiated authn=SASL
1215 21:02:23.205034 (+ 935us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1215 21:02:23.205040 (+ 6us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1215 21:02:23.205048 (+ 8us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1215 21:02:23.205061 (+ 13us) server_negotiation.cc:923] calling sasl_server_start()
1215 21:02:23.205112 (+ 51us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1215 21:02:23.205124 (+ 12us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1215 21:02:23.205762 (+ 638us) server_negotiation.cc:1036] Waiting for connection context
1215 21:02:23.205814 (+ 52us) server_negotiation.cc:300] Negotiation successful
1215 21:02:23.205873 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":503,"thread_start_us":101,"threads_started":1}
I20251215 21:02:23.207047 2284 tablet_service.cc:1505] Processing CreateTablet for tablet 74e1c0daac65441281f2bb73960e2498 (DEFAULT_TABLE table=test-table [id=f3e5e7fe8dec4ec8940d1087034d218a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251215 21:02:23.207034 2153 tablet_service.cc:1505] Processing CreateTablet for tablet 74e1c0daac65441281f2bb73960e2498 (DEFAULT_TABLE table=test-table [id=f3e5e7fe8dec4ec8940d1087034d218a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251215 21:02:23.207823 2284 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 74e1c0daac65441281f2bb73960e2498. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 21:02:23.207914 2153 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 74e1c0daac65441281f2bb73960e2498. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 21:02:23.208411 2022 tablet_service.cc:1505] Processing CreateTablet for tablet 74e1c0daac65441281f2bb73960e2498 (DEFAULT_TABLE table=test-table [id=f3e5e7fe8dec4ec8940d1087034d218a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251215 21:02:23.209218 2022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 74e1c0daac65441281f2bb73960e2498. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 21:02:23.212879 2364 tablet_bootstrap.cc:492] T 74e1c0daac65441281f2bb73960e2498 P dd475f46eef34091aa6005dbad4acaea: Bootstrap starting.
I20251215 21:02:23.214078 2365 tablet_bootstrap.cc:492] T 74e1c0daac65441281f2bb73960e2498 P bb55fdf9d26e4b58a34f78c5cb9b58e5: Bootstrap starting.
I20251215 21:02:23.214897 2366 tablet_bootstrap.cc:492] T 74e1c0daac65441281f2bb73960e2498 P aba26dcf9c534ea4acf32b81b5b169d9: Bootstrap starting.
I20251215 21:02:23.214924 2364 tablet_bootstrap.cc:654] T 74e1c0daac65441281f2bb73960e2498 P dd475f46eef34091aa6005dbad4acaea: Neither blocks nor log segments found. Creating new log.
I20251215 21:02:23.215739 2364 log.cc:826] T 74e1c0daac65441281f2bb73960e2498 P dd475f46eef34091aa6005dbad4acaea: Log is configured to *not* fsync() on all Append() calls
I20251215 21:02:23.216024 2365 tablet_bootstrap.cc:654] T 74e1c0daac65441281f2bb73960e2498 P bb55fdf9d26e4b58a34f78c5cb9b58e5: Neither blocks nor log segments found. Creating new log.
I20251215 21:02:23.216733 2365 log.cc:826] T 74e1c0daac65441281f2bb73960e2498 P bb55fdf9d26e4b58a34f78c5cb9b58e5: Log is configured to *not* fsync() on all Append() calls
I20251215 21:02:23.216989 2366 tablet_bootstrap.cc:654] T 74e1c0daac65441281f2bb73960e2498 P aba26dcf9c534ea4acf32b81b5b169d9: Neither blocks nor log segments found. Creating new log.
I20251215 21:02:23.217556 2364 tablet_bootstrap.cc:492] T 74e1c0daac65441281f2bb73960e2498 P dd475f46eef34091aa6005dbad4acaea: No bootstrap required, opened a new log
I20251215 21:02:23.217690 2366 log.cc:826] T 74e1c0daac65441281f2bb73960e2498 P aba26dcf9c534ea4acf32b81b5b169d9: Log is configured to *not* fsync() on all Append() calls
I20251215 21:02:23.217736 2364 ts_tablet_manager.cc:1403] T 74e1c0daac65441281f2bb73960e2498 P dd475f46eef34091aa6005dbad4acaea: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20251215 21:02:23.218256 2365 tablet_bootstrap.cc:492] T 74e1c0daac65441281f2bb73960e2498 P bb55fdf9d26e4b58a34f78c5cb9b58e5: No bootstrap required, opened a new log
I20251215 21:02:23.218433 2365 ts_tablet_manager.cc:1403] T 74e1c0daac65441281f2bb73960e2498 P bb55fdf9d26e4b58a34f78c5cb9b58e5: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20251215 21:02:23.219336 2366 tablet_bootstrap.cc:492] T 74e1c0daac65441281f2bb73960e2498 P aba26dcf9c534ea4acf32b81b5b169d9: No bootstrap required, opened a new log
I20251215 21:02:23.219501 2366 ts_tablet_manager.cc:1403] T 74e1c0daac65441281f2bb73960e2498 P aba26dcf9c534ea4acf32b81b5b169d9: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20251215 21:02:23.220603 2364 raft_consensus.cc:359] T 74e1c0daac65441281f2bb73960e2498 P dd475f46eef34091aa6005dbad4acaea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb55fdf9d26e4b58a34f78c5cb9b58e5" member_type: VOTER last_known_addr { host: "127.28.113.2" port: 45811 } } peers { permanent_uuid: "aba26dcf9c534ea4acf32b81b5b169d9" member_type: VOTER last_known_addr { host: "127.28.113.1" port: 34333 } } peers { permanent_uuid: "dd475f46eef34091aa6005dbad4acaea" member_type: VOTER last_known_addr { host: "127.28.113.3" port: 41603 } }
I20251215 21:02:23.220803 2364 raft_consensus.cc:385] T 74e1c0daac65441281f2bb73960e2498 P dd475f46eef34091aa6005dbad4acaea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251215 21:02:23.220849 2364 raft_consensus.cc:740] T 74e1c0daac65441281f2bb73960e2498 P dd475f46eef34091aa6005dbad4acaea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd475f46eef34091aa6005dbad4acaea, State: Initialized, Role: FOLLOWER
I20251215 21:02:23.220750 2365 raft_consensus.cc:359] T 74e1c0daac65441281f2bb73960e2498 P bb55fdf9d26e4b58a34f78c5cb9b58e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb55fdf9d26e4b58a34f78c5cb9b58e5" member_type: VOTER last_known_addr { host: "127.28.113.2" port: 45811 } } peers { permanent_uuid: "aba26dcf9c534ea4acf32b81b5b169d9" member_type: VOTER last_known_addr { host: "127.28.113.1" port: 34333 } } peers { permanent_uuid: "dd475f46eef34091aa6005dbad4acaea" member_type: VOTER last_known_addr { host: "127.28.113.3" port: 41603 } }
I20251215 21:02:23.220955 2365 raft_consensus.cc:385] T 74e1c0daac65441281f2bb73960e2498 P bb55fdf9d26e4b58a34f78c5cb9b58e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251215 21:02:23.221016 2365 raft_consensus.cc:740] T 74e1c0daac65441281f2bb73960e2498 P bb55fdf9d26e4b58a34f78c5cb9b58e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb55fdf9d26e4b58a34f78c5cb9b58e5, State: Initialized, Role: FOLLOWER
I20251215 21:02:23.221362 2364 consensus_queue.cc:260] T 74e1c0daac65441281f2bb73960e2498 P dd475f46eef34091aa6005dbad4acaea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb55fdf9d26e4b58a34f78c5cb9b58e5" member_type: VOTER last_known_addr { host: "127.28.113.2" port: 45811 } } peers { permanent_uuid: "aba26dcf9c534ea4acf32b81b5b169d9" member_type: VOTER last_known_addr { host: "127.28.113.1" port: 34333 } } peers { permanent_uuid: "dd475f46eef34091aa6005dbad4acaea" member_type: VOTER last_known_addr { host: "127.28.113.3" port: 41603 } }
I20251215 21:02:23.221398 2365 consensus_queue.cc:260] T 74e1c0daac65441281f2bb73960e2498 P bb55fdf9d26e4b58a34f78c5cb9b58e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb55fdf9d26e4b58a34f78c5cb9b58e5" member_type: VOTER last_known_addr { host: "127.28.113.2" port: 45811 } } peers { permanent_uuid: "aba26dcf9c534ea4acf32b81b5b169d9" member_type: VOTER last_known_addr { host: "127.28.113.1" port: 34333 } } peers { permanent_uuid: "dd475f46eef34091aa6005dbad4acaea" member_type: VOTER last_known_addr { host: "127.28.113.3" port: 41603 } }
I20251215 21:02:23.222002 2350 heartbeater.cc:499] Master 127.28.113.62:46529 was elected leader, sending a full tablet report...
I20251215 21:02:23.222172 2365 ts_tablet_manager.cc:1434] T 74e1c0daac65441281f2bb73960e2498 P bb55fdf9d26e4b58a34f78c5cb9b58e5: Time spent starting tablet: real 0.004s user 0.004s sys 0.001s
I20251215 21:02:23.222325 2366 raft_consensus.cc:359] T 74e1c0daac65441281f2bb73960e2498 P aba26dcf9c534ea4acf32b81b5b169d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb55fdf9d26e4b58a34f78c5cb9b58e5" member_type: VOTER last_known_addr { host: "127.28.113.2" port: 45811 } } peers { permanent_uuid: "aba26dcf9c534ea4acf32b81b5b169d9" member_type: VOTER last_known_addr { host: "127.28.113.1" port: 34333 } } peers { permanent_uuid: "dd475f46eef34091aa6005dbad4acaea" member_type: VOTER last_known_addr { host: "127.28.113.3" port: 41603 } }
I20251215 21:02:23.222468 2219 heartbeater.cc:499] Master 127.28.113.62:46529 was elected leader, sending a full tablet report...
I20251215 21:02:23.222532 2366 raft_consensus.cc:385] T 74e1c0daac65441281f2bb73960e2498 P aba26dcf9c534ea4acf32b81b5b169d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251215 21:02:23.222592 2366 raft_consensus.cc:740] T 74e1c0daac65441281f2bb73960e2498 P aba26dcf9c534ea4acf32b81b5b169d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aba26dcf9c534ea4acf32b81b5b169d9, State: Initialized, Role: FOLLOWER
I20251215 21:02:23.223238 2364 ts_tablet_manager.cc:1434] T 74e1c0daac65441281f2bb73960e2498 P dd475f46eef34091aa6005dbad4acaea: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20251215 21:02:23.223228 2366 consensus_queue.cc:260] T 74e1c0daac65441281f2bb73960e2498 P aba26dcf9c534ea4acf32b81b5b169d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb55fdf9d26e4b58a34f78c5cb9b58e5" member_type: VOTER last_known_addr { host: "127.28.113.2" port: 45811 } } peers { permanent_uuid: "aba26dcf9c534ea4acf32b81b5b169d9" member_type: VOTER last_known_addr { host: "127.28.113.1" port: 34333 } } peers { permanent_uuid: "dd475f46eef34091aa6005dbad4acaea" member_type: VOTER last_known_addr { host: "127.28.113.3" port: 41603 } }
I20251215 21:02:23.223843 2088 heartbeater.cc:499] Master 127.28.113.62:46529 was elected leader, sending a full tablet report...
I20251215 21:02:23.223982 2366 ts_tablet_manager.cc:1434] T 74e1c0daac65441281f2bb73960e2498 P aba26dcf9c534ea4acf32b81b5b169d9: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
W20251215 21:02:23.259346 2220 tablet.cc:2378] T 74e1c0daac65441281f2bb73960e2498 P bb55fdf9d26e4b58a34f78c5cb9b58e5: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20251215 21:02:23.358858 2089 tablet.cc:2378] T 74e1c0daac65441281f2bb73960e2498 P aba26dcf9c534ea4acf32b81b5b169d9: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20251215 21:02:23.424386 2351 tablet.cc:2378] T 74e1c0daac65441281f2bb73960e2498 P dd475f46eef34091aa6005dbad4acaea: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20251215 21:02:23.524538 2372 raft_consensus.cc:493] T 74e1c0daac65441281f2bb73960e2498 P aba26dcf9c534ea4acf32b81b5b169d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 21:02:23.524736 2372 raft_consensus.cc:515] T 74e1c0daac65441281f2bb73960e2498 P aba26dcf9c534ea4acf32b81b5b169d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb55fdf9d26e4b58a34f78c5cb9b58e5" member_type: VOTER last_known_addr { host: "127.28.113.2" port: 45811 } } peers { permanent_uuid: "aba26dcf9c534ea4acf32b81b5b169d9" member_type: VOTER last_known_addr { host: "127.28.113.1" port: 34333 } } peers { permanent_uuid: "dd475f46eef34091aa6005dbad4acaea" member_type: VOTER last_known_addr { host: "127.28.113.3" port: 41603 } }
I20251215 21:02:23.525802 2372 leader_election.cc:290] T 74e1c0daac65441281f2bb73960e2498 P aba26dcf9c534ea4acf32b81b5b169d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bb55fdf9d26e4b58a34f78c5cb9b58e5 (127.28.113.2:45811), dd475f46eef34091aa6005dbad4acaea (127.28.113.3:41603)
I20251215 21:02:23.527771 2362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:02:23.526203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.113.1:52577 (local address 127.28.113.3:41603)
1215 21:02:23.526341 (+ 138us) server_negotiation.cc:207] Beginning negotiation
1215 21:02:23.526346 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1215 21:02:23.526501 (+ 155us) server_negotiation.cc:408] Connection header received
1215 21:02:23.526591 (+ 90us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1215 21:02:23.526595 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1215 21:02:23.526666 (+ 71us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1215 21:02:23.526746 (+ 80us) server_negotiation.cc:227] Negotiated authn=SASL
1215 21:02:23.527426 (+ 680us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1215 21:02:23.527429 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1215 21:02:23.527431 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1215 21:02:23.527440 (+ 9us) server_negotiation.cc:923] calling sasl_server_start()
1215 21:02:23.527458 (+ 18us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1215 21:02:23.527465 (+ 7us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1215 21:02:23.527536 (+ 71us) server_negotiation.cc:1036] Waiting for connection context
1215 21:02:23.527614 (+ 78us) server_negotiation.cc:300] Negotiation successful
1215 21:02:23.527647 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20251215 21:02:23.527801 2374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:02:23.526087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.113.3:41603 (local address 127.28.113.1:52577)
1215 21:02:23.526416 (+ 329us) negotiation.cc:107] Waiting for socket to connect
1215 21:02:23.526428 (+ 12us) client_negotiation.cc:174] Beginning negotiation
1215 21:02:23.526516 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1215 21:02:23.526777 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1215 21:02:23.526780 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1215 21:02:23.526795 (+ 15us) client_negotiation.cc:189] Negotiated authn=SASL
1215 21:02:23.527289 (+ 494us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
1215 21:02:23.527291 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
1215 21:02:23.527300 (+ 9us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
1215 21:02:23.527320 (+ 20us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1215 21:02:23.527531 (+ 211us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1215 21:02:23.527534 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1215 21:02:23.527537 (+ 3us) client_negotiation.cc:769] Sending connection context
1215 21:02:23.527608 (+ 71us) client_negotiation.cc:240] Negotiation successful
1215 21:02:23.527644 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":99,"threads_started":1}
I20251215 21:02:23.528381 2304 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "74e1c0daac65441281f2bb73960e2498" candidate_uuid: "aba26dcf9c534ea4acf32b81b5b169d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd475f46eef34091aa6005dbad4acaea" is_pre_election: true
I20251215 21:02:23.528541 2359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:02:23.526212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.113.1:55451 (local address 127.28.113.2:45811)
1215 21:02:23.526341 (+ 129us) server_negotiation.cc:207] Beginning negotiation
1215 21:02:23.526345 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1215 21:02:23.526591 (+ 246us) server_negotiation.cc:408] Connection header received
1215 21:02:23.526695 (+ 104us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1215 21:02:23.526697 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1215 21:02:23.526734 (+ 37us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1215 21:02:23.526812 (+ 78us) server_negotiation.cc:227] Negotiated authn=SASL
1215 21:02:23.528176 (+ 1364us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1215 21:02:23.528179 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1215 21:02:23.528182 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1215 21:02:23.528194 (+ 12us) server_negotiation.cc:923] calling sasl_server_start()
1215 21:02:23.528217 (+ 23us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1215 21:02:23.528226 (+ 9us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1215 21:02:23.528310 (+ 84us) server_negotiation.cc:1036] Waiting for connection context
1215 21:02:23.528395 (+ 85us) server_negotiation.cc:300] Negotiation successful
1215 21:02:23.528435 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20251215 21:02:23.528570 2373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:02:23.526131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.113.2:45811 (local address 127.28.113.1:55451)
1215 21:02:23.526480 (+ 349us) negotiation.cc:107] Waiting for socket to connect
1215 21:02:23.526494 (+ 14us) client_negotiation.cc:174] Beginning negotiation
1215 21:02:23.526598 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1215 21:02:23.526848 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1215 21:02:23.526850 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1215 21:02:23.526869 (+ 19us) client_negotiation.cc:189] Negotiated authn=SASL
1215 21:02:23.528004 (+ 1135us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
1215 21:02:23.528007 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
1215 21:02:23.528021 (+ 14us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
1215 21:02:23.528047 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1215 21:02:23.528307 (+ 260us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1215 21:02:23.528310 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1215 21:02:23.528313 (+ 3us) client_negotiation.cc:769] Sending connection context
1215 21:02:23.528382 (+ 69us) client_negotiation.cc:240] Negotiation successful
1215 21:02:23.528410 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":84,"threads_started":1}
I20251215 21:02:23.528649 2304 raft_consensus.cc:2468] T 74e1c0daac65441281f2bb73960e2498 P dd475f46eef34091aa6005dbad4acaea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aba26dcf9c534ea4acf32b81b5b169d9 in term 0.
I20251215 21:02:23.529099 2173 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "74e1c0daac65441281f2bb73960e2498" candidate_uuid: "aba26dcf9c534ea4acf32b81b5b169d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb55fdf9d26e4b58a34f78c5cb9b58e5" is_pre_election: true
I20251215 21:02:23.529100 1978 leader_election.cc:304] T 74e1c0daac65441281f2bb73960e2498 P aba26dcf9c534ea4acf32b81b5b169d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: aba26dcf9c534ea4acf32b81b5b169d9, dd475f46eef34091aa6005dbad4acaea; no voters:
I20251215 21:02:23.529323 2173 raft_consensus.cc:2468] T 74e1c0daac65441281f2bb73960e2498 P bb55fdf9d26e4b58a34f78c5cb9b58e5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aba26dcf9c534ea4acf32b81b5b169d9 in term 0.
I20251215 21:02:23.529350 2372 raft_consensus.cc:2804] T 74e1c0daac65441281f2bb73960e2498 P aba26dcf9c534ea4acf32b81b5b169d9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251215 21:02:23.529429 2372 raft_consensus.cc:493] T 74e1c0daac65441281f2bb73960e2498 P aba26dcf9c534ea4acf32b81b5b169d9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 21:02:23.529484 2372 raft_consensus.cc:3060] T 74e1c0daac65441281f2bb73960e2498 P aba26dcf9c534ea4acf32b81b5b169d9 [term 0 FOLLOWER]: Advancing to term 1
I20251215 21:02:23.530470 2372 raft_consensus.cc:515] T 74e1c0daac65441281f2bb73960e2498 P aba26dcf9c534ea4acf32b81b5b169d9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb55fdf9d26e4b58a34f78c5cb9b58e5" member_type: VOTER last_known_addr { host: "127.28.113.2" port: 45811 } } peers { permanent_uuid: "aba26dcf9c534ea4acf32b81b5b169d9" member_type: VOTER last_known_addr { host: "127.28.113.1" port: 34333 } } peers { permanent_uuid: "dd475f46eef34091aa6005dbad4acaea" member_type: VOTER last_known_addr { host: "127.28.113.3" port: 41603 } }
I20251215 21:02:23.530826 2372 leader_election.cc:290] T 74e1c0daac65441281f2bb73960e2498 P aba26dcf9c534ea4acf32b81b5b169d9 [CANDIDATE]: Term 1 election: Requested vote from peers bb55fdf9d26e4b58a34f78c5cb9b58e5 (127.28.113.2:45811), dd475f46eef34091aa6005dbad4acaea (127.28.113.3:41603)
I20251215 21:02:23.531143 2173 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "74e1c0daac65441281f2bb73960e2498" candidate_uuid: "aba26dcf9c534ea4acf32b81b5b169d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb55fdf9d26e4b58a34f78c5cb9b58e5"
I20251215 21:02:23.531143 2304 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "74e1c0daac65441281f2bb73960e2498" candidate_uuid: "aba26dcf9c534ea4acf32b81b5b169d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd475f46eef34091aa6005dbad4acaea"
I20251215 21:02:23.531268 2304 raft_consensus.cc:3060] T 74e1c0daac65441281f2bb73960e2498 P dd475f46eef34091aa6005dbad4acaea [term 0 FOLLOWER]: Advancing to term 1
I20251215 21:02:23.531267 2173 raft_consensus.cc:3060] T 74e1c0daac65441281f2bb73960e2498 P bb55fdf9d26e4b58a34f78c5cb9b58e5 [term 0 FOLLOWER]: Advancing to term 1
I20251215 21:02:23.532538 2173 raft_consensus.cc:2468] T 74e1c0daac65441281f2bb73960e2498 P bb55fdf9d26e4b58a34f78c5cb9b58e5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aba26dcf9c534ea4acf32b81b5b169d9 in term 1.
I20251215 21:02:23.532861 1977 leader_election.cc:304] T 74e1c0daac65441281f2bb73960e2498 P aba26dcf9c534ea4acf32b81b5b169d9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: aba26dcf9c534ea4acf32b81b5b169d9, bb55fdf9d26e4b58a34f78c5cb9b58e5; no voters:
I20251215 21:02:23.533023 2304 raft_consensus.cc:2468] T 74e1c0daac65441281f2bb73960e2498 P dd475f46eef34091aa6005dbad4acaea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aba26dcf9c534ea4acf32b81b5b169d9 in term 1.
I20251215 21:02:23.533035 2372 raft_consensus.cc:2804] T 74e1c0daac65441281f2bb73960e2498 P aba26dcf9c534ea4acf32b81b5b169d9 [term 1 FOLLOWER]: Leader election won for term 1
I20251215 21:02:23.533246 2372 raft_consensus.cc:697] T 74e1c0daac65441281f2bb73960e2498 P aba26dcf9c534ea4acf32b81b5b169d9 [term 1 LEADER]: Becoming Leader. State: Replica: aba26dcf9c534ea4acf32b81b5b169d9, State: Running, Role: LEADER
I20251215 21:02:23.533586 2372 consensus_queue.cc:237] T 74e1c0daac65441281f2bb73960e2498 P aba26dcf9c534ea4acf32b81b5b169d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb55fdf9d26e4b58a34f78c5cb9b58e5" member_type: VOTER last_known_addr { host: "127.28.113.2" port: 45811 } } peers { permanent_uuid: "aba26dcf9c534ea4acf32b81b5b169d9" member_type: VOTER last_known_addr { host: "127.28.113.1" port: 34333 } } peers { permanent_uuid: "dd475f46eef34091aa6005dbad4acaea" member_type: VOTER last_known_addr { host: "127.28.113.3" port: 41603 } }
I20251215 21:02:23.536602 1900 catalog_manager.cc:5654] T 74e1c0daac65441281f2bb73960e2498 P aba26dcf9c534ea4acf32b81b5b169d9 reported cstate change: term changed from 0 to 1, leader changed from <none> to aba26dcf9c534ea4acf32b81b5b169d9 (127.28.113.1). New cstate: current_term: 1 leader_uuid: "aba26dcf9c534ea4acf32b81b5b169d9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb55fdf9d26e4b58a34f78c5cb9b58e5" member_type: VOTER last_known_addr { host: "127.28.113.2" port: 45811 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aba26dcf9c534ea4acf32b81b5b169d9" member_type: VOTER last_known_addr { host: "127.28.113.1" port: 34333 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dd475f46eef34091aa6005dbad4acaea" member_type: VOTER last_known_addr { host: "127.28.113.3" port: 41603 } health_report { overall_health: UNKNOWN } } }
I20251215 21:02:23.569211 2363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:02:23.567558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44634 (local address 127.28.113.1:34333)
1215 21:02:23.567671 (+ 113us) server_negotiation.cc:207] Beginning negotiation
1215 21:02:23.567675 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1215 21:02:23.567689 (+ 14us) server_negotiation.cc:408] Connection header received
1215 21:02:23.567860 (+ 171us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1215 21:02:23.567863 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1215 21:02:23.567935 (+ 72us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1215 21:02:23.568034 (+ 99us) server_negotiation.cc:227] Negotiated authn=SASL
1215 21:02:23.568836 (+ 802us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1215 21:02:23.568840 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1215 21:02:23.568843 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1215 21:02:23.568855 (+ 12us) server_negotiation.cc:923] calling sasl_server_start()
1215 21:02:23.568877 (+ 22us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1215 21:02:23.568888 (+ 11us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1215 21:02:23.568972 (+ 84us) server_negotiation.cc:1036] Waiting for connection context
1215 21:02:23.569045 (+ 73us) server_negotiation.cc:300] Negotiation successful
1215 21:02:23.569077 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20251215 21:02:23.574599 2304 raft_consensus.cc:1275] T 74e1c0daac65441281f2bb73960e2498 P dd475f46eef34091aa6005dbad4acaea [term 1 FOLLOWER]: Refusing update from remote peer aba26dcf9c534ea4acf32b81b5b169d9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251215 21:02:23.574612 2173 raft_consensus.cc:1275] T 74e1c0daac65441281f2bb73960e2498 P bb55fdf9d26e4b58a34f78c5cb9b58e5 [term 1 FOLLOWER]: Refusing update from remote peer aba26dcf9c534ea4acf32b81b5b169d9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251215 21:02:23.575197 2372 consensus_queue.cc:1048] T 74e1c0daac65441281f2bb73960e2498 P aba26dcf9c534ea4acf32b81b5b169d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb55fdf9d26e4b58a34f78c5cb9b58e5" member_type: VOTER last_known_addr { host: "127.28.113.2" port: 45811 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251215 21:02:23.575395 2375 consensus_queue.cc:1048] T 74e1c0daac65441281f2bb73960e2498 P aba26dcf9c534ea4acf32b81b5b169d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd475f46eef34091aa6005dbad4acaea" member_type: VOTER last_known_addr { host: "127.28.113.3" port: 41603 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251215 21:02:23.582988 2380 mvcc.cc:204] Tried to move back new op lower bound from 7232850098474725376 to 7232850098316169216. Current Snapshot: MvccSnapshot[applied={T|T < 7232850098474725376}]
I20251215 21:02:23.583791 2379 mvcc.cc:204] Tried to move back new op lower bound from 7232850098474725376 to 7232850098316169216. Current Snapshot: MvccSnapshot[applied={T|T < 7232850098474725376}]
I20251215 21:02:23.592456 1900 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34084:
table { table_name: "test-table" } modify_external_catalogs: true
I20251215 21:02:23.592607 1900 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34084:
table { table_name: "test-table" } modify_external_catalogs: true
I20251215 21:02:23.594758 1900 catalog_manager.cc:5941] T 00000000000000000000000000000000 P 63060321ad3e48e3bdb23dbe3909d483: Sending DeleteTablet for 3 replicas of tablet 74e1c0daac65441281f2bb73960e2498
I20251215 21:02:23.595558 2022 tablet_service.cc:1552] Processing DeleteTablet for tablet 74e1c0daac65441281f2bb73960e2498 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-15 21:02:23 UTC) from {username='slave'} at 127.0.0.1:44620
I20251215 21:02:23.595525 2153 tablet_service.cc:1552] Processing DeleteTablet for tablet 74e1c0daac65441281f2bb73960e2498 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-15 21:02:23 UTC) from {username='slave'} at 127.0.0.1:47558
I20251215 21:02:23.595597 2284 tablet_service.cc:1552] Processing DeleteTablet for tablet 74e1c0daac65441281f2bb73960e2498 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-15 21:02:23 UTC) from {username='slave'} at 127.0.0.1:34766
I20251215 21:02:23.596112 2387 tablet_replica.cc:333] T 74e1c0daac65441281f2bb73960e2498 P bb55fdf9d26e4b58a34f78c5cb9b58e5: stopping tablet replica
I20251215 21:02:23.596357 2389 tablet_replica.cc:333] T 74e1c0daac65441281f2bb73960e2498 P dd475f46eef34091aa6005dbad4acaea: stopping tablet replica
I20251215 21:02:23.596608 2389 raft_consensus.cc:2243] T 74e1c0daac65441281f2bb73960e2498 P dd475f46eef34091aa6005dbad4acaea [term 1 FOLLOWER]: Raft consensus shutting down.
I20251215 21:02:23.596611 2387 raft_consensus.cc:2243] T 74e1c0daac65441281f2bb73960e2498 P bb55fdf9d26e4b58a34f78c5cb9b58e5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251215 21:02:23.596910 29124 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu with pid 1960
I20251215 21:02:23.597615 2389 raft_consensus.cc:2272] T 74e1c0daac65441281f2bb73960e2498 P dd475f46eef34091aa6005dbad4acaea [term 1 FOLLOWER]: Raft consensus is shut down!
I20251215 21:02:23.597805 2387 raft_consensus.cc:2272] T 74e1c0daac65441281f2bb73960e2498 P bb55fdf9d26e4b58a34f78c5cb9b58e5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251215 21:02:23.599254 2387 ts_tablet_manager.cc:1916] T 74e1c0daac65441281f2bb73960e2498 P bb55fdf9d26e4b58a34f78c5cb9b58e5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251215 21:02:23.599709 2389 ts_tablet_manager.cc:1916] T 74e1c0daac65441281f2bb73960e2498 P dd475f46eef34091aa6005dbad4acaea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251215 21:02:23.601269 2387 ts_tablet_manager.cc:1929] T 74e1c0daac65441281f2bb73960e2498 P bb55fdf9d26e4b58a34f78c5cb9b58e5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251215 21:02:23.601348 2387 log.cc:1199] T 74e1c0daac65441281f2bb73960e2498 P bb55fdf9d26e4b58a34f78c5cb9b58e5: Deleting WAL directory at /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1765832506145157-29124-0/minicluster-data/ts-1/wal/wals/74e1c0daac65441281f2bb73960e2498
I20251215 21:02:23.601574 2387 ts_tablet_manager.cc:1950] T 74e1c0daac65441281f2bb73960e2498 P bb55fdf9d26e4b58a34f78c5cb9b58e5: Deleting consensus metadata
I20251215 21:02:23.602031 2389 ts_tablet_manager.cc:1929] T 74e1c0daac65441281f2bb73960e2498 P dd475f46eef34091aa6005dbad4acaea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251215 21:02:23.602111 2389 log.cc:1199] T 74e1c0daac65441281f2bb73960e2498 P dd475f46eef34091aa6005dbad4acaea: Deleting WAL directory at /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1765832506145157-29124-0/minicluster-data/ts-2/wal/wals/74e1c0daac65441281f2bb73960e2498
I20251215 21:02:23.602361 2389 ts_tablet_manager.cc:1950] T 74e1c0daac65441281f2bb73960e2498 P dd475f46eef34091aa6005dbad4acaea: Deleting consensus metadata
I20251215 21:02:23.602684 1886 catalog_manager.cc:4985] TS bb55fdf9d26e4b58a34f78c5cb9b58e5 (127.28.113.2:45811): tablet 74e1c0daac65441281f2bb73960e2498 (table test-table [id=f3e5e7fe8dec4ec8940d1087034d218a]) successfully deleted
I20251215 21:02:23.603392 1887 catalog_manager.cc:4985] TS dd475f46eef34091aa6005dbad4acaea (127.28.113.3:41603): tablet 74e1c0daac65441281f2bb73960e2498 (table test-table [id=f3e5e7fe8dec4ec8940d1087034d218a]) successfully deleted
W20251215 21:02:23.603828 1886 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.28.113.1:34333 (error 108)
I20251215 21:02:23.604000 29124 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu with pid 2091
I20251215 21:02:23.605352 2361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:02:23.605088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.113.1:34333 (local address 127.0.0.1:44636)
1215 21:02:23.605218 (+ 130us) negotiation.cc:107] Waiting for socket to connect
1215 21:02:23.605280 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.113.1:34333: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20251215 21:02:23.605475 1886 catalog_manager.cc:4712] TS aba26dcf9c534ea4acf32b81b5b169d9 (127.28.113.1:34333): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 74e1c0daac65441281f2bb73960e2498: Network error: Client connection negotiation failed: client connection to 127.28.113.1:34333: connect: Connection refused (error 111)
I20251215 21:02:23.609171 29124 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu with pid 2222
I20251215 21:02:23.613963 29124 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu with pid 1869
2025-12-15T21:02:23Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (1134 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20251215 21:02:23.633893 29124 mini_oidc.cc:150] Starting JWKS server
I20251215 21:02:23.635217 29124 webserver.cc:492] Webserver started at https://127.0.0.1:38441/ using document root <none> and password file <none>
I20251215 21:02:23.635342 29124 mini_oidc.cc:186] Starting OIDC Discovery server
I20251215 21:02:23.635548 29124 webserver.cc:492] Webserver started at http://127.0.0.1:38947/ using document root <none> and password file <none>
2025-12-15T21:02:23Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-12-15T21:02:23Z Disabled control of system clock
I20251215 21:02:23.642719 29124 external_mini_cluster.cc:1366] Running /tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu
/tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1765832506145157-29124-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1765832506145157-29124-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1765832506145157-29124-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1765832506145157-29124-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.113.62:36705
--webserver_interface=127.28.113.62
--webserver_port=0
--builtin_ntp_servers=127.28.113.20:43755
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.113.62:36705
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38441/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1765832506145157-29124-0/testchainca.cert with env {}
W20251215 21:02:23.746508 2396 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251215 21:02:23.746750 2396 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251215 21:02:23.746820 2396 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251215 21:02:23.750032 2396 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251215 21:02:23.750095 2396 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251215 21:02:23.750118 2396 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251215 21:02:23.750141 2396 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251215 21:02:23.750169 2396 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20251215 21:02:23.750202 2396 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20251215 21:02:23.750231 2396 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:38441/jwks.json
I20251215 21:02:23.754447 2396 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.113.20:43755
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1765832506145157-29124-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1765832506145157-29124-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.113.62:36705
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.113.62:36705
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38441/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1765832506145157-29124-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.113.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1765832506145157-29124-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2396
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1765832506145157-29124-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision dc741732bb6a40213e11a13869fad06a49a209a3
build type DEBUG
built by None at 15 Dec 2025 20:43:14 UTC on 24a791456cd2
build id 9530
I20251215 21:02:23.755475 2396 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251215 21:02:23.756294 2396 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251215 21:02:23.761492 2401 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251215 21:02:23.761494 2402 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251215 21:02:23.761492 2404 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251215 21:02:23.761657 2396 server_base.cc:1047] running on GCE node
I20251215 21:02:23.762303 2396 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251215 21:02:23.763181 2396 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251215 21:02:23.764318 2396 hybrid_clock.cc:648] HybridClock initialized: now 1765832543764306 us; error 28 us; skew 500 ppm
I20251215 21:02:23.765900 2396 webserver.cc:492] Webserver started at http://127.28.113.62:40869/ using document root <none> and password file <none>
I20251215 21:02:23.766422 2396 fs_manager.cc:362] Metadata directory not provided
I20251215 21:02:23.766479 2396 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251215 21:02:23.766639 2396 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251215 21:02:23.768273 2396 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1765832506145157-29124-0/minicluster-data/master-0/data/instance:
uuid: "ebdba8712d144ed09ea3d97568575db8"
format_stamp: "Formatted at 2025-12-15 21:02:23 on dist-test-slave-bjsf"
server_key: "e73a8037d234579e4d83459c4c0b00bd"
server_key_iv: "1a7408188ebebff3927445c78e9d06bd"
server_key_version: "encryptionkey@0"
I20251215 21:02:23.768731 2396 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1765832506145157-29124-0/minicluster-data/master-0/wal/instance:
uuid: "ebdba8712d144ed09ea3d97568575db8"
format_stamp: "Formatted at 2025-12-15 21:02:23 on dist-test-slave-bjsf"
server_key: "e73a8037d234579e4d83459c4c0b00bd"
server_key_iv: "1a7408188ebebff3927445c78e9d06bd"
server_key_version: "encryptionkey@0"
I20251215 21:02:23.771795 2396 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20251215 21:02:23.773842 2410 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251215 21:02:23.774775 2396 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20251215 21:02:23.774871 2396 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1765832506145157-29124-0/minicluster-data/master-0/data,/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1765832506145157-29124-0/minicluster-data/master-0/wal
uuid: "ebdba8712d144ed09ea3d97568575db8"
format_stamp: "Formatted at 2025-12-15 21:02:23 on dist-test-slave-bjsf"
server_key: "e73a8037d234579e4d83459c4c0b00bd"
server_key_iv: "1a7408188ebebff3927445c78e9d06bd"
server_key_version: "encryptionkey@0"
I20251215 21:02:23.774950 2396 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1765832506145157-29124-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1765832506145157-29124-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1765832506145157-29124-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251215 21:02:23.793035 2396 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251215 21:02:23.793658 2396 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251215 21:02:23.793821 2396 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251215 21:02:23.800554 2396 rpc_server.cc:307] RPC server started. Bound to: 127.28.113.62:36705
I20251215 21:02:23.800552 2462 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.28.113.62:36705 every 8 connection(s)
I20251215 21:02:23.801520 2396 server_base.cc:1179] Dumped server information to /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1765832506145157-29124-0/minicluster-data/master-0/data/info.pb
I20251215 21:02:23.804376 2463 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 21:02:23.807590 29124 external_mini_cluster.cc:1428] Started /tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu as pid 2396
I20251215 21:02:23.807716 29124 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1765832506145157-29124-0/minicluster-data/master-0/wal/instance
I20251215 21:02:23.807996 29124 external_mini_cluster.cc:1468] Setting key cd10aa1df81e7db467a96fb666212a97
I20251215 21:02:23.810421 2463 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ebdba8712d144ed09ea3d97568575db8: Bootstrap starting.
I20251215 21:02:23.812515 2463 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ebdba8712d144ed09ea3d97568575db8: Neither blocks nor log segments found. Creating new log.
I20251215 21:02:23.813205 2463 log.cc:826] T 00000000000000000000000000000000 P ebdba8712d144ed09ea3d97568575db8: Log is configured to *not* fsync() on all Append() calls
I20251215 21:02:23.815044 2463 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ebdba8712d144ed09ea3d97568575db8: No bootstrap required, opened a new log
I20251215 21:02:23.817870 2463 raft_consensus.cc:359] T 00000000000000000000000000000000 P ebdba8712d144ed09ea3d97568575db8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebdba8712d144ed09ea3d97568575db8" member_type: VOTER last_known_addr { host: "127.28.113.62" port: 36705 } }
I20251215 21:02:23.818056 2463 raft_consensus.cc:385] T 00000000000000000000000000000000 P ebdba8712d144ed09ea3d97568575db8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251215 21:02:23.818112 2463 raft_consensus.cc:740] T 00000000000000000000000000000000 P ebdba8712d144ed09ea3d97568575db8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ebdba8712d144ed09ea3d97568575db8, State: Initialized, Role: FOLLOWER
I20251215 21:02:23.818125 2466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:02:23.809271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38208 (local address 127.28.113.62:36705)
1215 21:02:23.809907 (+ 636us) server_negotiation.cc:207] Beginning negotiation
1215 21:02:23.809916 (+ 9us) server_negotiation.cc:400] Waiting for connection header
1215 21:02:23.809950 (+ 34us) server_negotiation.cc:408] Connection header received
1215 21:02:23.810459 (+ 509us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1215 21:02:23.810474 (+ 15us) server_negotiation.cc:462] Received NEGOTIATE request from client
1215 21:02:23.810842 (+ 368us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1215 21:02:23.811135 (+ 293us) server_negotiation.cc:227] Negotiated authn=SASL
1215 21:02:23.811589 (+ 454us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:02:23.812387 (+ 798us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1215 21:02:23.813082 (+ 695us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:02:23.813365 (+ 283us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:02:23.816622 (+ 3257us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1215 21:02:23.816642 (+ 20us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1215 21:02:23.816652 (+ 10us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1215 21:02:23.817037 (+ 385us) server_negotiation.cc:923] calling sasl_server_start()
1215 21:02:23.817081 (+ 44us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1215 21:02:23.817096 (+ 15us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1215 21:02:23.817186 (+ 90us) server_negotiation.cc:1036] Waiting for connection context
1215 21:02:23.817334 (+ 148us) server_negotiation.cc:300] Negotiation successful
1215 21:02:23.817549 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":283,"thread_start_us":124,"threads_started":1}
I20251215 21:02:23.818554 2463 consensus_queue.cc:260] T 00000000000000000000000000000000 P ebdba8712d144ed09ea3d97568575db8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebdba8712d144ed09ea3d97568575db8" member_type: VOTER last_known_addr { host: "127.28.113.62" port: 36705 } }
I20251215 21:02:23.818711 2463 raft_consensus.cc:399] T 00000000000000000000000000000000 P ebdba8712d144ed09ea3d97568575db8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251215 21:02:23.818771 2463 raft_consensus.cc:493] T 00000000000000000000000000000000 P ebdba8712d144ed09ea3d97568575db8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251215 21:02:23.818853 2463 raft_consensus.cc:3060] T 00000000000000000000000000000000 P ebdba8712d144ed09ea3d97568575db8 [term 0 FOLLOWER]: Advancing to term 1
I20251215 21:02:23.819690 2463 raft_consensus.cc:515] T 00000000000000000000000000000000 P ebdba8712d144ed09ea3d97568575db8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebdba8712d144ed09ea3d97568575db8" member_type: VOTER last_known_addr { host: "127.28.113.62" port: 36705 } }
I20251215 21:02:23.820057 2463 leader_election.cc:304] T 00000000000000000000000000000000 P ebdba8712d144ed09ea3d97568575db8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ebdba8712d144ed09ea3d97568575db8; no voters:
I20251215 21:02:23.820341 2463 leader_election.cc:290] T 00000000000000000000000000000000 P ebdba8712d144ed09ea3d97568575db8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20251215 21:02:23.820405 2468 raft_consensus.cc:2804] T 00000000000000000000000000000000 P ebdba8712d144ed09ea3d97568575db8 [term 1 FOLLOWER]: Leader election won for term 1
I20251215 21:02:23.820576 2468 raft_consensus.cc:697] T 00000000000000000000000000000000 P ebdba8712d144ed09ea3d97568575db8 [term 1 LEADER]: Becoming Leader. State: Replica: ebdba8712d144ed09ea3d97568575db8, State: Running, Role: LEADER
I20251215 21:02:23.820740 29124 external_mini_cluster.cc:949] 0 TS(s) registered with all masters
I20251215 21:02:23.820849 2468 consensus_queue.cc:237] T 00000000000000000000000000000000 P ebdba8712d144ed09ea3d97568575db8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebdba8712d144ed09ea3d97568575db8" member_type: VOTER last_known_addr { host: "127.28.113.62" port: 36705 } }
I20251215 21:02:23.821493 2463 sys_catalog.cc:565] T 00000000000000000000000000000000 P ebdba8712d144ed09ea3d97568575db8 [sys.catalog]: configured and running, proceeding with master startup.
I20251215 21:02:23.822476 2470 sys_catalog.cc:455] T 00000000000000000000000000000000 P ebdba8712d144ed09ea3d97568575db8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ebdba8712d144ed09ea3d97568575db8. Latest consensus state: current_term: 1 leader_uuid: "ebdba8712d144ed09ea3d97568575db8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebdba8712d144ed09ea3d97568575db8" member_type: VOTER last_known_addr { host: "127.28.113.62" port: 36705 } } }
I20251215 21:02:23.822615 2470 sys_catalog.cc:458] T 00000000000000000000000000000000 P ebdba8712d144ed09ea3d97568575db8 [sys.catalog]: This master's current role is: LEADER
I20251215 21:02:23.822845 2469 sys_catalog.cc:455] T 00000000000000000000000000000000 P ebdba8712d144ed09ea3d97568575db8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ebdba8712d144ed09ea3d97568575db8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebdba8712d144ed09ea3d97568575db8" member_type: VOTER last_known_addr { host: "127.28.113.62" port: 36705 } } }
I20251215 21:02:23.822930 2469 sys_catalog.cc:458] T 00000000000000000000000000000000 P ebdba8712d144ed09ea3d97568575db8 [sys.catalog]: This master's current role is: LEADER
I20251215 21:02:23.823057 2477 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251215 21:02:23.825896 2477 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251215 21:02:23.831036 2477 catalog_manager.cc:1357] Generated new cluster ID: 9aa146fb9fdf4830afb090d19a00af76
I20251215 21:02:23.831103 2477 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251215 21:02:23.838343 2477 catalog_manager.cc:1380] Generated new certificate authority record
I20251215 21:02:23.839196 2477 catalog_manager.cc:1514] Loading token signing keys...
I20251215 21:02:23.847605 2477 catalog_manager.cc:6027] T 00000000000000000000000000000000 P ebdba8712d144ed09ea3d97568575db8: Generated new TSK 0
I20251215 21:02:23.848313 2477 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251215 21:02:23.881300 2466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:02:23.858072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38220 (local address 127.28.113.62:36705)
1215 21:02:23.858201 (+ 129us) server_negotiation.cc:207] Beginning negotiation
1215 21:02:23.858205 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1215 21:02:23.858304 (+ 99us) server_negotiation.cc:408] Connection header received
1215 21:02:23.858482 (+ 178us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1215 21:02:23.858486 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1215 21:02:23.858545 (+ 59us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1215 21:02:23.858646 (+ 101us) server_negotiation.cc:227] Negotiated authn=JWT
1215 21:02:23.858977 (+ 331us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:02:23.859772 (+ 795us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1215 21:02:23.860403 (+ 631us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:02:23.860601 (+ 198us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:02:23.860699 (+ 98us) server_negotiation.cc:366] Received JWT_EXCHANGE NegotiatePB request
1215 21:02:23.880835 (+ 20136us) server_negotiation.cc:378] Sending JWT_EXCHANGE NegotiatePB response
1215 21:02:23.880956 (+ 121us) server_negotiation.cc:1036] Waiting for connection context
1215 21:02:23.881080 (+ 124us) server_negotiation.cc:300] Negotiation successful
1215 21:02:23.881142 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53,"thread_start_us":134,"threads_started":1}
I20251215 21:02:23.892421 2466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:02:23.888643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38236 (local address 127.28.113.62:36705)
1215 21:02:23.888767 (+ 124us) server_negotiation.cc:207] Beginning negotiation
1215 21:02:23.888771 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1215 21:02:23.888874 (+ 103us) server_negotiation.cc:408] Connection header received
1215 21:02:23.888982 (+ 108us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1215 21:02:23.888985 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1215 21:02:23.889032 (+ 47us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1215 21:02:23.889111 (+ 79us) server_negotiation.cc:227] Negotiated authn=JWT
1215 21:02:23.889396 (+ 285us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:02:23.890171 (+ 775us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1215 21:02:23.890754 (+ 583us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:02:23.890952 (+ 198us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:02:23.891000 (+ 48us) server_negotiation.cc:366] Received JWT_EXCHANGE NegotiatePB request
1215 21:02:23.892102 (+ 1102us) server_negotiation.cc:395] Sending RPC error: FATAL_INVALID_JWT: Not authorized: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:../crypto/rsa/rsa_pk1.c:67 error:04067072:rsa routines:rsa_ossl_public_decrypt:padding check failed:../crypto/rsa/rsa_ossl.c:581)
1215 21:02:23.892303 (+ 201us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:38236: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:../crypto/rsa/rsa_pk1.c:67 error:04067072:rsa routines:rsa_ossl_public_decrypt:padding check failed:../crypto/rsa/rsa_ossl.c:581)
Metrics: {"server-negotiator.queue_time_us":51}
W20251215 21:02:23.892491 2466 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:38236: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:../crypto/rsa/rsa_pk1.c:67 error:04067072:rsa routines:rsa_ossl_public_decrypt:padding check failed:../crypto/rsa/rsa_ossl.c:581)
W20251215 21:02:23.893149 2499 negotiation.cc:337] Failed RPC negotiation. Trace:
1215 21:02:23.888544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.113.62:36705 (local address 127.0.0.1:38236)
1215 21:02:23.888788 (+ 244us) negotiation.cc:107] Waiting for socket to connect
1215 21:02:23.888802 (+ 14us) client_negotiation.cc:174] Beginning negotiation
1215 21:02:23.888898 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1215 21:02:23.889119 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1215 21:02:23.889121 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1215 21:02:23.889127 (+ 6us) client_negotiation.cc:189] Negotiated authn=JWT
1215 21:02:23.889310 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:02:23.889316 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:02:23.890261 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1215 21:02:23.890264 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1215 21:02:23.890665 (+ 401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:02:23.890672 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:02:23.890760 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:02:23.890779 (+ 19us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
1215 21:02:23.892275 (+ 1496us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:../crypto/rsa/rsa_pk1.c:67 error:04067072:rsa routines:rsa_ossl_public_decrypt:padding check failed:../crypto/rsa/rsa_ossl.c:581)
1215 21:02:23.892409 (+ 134us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.28.113.62:36705: FATAL_INVALID_JWT: Not authorized: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:../crypto/rsa/rsa_pk1.c:67 error:04067072:rsa routines:rsa_ossl_public_decrypt:padding check failed:../crypto/rsa/rsa_ossl.c:581)
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":92,"threads_started":1}
I20251215 21:02:38.900396 2505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:02:38.897188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35932 (local address 127.28.113.62:36705)
1215 21:02:38.897464 (+ 276us) server_negotiation.cc:207] Beginning negotiation
1215 21:02:38.897468 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1215 21:02:38.897485 (+ 17us) server_negotiation.cc:408] Connection header received
1215 21:02:38.897588 (+ 103us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1215 21:02:38.897591 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1215 21:02:38.897639 (+ 48us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1215 21:02:38.897712 (+ 73us) server_negotiation.cc:227] Negotiated authn=JWT
1215 21:02:38.898091 (+ 379us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:02:38.898611 (+ 520us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1215 21:02:38.899237 (+ 626us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:02:38.899402 (+ 165us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:02:38.899447 (+ 45us) server_negotiation.cc:366] Received JWT_EXCHANGE NegotiatePB request
1215 21:02:38.900088 (+ 641us) server_negotiation.cc:395] Sending RPC error: FATAL_INVALID_JWT: Not authorized: JWT verification failed: token expired
1215 21:02:38.900243 (+ 155us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:35932: JWT verification failed: token expired
Metrics: {"server-negotiator.queue_time_us":192,"thread_start_us":95,"threads_started":1}
W20251215 21:02:38.900449 2504 negotiation.cc:337] Failed RPC negotiation. Trace:
1215 21:02:38.897034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.113.62:36705 (local address 127.0.0.1:35932)
1215 21:02:38.897349 (+ 315us) negotiation.cc:107] Waiting for socket to connect
1215 21:02:38.897365 (+ 16us) client_negotiation.cc:174] Beginning negotiation
1215 21:02:38.897478 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1215 21:02:38.897764 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1215 21:02:38.897768 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
1215 21:02:38.897776 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
1215 21:02:38.897975 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:02:38.897981 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:02:38.898729 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1215 21:02:38.898733 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1215 21:02:38.899129 (+ 396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:02:38.899137 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:02:38.899235 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:02:38.899259 (+ 24us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
1215 21:02:38.900204 (+ 945us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: JWT verification failed: token expired
1215 21:02:38.900321 (+ 117us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.28.113.62:36705: FATAL_INVALID_JWT: Not authorized: JWT verification failed: token expired
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":138,"threads_started":1}
W20251215 21:02:38.900475 2505 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:35932: JWT verification failed: token expired
W20251215 21:02:38.902632 2505 negotiation.cc:337] Failed RPC negotiation. Trace:
1215 21:02:38.901993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35942 (local address 127.28.113.62:36705)
1215 21:02:38.902106 (+ 113us) server_negotiation.cc:207] Beginning negotiation
1215 21:02:38.902109 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1215 21:02:38.902193 (+ 84us) server_negotiation.cc:408] Connection header received
1215 21:02:38.902283 (+ 90us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1215 21:02:38.902286 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1215 21:02:38.902324 (+ 38us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1215 21:02:38.902385 (+ 61us) server_negotiation.cc:227] Negotiated authn=SASL
1215 21:02:38.902531 (+ 146us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35942: BlockingRecv error: recv got EOF from 127.0.0.1:35942 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
W20251215 21:02:38.905769 2515 client_negotiation.cc:375] the client has a JWT but it isn't advertising its JWT-based authn capability since it doesn't trust the certificate of the server at 127.28.113.62:36705
W20251215 21:02:38.906279 2505 negotiation.cc:337] Failed RPC negotiation. Trace:
1215 21:02:38.905494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35956 (local address 127.28.113.62:36705)
1215 21:02:38.905620 (+ 126us) server_negotiation.cc:207] Beginning negotiation
1215 21:02:38.905624 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1215 21:02:38.905695 (+ 71us) server_negotiation.cc:408] Connection header received
1215 21:02:38.905933 (+ 238us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1215 21:02:38.905937 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1215 21:02:38.905981 (+ 44us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1215 21:02:38.906043 (+ 62us) server_negotiation.cc:227] Negotiated authn=SASL
1215 21:02:38.906194 (+ 151us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35956: BlockingRecv error: recv got EOF from 127.0.0.1:35956 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
I20251215 21:02:38.906452 29124 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu with pid 2396
2025-12-15T21:02:38Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniCluster (15311 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20251215 21:02:38.945585 29124 mini_oidc.cc:150] Starting JWKS server
I20251215 21:02:38.946455 29124 webserver.cc:492] Webserver started at https://127.0.0.1:38071/ using document root <none> and password file <none>
I20251215 21:02:38.946540 29124 mini_oidc.cc:186] Starting OIDC Discovery server
I20251215 21:02:38.946748 29124 webserver.cc:492] Webserver started at http://127.0.0.1:37223/ using document root <none> and password file <none>
2025-12-15T21:02:38Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-12-15T21:02:38Z Disabled control of system clock
I20251215 21:02:38.953672 29124 external_mini_cluster.cc:1366] Running /tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu
/tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1765832506145157-29124-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1765832506145157-29124-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1765832506145157-29124-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1765832506145157-29124-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.113.62:40769
--webserver_interface=127.28.113.62
--webserver_port=0
--builtin_ntp_servers=127.28.113.20:42041
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.113.62:40769
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38071/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1765832506145157-29124-0/testchainca.cert with env {}
W20251215 21:02:39.054601 2521 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251215 21:02:39.054850 2521 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251215 21:02:39.054919 2521 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251215 21:02:39.058296 2521 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251215 21:02:39.058357 2521 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251215 21:02:39.058378 2521 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251215 21:02:39.058403 2521 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251215 21:02:39.058434 2521 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20251215 21:02:39.058466 2521 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20251215 21:02:39.058493 2521 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:38071/jwks.json
I20251215 21:02:39.062706 2521 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.113.20:42041
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1765832506145157-29124-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1765832506145157-29124-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.113.62:40769
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.113.62:40769
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38071/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1765832506145157-29124-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.113.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1765832506145157-29124-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2521
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1765832506145157-29124-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision dc741732bb6a40213e11a13869fad06a49a209a3
build type DEBUG
built by None at 15 Dec 2025 20:43:14 UTC on 24a791456cd2
build id 9530
I20251215 21:02:39.063679 2521 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251215 21:02:39.064489 2521 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251215 21:02:39.069851 2527 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251215 21:02:39.069878 2526 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251215 21:02:39.069988 2521 server_base.cc:1047] running on GCE node
W20251215 21:02:39.069883 2529 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251215 21:02:39.070540 2521 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251215 21:02:39.071419 2521 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251215 21:02:39.072579 2521 hybrid_clock.cc:648] HybridClock initialized: now 1765832559072570 us; error 27 us; skew 500 ppm
I20251215 21:02:39.074304 2521 webserver.cc:492] Webserver started at http://127.28.113.62:38855/ using document root <none> and password file <none>
I20251215 21:02:39.074838 2521 fs_manager.cc:362] Metadata directory not provided
I20251215 21:02:39.074898 2521 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251215 21:02:39.075085 2521 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251215 21:02:39.076756 2521 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1765832506145157-29124-0/minicluster-data/master-0/data/instance:
uuid: "1b6f609b870547b9a6e367b99daf0eac"
format_stamp: "Formatted at 2025-12-15 21:02:39 on dist-test-slave-bjsf"
server_key: "27b3a7e8ba746761e13e43783f4705ae"
server_key_iv: "13f8887801f2997d947d0c4c3d62aab5"
server_key_version: "encryptionkey@0"
I20251215 21:02:39.077225 2521 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1765832506145157-29124-0/minicluster-data/master-0/wal/instance:
uuid: "1b6f609b870547b9a6e367b99daf0eac"
format_stamp: "Formatted at 2025-12-15 21:02:39 on dist-test-slave-bjsf"
server_key: "27b3a7e8ba746761e13e43783f4705ae"
server_key_iv: "13f8887801f2997d947d0c4c3d62aab5"
server_key_version: "encryptionkey@0"
I20251215 21:02:39.080453 2521 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20251215 21:02:39.082490 2535 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251215 21:02:39.083405 2521 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20251215 21:02:39.083510 2521 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1765832506145157-29124-0/minicluster-data/master-0/data,/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1765832506145157-29124-0/minicluster-data/master-0/wal
uuid: "1b6f609b870547b9a6e367b99daf0eac"
format_stamp: "Formatted at 2025-12-15 21:02:39 on dist-test-slave-bjsf"
server_key: "27b3a7e8ba746761e13e43783f4705ae"
server_key_iv: "13f8887801f2997d947d0c4c3d62aab5"
server_key_version: "encryptionkey@0"
I20251215 21:02:39.083595 2521 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1765832506145157-29124-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1765832506145157-29124-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1765832506145157-29124-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251215 21:02:39.100338 2521 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251215 21:02:39.101029 2521 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251215 21:02:39.101207 2521 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251215 21:02:39.108562 2587 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.28.113.62:40769 every 8 connection(s)
I20251215 21:02:39.108568 2521 rpc_server.cc:307] RPC server started. Bound to: 127.28.113.62:40769
I20251215 21:02:39.109606 2521 server_base.cc:1179] Dumped server information to /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1765832506145157-29124-0/minicluster-data/master-0/data/info.pb
I20251215 21:02:39.112228 2588 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 21:02:39.117266 2588 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1b6f609b870547b9a6e367b99daf0eac: Bootstrap starting.
I20251215 21:02:39.118378 29124 external_mini_cluster.cc:1428] Started /tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu as pid 2521
I20251215 21:02:39.118469 29124 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1765832506145157-29124-0/minicluster-data/master-0/wal/instance
I20251215 21:02:39.118708 29124 external_mini_cluster.cc:1468] Setting key 0d998dc2905e4d4bcb146952156d2f84
I20251215 21:02:39.119525 2588 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1b6f609b870547b9a6e367b99daf0eac: Neither blocks nor log segments found. Creating new log.
I20251215 21:02:39.120316 2588 log.cc:826] T 00000000000000000000000000000000 P 1b6f609b870547b9a6e367b99daf0eac: Log is configured to *not* fsync() on all Append() calls
I20251215 21:02:39.122267 2588 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1b6f609b870547b9a6e367b99daf0eac: No bootstrap required, opened a new log
I20251215 21:02:39.124924 2588 raft_consensus.cc:359] T 00000000000000000000000000000000 P 1b6f609b870547b9a6e367b99daf0eac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b6f609b870547b9a6e367b99daf0eac" member_type: VOTER last_known_addr { host: "127.28.113.62" port: 40769 } }
I20251215 21:02:39.125089 2588 raft_consensus.cc:385] T 00000000000000000000000000000000 P 1b6f609b870547b9a6e367b99daf0eac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251215 21:02:39.125128 2588 raft_consensus.cc:740] T 00000000000000000000000000000000 P 1b6f609b870547b9a6e367b99daf0eac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b6f609b870547b9a6e367b99daf0eac, State: Initialized, Role: FOLLOWER
I20251215 21:02:39.125463 2588 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1b6f609b870547b9a6e367b99daf0eac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b6f609b870547b9a6e367b99daf0eac" member_type: VOTER last_known_addr { host: "127.28.113.62" port: 40769 } }
I20251215 21:02:39.125560 2588 raft_consensus.cc:399] T 00000000000000000000000000000000 P 1b6f609b870547b9a6e367b99daf0eac [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251215 21:02:39.125600 2588 raft_consensus.cc:493] T 00000000000000000000000000000000 P 1b6f609b870547b9a6e367b99daf0eac [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251215 21:02:39.125661 2588 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 1b6f609b870547b9a6e367b99daf0eac [term 0 FOLLOWER]: Advancing to term 1
I20251215 21:02:39.126361 2588 raft_consensus.cc:515] T 00000000000000000000000000000000 P 1b6f609b870547b9a6e367b99daf0eac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b6f609b870547b9a6e367b99daf0eac" member_type: VOTER last_known_addr { host: "127.28.113.62" port: 40769 } }
I20251215 21:02:39.126645 2588 leader_election.cc:304] T 00000000000000000000000000000000 P 1b6f609b870547b9a6e367b99daf0eac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1b6f609b870547b9a6e367b99daf0eac; no voters:
I20251215 21:02:39.126871 2588 leader_election.cc:290] T 00000000000000000000000000000000 P 1b6f609b870547b9a6e367b99daf0eac [CANDIDATE]: Term 1 election: Requested vote from peers
I20251215 21:02:39.126966 2593 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 1b6f609b870547b9a6e367b99daf0eac [term 1 FOLLOWER]: Leader election won for term 1
I20251215 21:02:39.127153 2593 raft_consensus.cc:697] T 00000000000000000000000000000000 P 1b6f609b870547b9a6e367b99daf0eac [term 1 LEADER]: Becoming Leader. State: Replica: 1b6f609b870547b9a6e367b99daf0eac, State: Running, Role: LEADER
I20251215 21:02:39.127470 2593 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1b6f609b870547b9a6e367b99daf0eac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b6f609b870547b9a6e367b99daf0eac" member_type: VOTER last_known_addr { host: "127.28.113.62" port: 40769 } }
I20251215 21:02:39.127825 2588 sys_catalog.cc:565] T 00000000000000000000000000000000 P 1b6f609b870547b9a6e367b99daf0eac [sys.catalog]: configured and running, proceeding with master startup.
I20251215 21:02:39.128899 2595 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1b6f609b870547b9a6e367b99daf0eac [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1b6f609b870547b9a6e367b99daf0eac. Latest consensus state: current_term: 1 leader_uuid: "1b6f609b870547b9a6e367b99daf0eac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b6f609b870547b9a6e367b99daf0eac" member_type: VOTER last_known_addr { host: "127.28.113.62" port: 40769 } } }
I20251215 21:02:39.129007 2595 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1b6f609b870547b9a6e367b99daf0eac [sys.catalog]: This master's current role is: LEADER
I20251215 21:02:39.129356 2602 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251215 21:02:39.130079 2594 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1b6f609b870547b9a6e367b99daf0eac [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1b6f609b870547b9a6e367b99daf0eac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b6f609b870547b9a6e367b99daf0eac" member_type: VOTER last_known_addr { host: "127.28.113.62" port: 40769 } } }
I20251215 21:02:39.130285 2594 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1b6f609b870547b9a6e367b99daf0eac [sys.catalog]: This master's current role is: LEADER
I20251215 21:02:39.131299 2591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:02:39.119676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45146 (local address 127.28.113.62:40769)
1215 21:02:39.120361 (+ 685us) server_negotiation.cc:207] Beginning negotiation
1215 21:02:39.120372 (+ 11us) server_negotiation.cc:400] Waiting for connection header
1215 21:02:39.120411 (+ 39us) server_negotiation.cc:408] Connection header received
1215 21:02:39.120981 (+ 570us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1215 21:02:39.121000 (+ 19us) server_negotiation.cc:462] Received NEGOTIATE request from client
1215 21:02:39.121330 (+ 330us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1215 21:02:39.121648 (+ 318us) server_negotiation.cc:227] Negotiated authn=SASL
1215 21:02:39.122095 (+ 447us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:02:39.123177 (+ 1082us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1215 21:02:39.123783 (+ 606us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:02:39.124084 (+ 301us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:02:39.128858 (+ 4774us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1215 21:02:39.128884 (+ 26us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1215 21:02:39.128898 (+ 14us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1215 21:02:39.129809 (+ 911us) server_negotiation.cc:923] calling sasl_server_start()
1215 21:02:39.129861 (+ 52us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1215 21:02:39.129879 (+ 18us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1215 21:02:39.129975 (+ 96us) server_negotiation.cc:1036] Waiting for connection context
1215 21:02:39.130379 (+ 404us) server_negotiation.cc:300] Negotiation successful
1215 21:02:39.130615 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":296,"thread_start_us":130,"threads_started":1}
I20251215 21:02:39.133095 2602 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251215 21:02:39.134055 29124 external_mini_cluster.cc:949] 0 TS(s) registered with all masters
I20251215 21:02:39.138242 2602 catalog_manager.cc:1357] Generated new cluster ID: 23924721c70a46b48360a6e0ea10e3ea
I20251215 21:02:39.138298 2602 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251215 21:02:39.183313 2602 catalog_manager.cc:1380] Generated new certificate authority record
I20251215 21:02:39.184262 2602 catalog_manager.cc:1514] Loading token signing keys...
I20251215 21:02:39.195248 2602 catalog_manager.cc:6027] T 00000000000000000000000000000000 P 1b6f609b870547b9a6e367b99daf0eac: Generated new TSK 0
I20251215 21:02:39.195924 2602 catalog_manager.cc:1524] Initializing in-progress tserver states...
E20251215 21:02:39.232160 2618 webserver.cc:578] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20251215 21:02:39.232246 2618 webserver.cc:578] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20251215 21:02:39.232899 2591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:02:39.209070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45158 (local address 127.28.113.62:40769)
1215 21:02:39.209182 (+ 112us) server_negotiation.cc:207] Beginning negotiation
1215 21:02:39.209187 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1215 21:02:39.209329 (+ 142us) server_negotiation.cc:408] Connection header received
1215 21:02:39.209491 (+ 162us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1215 21:02:39.209494 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1215 21:02:39.209542 (+ 48us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1215 21:02:39.209620 (+ 78us) server_negotiation.cc:227] Negotiated authn=JWT
1215 21:02:39.209965 (+ 345us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:02:39.210465 (+ 500us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1215 21:02:39.211077 (+ 612us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:02:39.211247 (+ 170us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:02:39.211330 (+ 83us) server_negotiation.cc:366] Received JWT_EXCHANGE NegotiatePB request
1215 21:02:39.232470 (+ 21140us) server_negotiation.cc:395] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38071/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
1215 21:02:39.232746 (+ 276us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:45158: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38071/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":35}
W20251215 21:02:39.232980 2591 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:45158: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38071/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20251215 21:02:39.233011 2617 negotiation.cc:337] Failed RPC negotiation. Trace:
1215 21:02:39.209010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.113.62:40769 (local address 127.0.0.1:45158)
1215 21:02:39.209259 (+ 249us) negotiation.cc:107] Waiting for socket to connect
1215 21:02:39.209271 (+ 12us) client_negotiation.cc:174] Beginning negotiation
1215 21:02:39.209368 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1215 21:02:39.209630 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1215 21:02:39.209633 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1215 21:02:39.209640 (+ 7us) client_negotiation.cc:189] Negotiated authn=JWT
1215 21:02:39.209867 (+ 227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:02:39.209874 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:02:39.210565 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1215 21:02:39.210568 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1215 21:02:39.210974 (+ 406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:02:39.210981 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:02:39.211084 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:02:39.211107 (+ 23us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
1215 21:02:39.232686 (+ 21579us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38071/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
1215 21:02:39.232845 (+ 159us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.28.113.62:40769: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38071/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":101,"threads_started":1}
I20251215 21:02:39.233639 29124 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu with pid 2521
2025-12-15T21:02:39Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (326 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20251215 21:02:39.271793 29124 mini_oidc.cc:150] Starting JWKS server
I20251215 21:02:39.272713 29124 webserver.cc:492] Webserver started at https://127.0.0.1:35055/ using document root <none> and password file <none>
I20251215 21:02:39.272804 29124 mini_oidc.cc:186] Starting OIDC Discovery server
I20251215 21:02:39.273003 29124 webserver.cc:492] Webserver started at http://127.0.0.1:44613/ using document root <none> and password file <none>
2025-12-15T21:02:39Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-12-15T21:02:39Z Disabled control of system clock
I20251215 21:02:39.280174 29124 external_mini_cluster.cc:1366] Running /tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu
/tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1765832506145157-29124-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1765832506145157-29124-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1765832506145157-29124-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1765832506145157-29124-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.113.62:41857
--webserver_interface=127.28.113.62
--webserver_port=0
--builtin_ntp_servers=127.28.113.20:39787
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.113.62:41857
--enable_jwt_token_auth=true
--jwks_url=https://localhost:35055/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20251215 21:02:39.381261 2624 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251215 21:02:39.381492 2624 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251215 21:02:39.381538 2624 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251215 21:02:39.384753 2624 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251215 21:02:39.384814 2624 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251215 21:02:39.384829 2624 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251215 21:02:39.384846 2624 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251215 21:02:39.384869 2624 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20251215 21:02:39.384886 2624 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20251215 21:02:39.384904 2624 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:35055/jwks.json
I20251215 21:02:39.388736 2624 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.113.20:39787
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1765832506145157-29124-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1765832506145157-29124-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.113.62:41857
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.113.62:41857
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:35055/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1765832506145157-29124-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.113.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2624
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1765832506145157-29124-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision dc741732bb6a40213e11a13869fad06a49a209a3
build type DEBUG
built by None at 15 Dec 2025 20:43:14 UTC on 24a791456cd2
build id 9530
I20251215 21:02:39.389730 2624 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251215 21:02:39.390755 2624 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251215 21:02:39.395922 2629 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251215 21:02:39.396016 2624 server_base.cc:1047] running on GCE node
W20251215 21:02:39.395939 2630 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251215 21:02:39.395920 2632 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251215 21:02:39.396621 2624 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251215 21:02:39.397475 2624 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251215 21:02:39.398612 2624 hybrid_clock.cc:648] HybridClock initialized: now 1765832559398594 us; error 38 us; skew 500 ppm
I20251215 21:02:39.400300 2624 webserver.cc:492] Webserver started at http://127.28.113.62:39895/ using document root <none> and password file <none>
I20251215 21:02:39.400815 2624 fs_manager.cc:362] Metadata directory not provided
I20251215 21:02:39.400871 2624 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251215 21:02:39.401043 2624 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251215 21:02:39.402666 2624 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1765832506145157-29124-0/minicluster-data/master-0/data/instance:
uuid: "1743156790e4484991de906c6bfccc6c"
format_stamp: "Formatted at 2025-12-15 21:02:39 on dist-test-slave-bjsf"
server_key: "b2b42f43f8e4da770875c5e6c06b784b"
server_key_iv: "9954a65a1d4ba84043c4d84e8bff2a00"
server_key_version: "encryptionkey@0"
I20251215 21:02:39.403111 2624 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1765832506145157-29124-0/minicluster-data/master-0/wal/instance:
uuid: "1743156790e4484991de906c6bfccc6c"
format_stamp: "Formatted at 2025-12-15 21:02:39 on dist-test-slave-bjsf"
server_key: "b2b42f43f8e4da770875c5e6c06b784b"
server_key_iv: "9954a65a1d4ba84043c4d84e8bff2a00"
server_key_version: "encryptionkey@0"
I20251215 21:02:39.406201 2624 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20251215 21:02:39.408226 2638 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251215 21:02:39.409133 2624 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20251215 21:02:39.409242 2624 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1765832506145157-29124-0/minicluster-data/master-0/data,/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1765832506145157-29124-0/minicluster-data/master-0/wal
uuid: "1743156790e4484991de906c6bfccc6c"
format_stamp: "Formatted at 2025-12-15 21:02:39 on dist-test-slave-bjsf"
server_key: "b2b42f43f8e4da770875c5e6c06b784b"
server_key_iv: "9954a65a1d4ba84043c4d84e8bff2a00"
server_key_version: "encryptionkey@0"
I20251215 21:02:39.409320 2624 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1765832506145157-29124-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1765832506145157-29124-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1765832506145157-29124-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251215 21:02:39.432869 2624 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251215 21:02:39.433506 2624 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251215 21:02:39.433660 2624 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251215 21:02:39.440414 2690 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.28.113.62:41857 every 8 connection(s)
I20251215 21:02:39.440426 2624 rpc_server.cc:307] RPC server started. Bound to: 127.28.113.62:41857
I20251215 21:02:39.441458 2624 server_base.cc:1179] Dumped server information to /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1765832506145157-29124-0/minicluster-data/master-0/data/info.pb
I20251215 21:02:39.444168 29124 external_mini_cluster.cc:1428] Started /tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu as pid 2624
I20251215 21:02:39.444293 29124 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1765832506145157-29124-0/minicluster-data/master-0/wal/instance
I20251215 21:02:39.444288 2691 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 21:02:39.444535 29124 external_mini_cluster.cc:1468] Setting key 989e0569d2cef05d225fefccea415261
I20251215 21:02:39.450632 2691 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1743156790e4484991de906c6bfccc6c: Bootstrap starting.
I20251215 21:02:39.452430 2694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:02:39.445671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35532 (local address 127.28.113.62:41857)
1215 21:02:39.446282 (+ 611us) server_negotiation.cc:207] Beginning negotiation
1215 21:02:39.446290 (+ 8us) server_negotiation.cc:400] Waiting for connection header
1215 21:02:39.446323 (+ 33us) server_negotiation.cc:408] Connection header received
1215 21:02:39.447415 (+ 1092us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1215 21:02:39.447436 (+ 21us) server_negotiation.cc:462] Received NEGOTIATE request from client
1215 21:02:39.447778 (+ 342us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1215 21:02:39.448136 (+ 358us) server_negotiation.cc:227] Negotiated authn=SASL
1215 21:02:39.448574 (+ 438us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:02:39.449304 (+ 730us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1215 21:02:39.450133 (+ 829us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:02:39.450330 (+ 197us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:02:39.450891 (+ 561us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1215 21:02:39.450935 (+ 44us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1215 21:02:39.450945 (+ 10us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1215 21:02:39.451302 (+ 357us) server_negotiation.cc:923] calling sasl_server_start()
1215 21:02:39.451342 (+ 40us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1215 21:02:39.451356 (+ 14us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1215 21:02:39.451431 (+ 75us) server_negotiation.cc:1036] Waiting for connection context
1215 21:02:39.451582 (+ 151us) server_negotiation.cc:300] Negotiation successful
1215 21:02:39.451816 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":266,"thread_start_us":125,"threads_started":1}
I20251215 21:02:39.452881 2691 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1743156790e4484991de906c6bfccc6c: Neither blocks nor log segments found. Creating new log.
I20251215 21:02:39.453527 2691 log.cc:826] T 00000000000000000000000000000000 P 1743156790e4484991de906c6bfccc6c: Log is configured to *not* fsync() on all Append() calls
I20251215 21:02:39.454823 29124 external_mini_cluster.cc:949] 0 TS(s) registered with all masters
I20251215 21:02:39.455359 2691 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1743156790e4484991de906c6bfccc6c: No bootstrap required, opened a new log
I20251215 21:02:39.457798 2691 raft_consensus.cc:359] T 00000000000000000000000000000000 P 1743156790e4484991de906c6bfccc6c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1743156790e4484991de906c6bfccc6c" member_type: VOTER last_known_addr { host: "127.28.113.62" port: 41857 } }
I20251215 21:02:39.457976 2691 raft_consensus.cc:385] T 00000000000000000000000000000000 P 1743156790e4484991de906c6bfccc6c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251215 21:02:39.458042 2691 raft_consensus.cc:740] T 00000000000000000000000000000000 P 1743156790e4484991de906c6bfccc6c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1743156790e4484991de906c6bfccc6c, State: Initialized, Role: FOLLOWER
I20251215 21:02:39.458429 2691 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1743156790e4484991de906c6bfccc6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1743156790e4484991de906c6bfccc6c" member_type: VOTER last_known_addr { host: "127.28.113.62" port: 41857 } }
I20251215 21:02:39.458545 2691 raft_consensus.cc:399] T 00000000000000000000000000000000 P 1743156790e4484991de906c6bfccc6c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251215 21:02:39.458600 2691 raft_consensus.cc:493] T 00000000000000000000000000000000 P 1743156790e4484991de906c6bfccc6c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251215 21:02:39.458686 2691 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 1743156790e4484991de906c6bfccc6c [term 0 FOLLOWER]: Advancing to term 1
I20251215 21:02:39.459446 2691 raft_consensus.cc:515] T 00000000000000000000000000000000 P 1743156790e4484991de906c6bfccc6c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1743156790e4484991de906c6bfccc6c" member_type: VOTER last_known_addr { host: "127.28.113.62" port: 41857 } }
I20251215 21:02:39.459753 2691 leader_election.cc:304] T 00000000000000000000000000000000 P 1743156790e4484991de906c6bfccc6c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1743156790e4484991de906c6bfccc6c; no voters:
I20251215 21:02:39.460029 2691 leader_election.cc:290] T 00000000000000000000000000000000 P 1743156790e4484991de906c6bfccc6c [CANDIDATE]: Term 1 election: Requested vote from peers
I20251215 21:02:39.460125 2696 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 1743156790e4484991de906c6bfccc6c [term 1 FOLLOWER]: Leader election won for term 1
I20251215 21:02:39.460314 2696 raft_consensus.cc:697] T 00000000000000000000000000000000 P 1743156790e4484991de906c6bfccc6c [term 1 LEADER]: Becoming Leader. State: Replica: 1743156790e4484991de906c6bfccc6c, State: Running, Role: LEADER
I20251215 21:02:39.460577 2696 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1743156790e4484991de906c6bfccc6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1743156790e4484991de906c6bfccc6c" member_type: VOTER last_known_addr { host: "127.28.113.62" port: 41857 } }
I20251215 21:02:39.461009 2691 sys_catalog.cc:565] T 00000000000000000000000000000000 P 1743156790e4484991de906c6bfccc6c [sys.catalog]: configured and running, proceeding with master startup.
I20251215 21:02:39.462163 2698 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1743156790e4484991de906c6bfccc6c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1743156790e4484991de906c6bfccc6c. Latest consensus state: current_term: 1 leader_uuid: "1743156790e4484991de906c6bfccc6c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1743156790e4484991de906c6bfccc6c" member_type: VOTER last_known_addr { host: "127.28.113.62" port: 41857 } } }
I20251215 21:02:39.462275 2698 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1743156790e4484991de906c6bfccc6c [sys.catalog]: This master's current role is: LEADER
I20251215 21:02:39.462653 2705 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251215 21:02:39.462718 2697 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1743156790e4484991de906c6bfccc6c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1743156790e4484991de906c6bfccc6c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1743156790e4484991de906c6bfccc6c" member_type: VOTER last_known_addr { host: "127.28.113.62" port: 41857 } } }
I20251215 21:02:39.462801 2697 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1743156790e4484991de906c6bfccc6c [sys.catalog]: This master's current role is: LEADER
I20251215 21:02:39.465638 2705 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251215 21:02:39.470463 2705 catalog_manager.cc:1357] Generated new cluster ID: f059436ef3694952b03f93c5efafbb98
I20251215 21:02:39.470527 2705 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251215 21:02:39.480310 2705 catalog_manager.cc:1380] Generated new certificate authority record
I20251215 21:02:39.481506 2705 catalog_manager.cc:1514] Loading token signing keys...
I20251215 21:02:39.492466 2705 catalog_manager.cc:6027] T 00000000000000000000000000000000 P 1743156790e4484991de906c6bfccc6c: Generated new TSK 0
I20251215 21:02:39.493366 2705 catalog_manager.cc:1524] Initializing in-progress tserver states...
E20251215 21:02:39.527873 2721 webserver.cc:578] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20251215 21:02:39.527981 2721 webserver.cc:578] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20251215 21:02:39.529218 2694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:02:39.508231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35542 (local address 127.28.113.62:41857)
1215 21:02:39.508365 (+ 134us) server_negotiation.cc:207] Beginning negotiation
1215 21:02:39.508368 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1215 21:02:39.508477 (+ 109us) server_negotiation.cc:408] Connection header received
1215 21:02:39.508653 (+ 176us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1215 21:02:39.508656 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1215 21:02:39.508708 (+ 52us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1215 21:02:39.508788 (+ 80us) server_negotiation.cc:227] Negotiated authn=JWT
1215 21:02:39.509216 (+ 428us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:02:39.509729 (+ 513us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1215 21:02:39.510624 (+ 895us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:02:39.510770 (+ 146us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:02:39.510864 (+ 94us) server_negotiation.cc:366] Received JWT_EXCHANGE NegotiatePB request
1215 21:02:39.528859 (+ 17995us) server_negotiation.cc:395] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35055/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
1215 21:02:39.529102 (+ 243us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:35542: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35055/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":57}
W20251215 21:02:39.529345 2694 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:35542: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35055/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20251215 21:02:39.529460 2720 negotiation.cc:337] Failed RPC negotiation. Trace:
1215 21:02:39.508095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.113.62:41857 (local address 127.0.0.1:35542)
1215 21:02:39.508396 (+ 301us) negotiation.cc:107] Waiting for socket to connect
1215 21:02:39.508412 (+ 16us) client_negotiation.cc:174] Beginning negotiation
1215 21:02:39.508527 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1215 21:02:39.508815 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1215 21:02:39.508818 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1215 21:02:39.508828 (+ 10us) client_negotiation.cc:189] Negotiated authn=JWT
1215 21:02:39.509122 (+ 294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:02:39.509130 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:02:39.509838 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1215 21:02:39.509841 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1215 21:02:39.510519 (+ 678us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:02:39.510530 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:02:39.510654 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:02:39.510680 (+ 26us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
1215 21:02:39.529090 (+ 18410us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35055/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
1215 21:02:39.529274 (+ 184us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.28.113.62:41857: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35055/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":104,"threads_started":1}
I20251215 21:02:39.529986 29124 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu with pid 2624
2025-12-15T21:02:39Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (296 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2025-12-15T21:02:39Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-12-15T21:02:39Z Disabled control of system clock
I20251215 21:02:39.575042 29124 external_mini_cluster.cc:1366] Running /tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu
/tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1765832506145157-29124-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1765832506145157-29124-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1765832506145157-29124-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1765832506145157-29124-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.113.62:43325
--webserver_interface=127.28.113.62
--webserver_port=0
--builtin_ntp_servers=127.28.113.20:38053
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.113.62:43325
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1765832506145157-29124-0/testchainca.cert with env {}
W20251215 21:02:39.677685 2725 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251215 21:02:39.677930 2725 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251215 21:02:39.677991 2725 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251215 21:02:39.681277 2725 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251215 21:02:39.681344 2725 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251215 21:02:39.681370 2725 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251215 21:02:39.681401 2725 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251215 21:02:39.681430 2725 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20251215 21:02:39.681463 2725 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20251215 21:02:39.681491 2725 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20251215 21:02:39.685843 2725 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.113.20:38053
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1765832506145157-29124-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1765832506145157-29124-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.113.62:43325
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.113.62:43325
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1765832506145157-29124-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.113.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1765832506145157-29124-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2725
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1765832506145157-29124-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision dc741732bb6a40213e11a13869fad06a49a209a3
build type DEBUG
built by None at 15 Dec 2025 20:43:14 UTC on 24a791456cd2
build id 9530
I20251215 21:02:39.686923 2725 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251215 21:02:39.687706 2725 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251215 21:02:39.692940 2733 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251215 21:02:39.692943 2730 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251215 21:02:39.692950 2731 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251215 21:02:39.693147 2725 server_base.cc:1047] running on GCE node
I20251215 21:02:39.693672 2725 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251215 21:02:39.694528 2725 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251215 21:02:39.695672 2725 hybrid_clock.cc:648] HybridClock initialized: now 1765832559695648 us; error 38 us; skew 500 ppm
I20251215 21:02:39.697381 2725 webserver.cc:492] Webserver started at http://127.28.113.62:38917/ using document root <none> and password file <none>
I20251215 21:02:39.697914 2725 fs_manager.cc:362] Metadata directory not provided
I20251215 21:02:39.697966 2725 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251215 21:02:39.698139 2725 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251215 21:02:39.699793 2725 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1765832506145157-29124-0/minicluster-data/master-0/data/instance:
uuid: "730faf0196934694a1a5a497b84faaeb"
format_stamp: "Formatted at 2025-12-15 21:02:39 on dist-test-slave-bjsf"
server_key: "549e2a619e218566d684bd5f568dc249"
server_key_iv: "1fc4e37b11b1917ca7c547a2c0c4db3c"
server_key_version: "encryptionkey@0"
I20251215 21:02:39.700270 2725 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1765832506145157-29124-0/minicluster-data/master-0/wal/instance:
uuid: "730faf0196934694a1a5a497b84faaeb"
format_stamp: "Formatted at 2025-12-15 21:02:39 on dist-test-slave-bjsf"
server_key: "549e2a619e218566d684bd5f568dc249"
server_key_iv: "1fc4e37b11b1917ca7c547a2c0c4db3c"
server_key_version: "encryptionkey@0"
I20251215 21:02:39.703311 2725 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20251215 21:02:39.705304 2739 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251215 21:02:39.706224 2725 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20251215 21:02:39.706324 2725 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1765832506145157-29124-0/minicluster-data/master-0/data,/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1765832506145157-29124-0/minicluster-data/master-0/wal
uuid: "730faf0196934694a1a5a497b84faaeb"
format_stamp: "Formatted at 2025-12-15 21:02:39 on dist-test-slave-bjsf"
server_key: "549e2a619e218566d684bd5f568dc249"
server_key_iv: "1fc4e37b11b1917ca7c547a2c0c4db3c"
server_key_version: "encryptionkey@0"
I20251215 21:02:39.706403 2725 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1765832506145157-29124-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1765832506145157-29124-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1765832506145157-29124-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251215 21:02:39.736479 2725 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251215 21:02:39.737152 2725 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251215 21:02:39.737315 2725 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251215 21:02:39.744094 2725 rpc_server.cc:307] RPC server started. Bound to: 127.28.113.62:43325
I20251215 21:02:39.744110 2791 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.28.113.62:43325 every 8 connection(s)
I20251215 21:02:39.745033 2725 server_base.cc:1179] Dumped server information to /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1765832506145157-29124-0/minicluster-data/master-0/data/info.pb
I20251215 21:02:39.747565 2792 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 21:02:39.749949 29124 external_mini_cluster.cc:1428] Started /tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu as pid 2725
I20251215 21:02:39.750047 29124 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1765832506145157-29124-0/minicluster-data/master-0/wal/instance
I20251215 21:02:39.750249 29124 external_mini_cluster.cc:1468] Setting key 7eb4004bb40baf4cfcae97757ca7e863
I20251215 21:02:39.752926 2792 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 730faf0196934694a1a5a497b84faaeb: Bootstrap starting.
I20251215 21:02:39.755203 2792 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 730faf0196934694a1a5a497b84faaeb: Neither blocks nor log segments found. Creating new log.
I20251215 21:02:39.755949 2792 log.cc:826] T 00000000000000000000000000000000 P 730faf0196934694a1a5a497b84faaeb: Log is configured to *not* fsync() on all Append() calls
I20251215 21:02:39.757431 2795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:02:39.751407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45226 (local address 127.28.113.62:43325)
1215 21:02:39.751953 (+ 546us) server_negotiation.cc:207] Beginning negotiation
1215 21:02:39.752000 (+ 47us) server_negotiation.cc:400] Waiting for connection header
1215 21:02:39.752035 (+ 35us) server_negotiation.cc:408] Connection header received
1215 21:02:39.752522 (+ 487us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1215 21:02:39.752537 (+ 15us) server_negotiation.cc:462] Received NEGOTIATE request from client
1215 21:02:39.752884 (+ 347us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1215 21:02:39.753155 (+ 271us) server_negotiation.cc:227] Negotiated authn=SASL
1215 21:02:39.753652 (+ 497us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:02:39.754379 (+ 727us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1215 21:02:39.755189 (+ 810us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:02:39.755442 (+ 253us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:02:39.755931 (+ 489us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1215 21:02:39.755948 (+ 17us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1215 21:02:39.755985 (+ 37us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1215 21:02:39.756370 (+ 385us) server_negotiation.cc:923] calling sasl_server_start()
1215 21:02:39.756422 (+ 52us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1215 21:02:39.756440 (+ 18us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1215 21:02:39.756538 (+ 98us) server_negotiation.cc:1036] Waiting for connection context
1215 21:02:39.756688 (+ 150us) server_negotiation.cc:300] Negotiation successful
1215 21:02:39.756891 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":227,"thread_start_us":102,"threads_started":1}
I20251215 21:02:39.757850 2792 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 730faf0196934694a1a5a497b84faaeb: No bootstrap required, opened a new log
I20251215 21:02:39.759884 29124 external_mini_cluster.cc:949] 0 TS(s) registered with all masters
I20251215 21:02:39.760632 2792 raft_consensus.cc:359] T 00000000000000000000000000000000 P 730faf0196934694a1a5a497b84faaeb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "730faf0196934694a1a5a497b84faaeb" member_type: VOTER last_known_addr { host: "127.28.113.62" port: 43325 } }
I20251215 21:02:39.760824 2792 raft_consensus.cc:385] T 00000000000000000000000000000000 P 730faf0196934694a1a5a497b84faaeb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251215 21:02:39.760892 2792 raft_consensus.cc:740] T 00000000000000000000000000000000 P 730faf0196934694a1a5a497b84faaeb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 730faf0196934694a1a5a497b84faaeb, State: Initialized, Role: FOLLOWER
I20251215 21:02:39.761281 2792 consensus_queue.cc:260] T 00000000000000000000000000000000 P 730faf0196934694a1a5a497b84faaeb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "730faf0196934694a1a5a497b84faaeb" member_type: VOTER last_known_addr { host: "127.28.113.62" port: 43325 } }
I20251215 21:02:39.761421 2792 raft_consensus.cc:399] T 00000000000000000000000000000000 P 730faf0196934694a1a5a497b84faaeb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251215 21:02:39.761474 2792 raft_consensus.cc:493] T 00000000000000000000000000000000 P 730faf0196934694a1a5a497b84faaeb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251215 21:02:39.761538 2792 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 730faf0196934694a1a5a497b84faaeb [term 0 FOLLOWER]: Advancing to term 1
I20251215 21:02:39.762264 2792 raft_consensus.cc:515] T 00000000000000000000000000000000 P 730faf0196934694a1a5a497b84faaeb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "730faf0196934694a1a5a497b84faaeb" member_type: VOTER last_known_addr { host: "127.28.113.62" port: 43325 } }
I20251215 21:02:39.762542 2792 leader_election.cc:304] T 00000000000000000000000000000000 P 730faf0196934694a1a5a497b84faaeb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 730faf0196934694a1a5a497b84faaeb; no voters:
I20251215 21:02:39.762778 2792 leader_election.cc:290] T 00000000000000000000000000000000 P 730faf0196934694a1a5a497b84faaeb [CANDIDATE]: Term 1 election: Requested vote from peers
I20251215 21:02:39.762874 2797 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 730faf0196934694a1a5a497b84faaeb [term 1 FOLLOWER]: Leader election won for term 1
I20251215 21:02:39.763073 2797 raft_consensus.cc:697] T 00000000000000000000000000000000 P 730faf0196934694a1a5a497b84faaeb [term 1 LEADER]: Becoming Leader. State: Replica: 730faf0196934694a1a5a497b84faaeb, State: Running, Role: LEADER
I20251215 21:02:39.763339 2797 consensus_queue.cc:237] T 00000000000000000000000000000000 P 730faf0196934694a1a5a497b84faaeb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "730faf0196934694a1a5a497b84faaeb" member_type: VOTER last_known_addr { host: "127.28.113.62" port: 43325 } }
I20251215 21:02:39.763742 2792 sys_catalog.cc:565] T 00000000000000000000000000000000 P 730faf0196934694a1a5a497b84faaeb [sys.catalog]: configured and running, proceeding with master startup.
I20251215 21:02:39.764840 2798 sys_catalog.cc:455] T 00000000000000000000000000000000 P 730faf0196934694a1a5a497b84faaeb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "730faf0196934694a1a5a497b84faaeb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "730faf0196934694a1a5a497b84faaeb" member_type: VOTER last_known_addr { host: "127.28.113.62" port: 43325 } } }
I20251215 21:02:39.764840 2799 sys_catalog.cc:455] T 00000000000000000000000000000000 P 730faf0196934694a1a5a497b84faaeb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 730faf0196934694a1a5a497b84faaeb. Latest consensus state: current_term: 1 leader_uuid: "730faf0196934694a1a5a497b84faaeb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "730faf0196934694a1a5a497b84faaeb" member_type: VOTER last_known_addr { host: "127.28.113.62" port: 43325 } } }
I20251215 21:02:39.765005 2798 sys_catalog.cc:458] T 00000000000000000000000000000000 P 730faf0196934694a1a5a497b84faaeb [sys.catalog]: This master's current role is: LEADER
I20251215 21:02:39.765025 2799 sys_catalog.cc:458] T 00000000000000000000000000000000 P 730faf0196934694a1a5a497b84faaeb [sys.catalog]: This master's current role is: LEADER
I20251215 21:02:39.765432 2806 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251215 21:02:39.768303 2806 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251215 21:02:39.772971 2806 catalog_manager.cc:1357] Generated new cluster ID: 81e713e733e943f199148d632e890192
I20251215 21:02:39.773032 2806 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251215 21:02:39.788239 2806 catalog_manager.cc:1380] Generated new certificate authority record
I20251215 21:02:39.789103 2806 catalog_manager.cc:1514] Loading token signing keys...
I20251215 21:02:39.794658 2806 catalog_manager.cc:6027] T 00000000000000000000000000000000 P 730faf0196934694a1a5a497b84faaeb: Generated new TSK 0
I20251215 21:02:39.795285 2806 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251215 21:02:39.881951 2795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:02:39.807721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45240 (local address 127.28.113.62:43325)
1215 21:02:39.807851 (+ 130us) server_negotiation.cc:207] Beginning negotiation
1215 21:02:39.807855 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1215 21:02:39.807977 (+ 122us) server_negotiation.cc:408] Connection header received
1215 21:02:39.808140 (+ 163us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1215 21:02:39.808143 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1215 21:02:39.808193 (+ 50us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1215 21:02:39.808272 (+ 79us) server_negotiation.cc:227] Negotiated authn=JWT
1215 21:02:39.808658 (+ 386us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:02:39.809155 (+ 497us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1215 21:02:39.810024 (+ 869us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:02:39.810155 (+ 131us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:02:39.810244 (+ 89us) server_negotiation.cc:366] Received JWT_EXCHANGE NegotiatePB request
1215 21:02:39.881560 (+ 71316us) server_negotiation.cc:395] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Could not resolve hostname: Could not resolve host: default.url
1215 21:02:39.881819 (+ 259us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:45240: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Could not resolve hostname: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":50}
W20251215 21:02:39.882017 2795 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:45240: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Could not resolve hostname: Could not resolve host: default.url
W20251215 21:02:39.882170 2821 negotiation.cc:337] Failed RPC negotiation. Trace:
1215 21:02:39.807609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.113.62:43325 (local address 127.0.0.1:45240)
1215 21:02:39.807877 (+ 268us) negotiation.cc:107] Waiting for socket to connect
1215 21:02:39.807891 (+ 14us) client_negotiation.cc:174] Beginning negotiation
1215 21:02:39.808025 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1215 21:02:39.808284 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1215 21:02:39.808287 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1215 21:02:39.808294 (+ 7us) client_negotiation.cc:189] Negotiated authn=JWT
1215 21:02:39.808569 (+ 275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:02:39.808578 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:02:39.809248 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1215 21:02:39.809252 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1215 21:02:39.809921 (+ 669us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:02:39.809931 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:02:39.810064 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:02:39.810091 (+ 27us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
1215 21:02:39.881786 (+ 71695us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Could not resolve hostname: Could not resolve host: default.url
1215 21:02:39.881978 (+ 192us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.28.113.62:43325: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Could not resolve hostname: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":97,"threads_started":1}
I20251215 21:02:39.882869 29124 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu with pid 2725
2025-12-15T21:02:39Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (332 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (109 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (109 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765832506145157-29124-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Dec 15 21:02:40 dist-test-slave-bjsf krb5kdc[2833](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Dec 15 21:02:40 dist-test-slave-bjsf krb5kdc[2833](info): set up 2 sockets
Dec 15 21:02:40 dist-test-slave-bjsf krb5kdc[2833](info): commencing operation
krb5kdc: starting...
W20251215 21:02:41.932245 29124 mini_kdc.cc:121] Time spent starting KDC: real 1.812s user 0.004s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765832506145157-29124-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765832506145157-29124-0/krb5kdc/test-user.keytab.
Dec 15 21:02:42 dist-test-slave-bjsf krb5kdc[2833](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765832562, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-12-15T21:02:42Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-12-15T21:02:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.113.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.113.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.113.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765832506145157-29124-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.28.113.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765832506145157-29124-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.28.113.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.28.113.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.28.113.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765832506145157-29124-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.28.113.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765832506145157-29124-0/krb5kdc/kudu.keytab.
I20251215 21:02:42.058662 29124 external_mini_cluster.cc:1366] Running /tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu
/tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765832506145157-29124-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765832506145157-29124-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765832506145157-29124-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765832506145157-29124-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.113.62:36713
--webserver_interface=127.28.113.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765832506145157-29124-0/krb5kdc/kudu.keytab
--principal=kudu/127.28.113.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.28.113.20:45453
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.113.62:36713
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765832506145157-29124-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765832506145157-29124-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765832506145157-29124-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251215 21:02:42.159667 2849 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251215 21:02:42.159907 2849 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251215 21:02:42.160004 2849 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251215 21:02:42.163434 2849 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251215 21:02:42.163498 2849 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251215 21:02:42.163529 2849 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251215 21:02:42.163554 2849 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251215 21:02:42.163586 2849 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251215 21:02:42.167819 2849 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.113.20:45453
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765832506145157-29124-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765832506145157-29124-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.113.62:36713
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765832506145157-29124-0/krb5kdc/kudu.keytab
--principal=kudu/127.28.113.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.113.62:36713
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765832506145157-29124-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.113.62
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2849
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765832506145157-29124-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision dc741732bb6a40213e11a13869fad06a49a209a3
build type DEBUG
built by None at 15 Dec 2025 20:43:14 UTC on 24a791456cd2
build id 9530
I20251215 21:02:42.168900 2849 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251215 21:02:42.169720 2849 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251215 21:02:42.175287 2857 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251215 21:02:42.175386 2855 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251215 21:02:42.175473 2849 server_base.cc:1047] running on GCE node
W20251215 21:02:42.175290 2854 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251215 21:02:42.176230 2849 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251215 21:02:42.177109 2849 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251215 21:02:42.178262 2849 hybrid_clock.cc:648] HybridClock initialized: now 1765832562178251 us; error 28 us; skew 500 ppm
Dec 15 21:02:42 dist-test-slave-bjsf krb5kdc[2833](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765832562, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.113.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251215 21:02:42.180640 2849 init.cc:377] Logged in from keytab as kudu/127.28.113.62@KRBTEST.COM (short username kudu)
I20251215 21:02:42.181651 2849 webserver.cc:492] Webserver started at http://127.28.113.62:45779/ using document root <none> and password file <none>
I20251215 21:02:42.182189 2849 fs_manager.cc:362] Metadata directory not provided
I20251215 21:02:42.182238 2849 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251215 21:02:42.182410 2849 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251215 21:02:42.184129 2849 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765832506145157-29124-0/minicluster-data/master-0/data/instance:
uuid: "90a836f6d5c24a79ae526324d3f13464"
format_stamp: "Formatted at 2025-12-15 21:02:42 on dist-test-slave-bjsf"
server_key: "8782789ff7f01ac0b2adbdfaff16c584"
server_key_iv: "cb9a5ca512c4b29ab0e9a7ba8bbabc37"
server_key_version: "encryptionkey@0"
I20251215 21:02:42.184603 2849 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765832506145157-29124-0/minicluster-data/master-0/wal/instance:
uuid: "90a836f6d5c24a79ae526324d3f13464"
format_stamp: "Formatted at 2025-12-15 21:02:42 on dist-test-slave-bjsf"
server_key: "8782789ff7f01ac0b2adbdfaff16c584"
server_key_iv: "cb9a5ca512c4b29ab0e9a7ba8bbabc37"
server_key_version: "encryptionkey@0"
I20251215 21:02:42.187829 2849 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20251215 21:02:42.189972 2864 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251215 21:02:42.190948 2849 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20251215 21:02:42.191061 2849 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765832506145157-29124-0/minicluster-data/master-0/data,/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765832506145157-29124-0/minicluster-data/master-0/wal
uuid: "90a836f6d5c24a79ae526324d3f13464"
format_stamp: "Formatted at 2025-12-15 21:02:42 on dist-test-slave-bjsf"
server_key: "8782789ff7f01ac0b2adbdfaff16c584"
server_key_iv: "cb9a5ca512c4b29ab0e9a7ba8bbabc37"
server_key_version: "encryptionkey@0"
I20251215 21:02:42.191147 2849 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765832506145157-29124-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765832506145157-29124-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765832506145157-29124-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251215 21:02:42.204195 2849 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251215 21:02:42.206679 2849 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251215 21:02:42.206837 2849 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251215 21:02:42.213563 2916 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.28.113.62:36713 every 8 connection(s)
I20251215 21:02:42.213572 2849 rpc_server.cc:307] RPC server started. Bound to: 127.28.113.62:36713
I20251215 21:02:42.214573 2849 server_base.cc:1179] Dumped server information to /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765832506145157-29124-0/minicluster-data/master-0/data/info.pb
I20251215 21:02:42.217176 2917 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 21:02:42.222333 2917 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 90a836f6d5c24a79ae526324d3f13464: Bootstrap starting.
I20251215 21:02:42.223692 29124 external_mini_cluster.cc:1428] Started /tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu as pid 2849
I20251215 21:02:42.223827 29124 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765832506145157-29124-0/minicluster-data/master-0/wal/instance
I20251215 21:02:42.224079 29124 external_mini_cluster.cc:1468] Setting key ada852b5ddda30ea988797d0d53cefae
I20251215 21:02:42.224593 2917 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 90a836f6d5c24a79ae526324d3f13464: Neither blocks nor log segments found. Creating new log.
I20251215 21:02:42.225327 2917 log.cc:826] T 00000000000000000000000000000000 P 90a836f6d5c24a79ae526324d3f13464: Log is configured to *not* fsync() on all Append() calls
I20251215 21:02:42.227167 2917 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 90a836f6d5c24a79ae526324d3f13464: No bootstrap required, opened a new log
I20251215 21:02:42.229951 2917 raft_consensus.cc:359] T 00000000000000000000000000000000 P 90a836f6d5c24a79ae526324d3f13464 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90a836f6d5c24a79ae526324d3f13464" member_type: VOTER last_known_addr { host: "127.28.113.62" port: 36713 } }
I20251215 21:02:42.230178 2917 raft_consensus.cc:385] T 00000000000000000000000000000000 P 90a836f6d5c24a79ae526324d3f13464 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251215 21:02:42.230239 2917 raft_consensus.cc:740] T 00000000000000000000000000000000 P 90a836f6d5c24a79ae526324d3f13464 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90a836f6d5c24a79ae526324d3f13464, State: Initialized, Role: FOLLOWER
I20251215 21:02:42.230710 2917 consensus_queue.cc:260] T 00000000000000000000000000000000 P 90a836f6d5c24a79ae526324d3f13464 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90a836f6d5c24a79ae526324d3f13464" member_type: VOTER last_known_addr { host: "127.28.113.62" port: 36713 } }
I20251215 21:02:42.230868 2917 raft_consensus.cc:399] T 00000000000000000000000000000000 P 90a836f6d5c24a79ae526324d3f13464 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251215 21:02:42.230934 2917 raft_consensus.cc:493] T 00000000000000000000000000000000 P 90a836f6d5c24a79ae526324d3f13464 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251215 21:02:42.231022 2917 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 90a836f6d5c24a79ae526324d3f13464 [term 0 FOLLOWER]: Advancing to term 1
Dec 15 21:02:42 dist-test-slave-bjsf krb5kdc[2833](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765832562, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.113.62@KRBTEST.COM
I20251215 21:02:42.231794 2917 raft_consensus.cc:515] T 00000000000000000000000000000000 P 90a836f6d5c24a79ae526324d3f13464 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90a836f6d5c24a79ae526324d3f13464" member_type: VOTER last_known_addr { host: "127.28.113.62" port: 36713 } }
I20251215 21:02:42.232139 2917 leader_election.cc:304] T 00000000000000000000000000000000 P 90a836f6d5c24a79ae526324d3f13464 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 90a836f6d5c24a79ae526324d3f13464; no voters:
I20251215 21:02:42.232398 2917 leader_election.cc:290] T 00000000000000000000000000000000 P 90a836f6d5c24a79ae526324d3f13464 [CANDIDATE]: Term 1 election: Requested vote from peers
I20251215 21:02:42.232493 2922 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 90a836f6d5c24a79ae526324d3f13464 [term 1 FOLLOWER]: Leader election won for term 1
I20251215 21:02:42.232659 2922 raft_consensus.cc:697] T 00000000000000000000000000000000 P 90a836f6d5c24a79ae526324d3f13464 [term 1 LEADER]: Becoming Leader. State: Replica: 90a836f6d5c24a79ae526324d3f13464, State: Running, Role: LEADER
I20251215 21:02:42.232983 2922 consensus_queue.cc:237] T 00000000000000000000000000000000 P 90a836f6d5c24a79ae526324d3f13464 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90a836f6d5c24a79ae526324d3f13464" member_type: VOTER last_known_addr { host: "127.28.113.62" port: 36713 } }
I20251215 21:02:42.233541 2917 sys_catalog.cc:565] T 00000000000000000000000000000000 P 90a836f6d5c24a79ae526324d3f13464 [sys.catalog]: configured and running, proceeding with master startup.
I20251215 21:02:42.234090 2922 sys_catalog.cc:455] T 00000000000000000000000000000000 P 90a836f6d5c24a79ae526324d3f13464 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 90a836f6d5c24a79ae526324d3f13464. Latest consensus state: current_term: 1 leader_uuid: "90a836f6d5c24a79ae526324d3f13464" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90a836f6d5c24a79ae526324d3f13464" member_type: VOTER last_known_addr { host: "127.28.113.62" port: 36713 } } }
I20251215 21:02:42.234241 2922 sys_catalog.cc:458] T 00000000000000000000000000000000 P 90a836f6d5c24a79ae526324d3f13464 [sys.catalog]: This master's current role is: LEADER
I20251215 21:02:42.234380 2923 sys_catalog.cc:455] T 00000000000000000000000000000000 P 90a836f6d5c24a79ae526324d3f13464 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "90a836f6d5c24a79ae526324d3f13464" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90a836f6d5c24a79ae526324d3f13464" member_type: VOTER last_known_addr { host: "127.28.113.62" port: 36713 } } }
I20251215 21:02:42.234476 2923 sys_catalog.cc:458] T 00000000000000000000000000000000 P 90a836f6d5c24a79ae526324d3f13464 [sys.catalog]: This master's current role is: LEADER
I20251215 21:02:42.234624 2928 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251215 21:02:42.237589 2920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:02:42.225076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50328 (local address 127.28.113.62:36713)
1215 21:02:42.225500 (+ 424us) server_negotiation.cc:207] Beginning negotiation
1215 21:02:42.225509 (+ 9us) server_negotiation.cc:400] Waiting for connection header
1215 21:02:42.225542 (+ 33us) server_negotiation.cc:408] Connection header received
1215 21:02:42.226187 (+ 645us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1215 21:02:42.226208 (+ 21us) server_negotiation.cc:462] Received NEGOTIATE request from client
1215 21:02:42.226492 (+ 284us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1215 21:02:42.226759 (+ 267us) server_negotiation.cc:227] Negotiated authn=SASL
1215 21:02:42.227734 (+ 975us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:02:42.228510 (+ 776us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1215 21:02:42.229363 (+ 853us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:02:42.229587 (+ 224us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:02:42.231888 (+ 2301us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1215 21:02:42.231910 (+ 22us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1215 21:02:42.231924 (+ 14us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1215 21:02:42.231995 (+ 71us) server_negotiation.cc:923] calling sasl_server_start()
1215 21:02:42.234246 (+ 2251us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1215 21:02:42.234737 (+ 491us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1215 21:02:42.234742 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1215 21:02:42.234748 (+ 6us) server_negotiation.cc:968] calling sasl_server_step()
1215 21:02:42.234840 (+ 92us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1215 21:02:42.235116 (+ 276us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1215 21:02:42.235120 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1215 21:02:42.235121 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1215 21:02:42.235532 (+ 411us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1215 21:02:42.236311 (+ 779us) server_negotiation.cc:1036] Waiting for connection context
1215 21:02:42.236483 (+ 172us) server_negotiation.cc:300] Negotiation successful
1215 21:02:42.236747 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":285,"thread_start_us":129,"threads_started":1}
I20251215 21:02:42.237656 2928 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251215 21:02:42.243155 2928 catalog_manager.cc:1357] Generated new cluster ID: b70f39a912a948168bc7d2192413681b
I20251215 21:02:42.243276 2928 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251215 21:02:42.258199 2928 catalog_manager.cc:1380] Generated new certificate authority record
I20251215 21:02:42.259132 2928 catalog_manager.cc:1514] Loading token signing keys...
I20251215 21:02:42.265647 2928 catalog_manager.cc:6027] T 00000000000000000000000000000000 P 90a836f6d5c24a79ae526324d3f13464: Generated new TSK 0
I20251215 21:02:42.266480 2928 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.113.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.113.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.113.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765832506145157-29124-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.28.113.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765832506145157-29124-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.28.113.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.28.113.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.28.113.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765832506145157-29124-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.28.113.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765832506145157-29124-0/krb5kdc/kudu.keytab.
I20251215 21:02:42.311168 29124 external_mini_cluster.cc:1366] Running /tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu
/tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765832506145157-29124-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765832506145157-29124-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765832506145157-29124-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765832506145157-29124-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.113.1:0
--local_ip_for_outbound_sockets=127.28.113.1
--webserver_interface=127.28.113.1
--webserver_port=0
--tserver_master_addrs=127.28.113.62:36713
--keytab_file=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765832506145157-29124-0/krb5kdc/kudu.keytab
--principal=kudu/127.28.113.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.28.113.20:45453
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765832506145157-29124-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765832506145157-29124-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765832506145157-29124-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251215 21:02:42.414476 2945 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251215 21:02:42.414703 2945 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251215 21:02:42.414778 2945 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251215 21:02:42.418061 2945 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251215 21:02:42.418128 2945 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251215 21:02:42.418224 2945 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.113.1
I20251215 21:02:42.422407 2945 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.113.20:45453
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765832506145157-29124-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765832506145157-29124-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765832506145157-29124-0/krb5kdc/kudu.keytab
--principal=kudu/127.28.113.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.113.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765832506145157-29124-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.113.1
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.28.113.62:36713
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2945
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.113.1
--log_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765832506145157-29124-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision dc741732bb6a40213e11a13869fad06a49a209a3
build type DEBUG
built by None at 15 Dec 2025 20:43:14 UTC on 24a791456cd2
build id 9530
I20251215 21:02:42.423448 2945 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251215 21:02:42.424288 2945 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251215 21:02:42.430423 2951 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251215 21:02:42.430452 2950 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251215 21:02:42.430583 2953 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251215 21:02:42.430568 2945 server_base.cc:1047] running on GCE node
I20251215 21:02:42.431085 2945 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251215 21:02:42.431583 2945 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251215 21:02:42.432723 2945 hybrid_clock.cc:648] HybridClock initialized: now 1765832562432716 us; error 28 us; skew 500 ppm
Dec 15 21:02:42 dist-test-slave-bjsf krb5kdc[2833](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765832562, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.113.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251215 21:02:42.435149 2945 init.cc:377] Logged in from keytab as kudu/127.28.113.1@KRBTEST.COM (short username kudu)
I20251215 21:02:42.436206 2945 webserver.cc:492] Webserver started at http://127.28.113.1:43063/ using document root <none> and password file <none>
I20251215 21:02:42.436707 2945 fs_manager.cc:362] Metadata directory not provided
I20251215 21:02:42.436755 2945 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251215 21:02:42.436925 2945 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251215 21:02:42.438537 2945 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765832506145157-29124-0/minicluster-data/ts-0/data/instance:
uuid: "5dbfa86a5b3340869439fe22c4ae4dfe"
format_stamp: "Formatted at 2025-12-15 21:02:42 on dist-test-slave-bjsf"
server_key: "1a1a4bd6bf49cce5961700a84ccf5345"
server_key_iv: "512cacf6a67ce150fbb81c34ac65011f"
server_key_version: "encryptionkey@0"
I20251215 21:02:42.438963 2945 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765832506145157-29124-0/minicluster-data/ts-0/wal/instance:
uuid: "5dbfa86a5b3340869439fe22c4ae4dfe"
format_stamp: "Formatted at 2025-12-15 21:02:42 on dist-test-slave-bjsf"
server_key: "1a1a4bd6bf49cce5961700a84ccf5345"
server_key_iv: "512cacf6a67ce150fbb81c34ac65011f"
server_key_version: "encryptionkey@0"
I20251215 21:02:42.442097 2945 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20251215 21:02:42.444108 2960 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251215 21:02:42.445027 2945 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20251215 21:02:42.445127 2945 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765832506145157-29124-0/minicluster-data/ts-0/data,/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765832506145157-29124-0/minicluster-data/ts-0/wal
uuid: "5dbfa86a5b3340869439fe22c4ae4dfe"
format_stamp: "Formatted at 2025-12-15 21:02:42 on dist-test-slave-bjsf"
server_key: "1a1a4bd6bf49cce5961700a84ccf5345"
server_key_iv: "512cacf6a67ce150fbb81c34ac65011f"
server_key_version: "encryptionkey@0"
I20251215 21:02:42.445207 2945 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765832506145157-29124-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765832506145157-29124-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765832506145157-29124-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251215 21:02:42.474258 2945 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251215 21:02:42.477058 2945 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251215 21:02:42.477236 2945 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251215 21:02:42.477764 2945 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251215 21:02:42.478621 2945 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251215 21:02:42.478667 2945 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251215 21:02:42.478741 2945 ts_tablet_manager.cc:616] Registered 0 tablets
I20251215 21:02:42.478770 2945 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251215 21:02:42.487366 2945 rpc_server.cc:307] RPC server started. Bound to: 127.28.113.1:40603
I20251215 21:02:42.487422 3073 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.28.113.1:40603 every 8 connection(s)
I20251215 21:02:42.488332 2945 server_base.cc:1179] Dumped server information to /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765832506145157-29124-0/minicluster-data/ts-0/data/info.pb
Dec 15 21:02:42 dist-test-slave-bjsf krb5kdc[2833](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765832562, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.113.1@KRBTEST.COM for kudu/127.28.113.62@KRBTEST.COM
I20251215 21:02:42.496490 29124 external_mini_cluster.cc:1428] Started /tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu as pid 2945
I20251215 21:02:42.496592 29124 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765832506145157-29124-0/minicluster-data/ts-0/wal/instance
I20251215 21:02:42.496811 29124 external_mini_cluster.cc:1468] Setting key 303061fc9563e6cfbc3d2a8266e5796f
I20251215 21:02:42.500284 2920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:02:42.490042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.113.1:37555 (local address 127.28.113.62:36713)
1215 21:02:42.490216 (+ 174us) server_negotiation.cc:207] Beginning negotiation
1215 21:02:42.490222 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1215 21:02:42.490871 (+ 649us) server_negotiation.cc:408] Connection header received
1215 21:02:42.491664 (+ 793us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1215 21:02:42.491667 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1215 21:02:42.491734 (+ 67us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1215 21:02:42.491818 (+ 84us) server_negotiation.cc:227] Negotiated authn=SASL
1215 21:02:42.493418 (+ 1600us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:02:42.493948 (+ 530us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1215 21:02:42.494529 (+ 581us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:02:42.494654 (+ 125us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:02:42.496847 (+ 2193us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1215 21:02:42.496873 (+ 26us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1215 21:02:42.496877 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1215 21:02:42.496908 (+ 31us) server_negotiation.cc:923] calling sasl_server_start()
1215 21:02:42.498509 (+ 1601us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1215 21:02:42.498960 (+ 451us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1215 21:02:42.498964 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1215 21:02:42.498966 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1215 21:02:42.499017 (+ 51us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1215 21:02:42.499238 (+ 221us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1215 21:02:42.499241 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1215 21:02:42.499243 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1215 21:02:42.499416 (+ 173us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1215 21:02:42.499505 (+ 89us) server_negotiation.cc:1036] Waiting for connection context
1215 21:02:42.499986 (+ 481us) server_negotiation.cc:300] Negotiation successful
1215 21:02:42.500123 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20251215 21:02:42.500818 3076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:02:42.490302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.113.62:36713 (local address 127.28.113.1:37555)
1215 21:02:42.490735 (+ 433us) negotiation.cc:107] Waiting for socket to connect
1215 21:02:42.490771 (+ 36us) client_negotiation.cc:174] Beginning negotiation
1215 21:02:42.491471 (+ 700us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1215 21:02:42.491988 (+ 517us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1215 21:02:42.491996 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
1215 21:02:42.492359 (+ 363us) client_negotiation.cc:189] Negotiated authn=SASL
1215 21:02:42.493252 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:02:42.493267 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:02:42.494045 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1215 21:02:42.494048 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1215 21:02:42.494448 (+ 400us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:02:42.494454 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:02:42.494615 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:02:42.495228 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1215 21:02:42.495251 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
1215 21:02:42.496683 (+ 1432us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1215 21:02:42.498620 (+ 1937us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1215 21:02:42.498624 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1215 21:02:42.498633 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
1215 21:02:42.498872 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1215 21:02:42.499092 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1215 21:02:42.499095 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1215 21:02:42.499096 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1215 21:02:42.499172 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1215 21:02:42.499494 (+ 322us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1215 21:02:42.499499 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1215 21:02:42.499732 (+ 233us) client_negotiation.cc:769] Sending connection context
1215 21:02:42.499898 (+ 166us) client_negotiation.cc:240] Negotiation successful
1215 21:02:42.500136 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":119,"threads_started":1}
I20251215 21:02:42.501905 3074 heartbeater.cc:344] Connected to a master server at 127.28.113.62:36713
I20251215 21:02:42.502136 3074 heartbeater.cc:461] Registering TS with master...
I20251215 21:02:42.502630 3074 heartbeater.cc:507] Master 127.28.113.62:36713 requested a full tablet report, sending...
I20251215 21:02:42.503942 2881 ts_manager.cc:194] Registered new tserver with Master: 5dbfa86a5b3340869439fe22c4ae4dfe (127.28.113.1:40603)
WARNING: no policy specified for kudu/127.28.113.2@KRBTEST.COM; defaulting to no policy
I20251215 21:02:42.505064 2881 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.113.1@KRBTEST.COM'} at 127.28.113.1:37555
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.113.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.113.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765832506145157-29124-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.28.113.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765832506145157-29124-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.28.113.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.28.113.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.28.113.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765832506145157-29124-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.28.113.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765832506145157-29124-0/krb5kdc/kudu.keytab.
I20251215 21:02:42.539077 29124 external_mini_cluster.cc:1366] Running /tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu
/tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765832506145157-29124-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765832506145157-29124-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765832506145157-29124-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765832506145157-29124-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.113.2:0
--local_ip_for_outbound_sockets=127.28.113.2
--webserver_interface=127.28.113.2
--webserver_port=0
--tserver_master_addrs=127.28.113.62:36713
--keytab_file=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765832506145157-29124-0/krb5kdc/kudu.keytab
--principal=kudu/127.28.113.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.28.113.20:45453
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765832506145157-29124-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765832506145157-29124-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765832506145157-29124-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251215 21:02:42.646091 3081 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251215 21:02:42.646325 3081 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251215 21:02:42.646386 3081 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251215 21:02:42.649571 3081 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251215 21:02:42.649652 3081 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251215 21:02:42.649752 3081 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.113.2
I20251215 21:02:42.653817 3081 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.113.20:45453
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765832506145157-29124-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765832506145157-29124-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765832506145157-29124-0/krb5kdc/kudu.keytab
--principal=kudu/127.28.113.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.113.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765832506145157-29124-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.113.2
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.28.113.62:36713
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3081
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.113.2
--log_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765832506145157-29124-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision dc741732bb6a40213e11a13869fad06a49a209a3
build type DEBUG
built by None at 15 Dec 2025 20:43:14 UTC on 24a791456cd2
build id 9530
I20251215 21:02:42.654794 3081 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251215 21:02:42.655572 3081 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251215 21:02:42.661731 3086 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251215 21:02:42.661717 3089 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251215 21:02:42.661871 3081 server_base.cc:1047] running on GCE node
W20251215 21:02:42.661722 3087 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251215 21:02:42.662369 3081 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251215 21:02:42.662894 3081 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251215 21:02:42.664023 3081 hybrid_clock.cc:648] HybridClock initialized: now 1765832562664001 us; error 37 us; skew 500 ppm
Dec 15 21:02:42 dist-test-slave-bjsf krb5kdc[2833](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765832562, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.113.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251215 21:02:42.666563 3081 init.cc:377] Logged in from keytab as kudu/127.28.113.2@KRBTEST.COM (short username kudu)
I20251215 21:02:42.667559 3081 webserver.cc:492] Webserver started at http://127.28.113.2:43755/ using document root <none> and password file <none>
I20251215 21:02:42.668154 3081 fs_manager.cc:362] Metadata directory not provided
I20251215 21:02:42.668212 3081 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251215 21:02:42.668390 3081 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251215 21:02:42.670080 3081 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765832506145157-29124-0/minicluster-data/ts-1/data/instance:
uuid: "55a3ccf6eca04250be4282649429a5b7"
format_stamp: "Formatted at 2025-12-15 21:02:42 on dist-test-slave-bjsf"
server_key: "30a3384d6fd7a9ffd00786dea47b2525"
server_key_iv: "a7cf8fbcec851a9c07f5a726a993ed23"
server_key_version: "encryptionkey@0"
I20251215 21:02:42.670547 3081 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765832506145157-29124-0/minicluster-data/ts-1/wal/instance:
uuid: "55a3ccf6eca04250be4282649429a5b7"
format_stamp: "Formatted at 2025-12-15 21:02:42 on dist-test-slave-bjsf"
server_key: "30a3384d6fd7a9ffd00786dea47b2525"
server_key_iv: "a7cf8fbcec851a9c07f5a726a993ed23"
server_key_version: "encryptionkey@0"
I20251215 21:02:42.673995 3081 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20251215 21:02:42.676116 3096 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251215 21:02:42.677073 3081 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20251215 21:02:42.677171 3081 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765832506145157-29124-0/minicluster-data/ts-1/data,/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765832506145157-29124-0/minicluster-data/ts-1/wal
uuid: "55a3ccf6eca04250be4282649429a5b7"
format_stamp: "Formatted at 2025-12-15 21:02:42 on dist-test-slave-bjsf"
server_key: "30a3384d6fd7a9ffd00786dea47b2525"
server_key_iv: "a7cf8fbcec851a9c07f5a726a993ed23"
server_key_version: "encryptionkey@0"
I20251215 21:02:42.677258 3081 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765832506145157-29124-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765832506145157-29124-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765832506145157-29124-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251215 21:02:42.685745 3081 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251215 21:02:42.688452 3081 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251215 21:02:42.688606 3081 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251215 21:02:42.689098 3081 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251215 21:02:42.689910 3081 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251215 21:02:42.689963 3081 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251215 21:02:42.690014 3081 ts_tablet_manager.cc:616] Registered 0 tablets
I20251215 21:02:42.690039 3081 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251215 21:02:42.698393 3081 rpc_server.cc:307] RPC server started. Bound to: 127.28.113.2:38339
I20251215 21:02:42.698416 3209 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.28.113.2:38339 every 8 connection(s)
I20251215 21:02:42.699287 3081 server_base.cc:1179] Dumped server information to /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765832506145157-29124-0/minicluster-data/ts-1/data/info.pb
I20251215 21:02:42.703621 29124 external_mini_cluster.cc:1428] Started /tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu as pid 3081
I20251215 21:02:42.703713 29124 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765832506145157-29124-0/minicluster-data/ts-1/wal/instance
I20251215 21:02:42.703931 29124 external_mini_cluster.cc:1468] Setting key 1a89126745fd83d5fa2dacf48e510f0f
Dec 15 21:02:42 dist-test-slave-bjsf krb5kdc[2833](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765832562, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.113.2@KRBTEST.COM for kudu/127.28.113.62@KRBTEST.COM
I20251215 21:02:42.711331 2920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:02:42.700878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.113.2:57393 (local address 127.28.113.62:36713)
1215 21:02:42.701019 (+ 141us) server_negotiation.cc:207] Beginning negotiation
1215 21:02:42.701022 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1215 21:02:42.701665 (+ 643us) server_negotiation.cc:408] Connection header received
1215 21:02:42.702521 (+ 856us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1215 21:02:42.702525 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1215 21:02:42.702603 (+ 78us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1215 21:02:42.702674 (+ 71us) server_negotiation.cc:227] Negotiated authn=SASL
1215 21:02:42.704441 (+ 1767us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:02:42.704948 (+ 507us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1215 21:02:42.705580 (+ 632us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:02:42.705771 (+ 191us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:02:42.708001 (+ 2230us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1215 21:02:42.708020 (+ 19us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1215 21:02:42.708023 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1215 21:02:42.708049 (+ 26us) server_negotiation.cc:923] calling sasl_server_start()
1215 21:02:42.709530 (+ 1481us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1215 21:02:42.709994 (+ 464us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1215 21:02:42.709998 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1215 21:02:42.710000 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1215 21:02:42.710044 (+ 44us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1215 21:02:42.710296 (+ 252us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1215 21:02:42.710299 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1215 21:02:42.710301 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1215 21:02:42.710456 (+ 155us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1215 21:02:42.710542 (+ 86us) server_negotiation.cc:1036] Waiting for connection context
1215 21:02:42.711038 (+ 496us) server_negotiation.cc:300] Negotiation successful
1215 21:02:42.711167 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
WARNING: no policy specified for kudu/127.28.113.3@KRBTEST.COM; defaulting to no policy
I20251215 21:02:42.711987 3212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:02:42.701104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.113.62:36713 (local address 127.28.113.2:57393)
1215 21:02:42.701517 (+ 413us) negotiation.cc:107] Waiting for socket to connect
1215 21:02:42.701553 (+ 36us) client_negotiation.cc:174] Beginning negotiation
1215 21:02:42.702311 (+ 758us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1215 21:02:42.702846 (+ 535us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1215 21:02:42.702853 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
1215 21:02:42.703254 (+ 401us) client_negotiation.cc:189] Negotiated authn=SASL
1215 21:02:42.704291 (+ 1037us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:02:42.704308 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:02:42.705064 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1215 21:02:42.705067 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1215 21:02:42.705468 (+ 401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:02:42.705475 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:02:42.705684 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:02:42.706283 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1215 21:02:42.706308 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
1215 21:02:42.707820 (+ 1512us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1215 21:02:42.709646 (+ 1826us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1215 21:02:42.709652 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1215 21:02:42.709666 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
1215 21:02:42.709900 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1215 21:02:42.710122 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1215 21:02:42.710125 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1215 21:02:42.710126 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1215 21:02:42.710215 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1215 21:02:42.710552 (+ 337us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1215 21:02:42.710560 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1215 21:02:42.710796 (+ 236us) client_negotiation.cc:769] Sending connection context
1215 21:02:42.710994 (+ 198us) client_negotiation.cc:240] Negotiation successful
1215 21:02:42.711277 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":107,"threads_started":1}
I20251215 21:02:42.713048 3210 heartbeater.cc:344] Connected to a master server at 127.28.113.62:36713
I20251215 21:02:42.713316 3210 heartbeater.cc:461] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.113.3@KRBTEST.COM" created.
I20251215 21:02:42.713930 3210 heartbeater.cc:507] Master 127.28.113.62:36713 requested a full tablet report, sending...
I20251215 21:02:42.714896 2881 ts_manager.cc:194] Registered new tserver with Master: 55a3ccf6eca04250be4282649429a5b7 (127.28.113.2:38339)
I20251215 21:02:42.715585 2881 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.113.2@KRBTEST.COM'} at 127.28.113.2:57393
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.113.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765832506145157-29124-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.28.113.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765832506145157-29124-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.28.113.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.28.113.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.28.113.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765832506145157-29124-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.28.113.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765832506145157-29124-0/krb5kdc/kudu.keytab.
I20251215 21:02:42.746263 29124 external_mini_cluster.cc:1366] Running /tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu
/tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765832506145157-29124-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765832506145157-29124-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765832506145157-29124-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765832506145157-29124-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.113.3:0
--local_ip_for_outbound_sockets=127.28.113.3
--webserver_interface=127.28.113.3
--webserver_port=0
--tserver_master_addrs=127.28.113.62:36713
--keytab_file=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765832506145157-29124-0/krb5kdc/kudu.keytab
--principal=kudu/127.28.113.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.28.113.20:45453
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765832506145157-29124-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765832506145157-29124-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765832506145157-29124-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251215 21:02:42.845082 3217 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251215 21:02:42.845288 3217 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251215 21:02:42.845352 3217 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251215 21:02:42.848519 3217 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251215 21:02:42.848577 3217 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251215 21:02:42.848665 3217 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.113.3
I20251215 21:02:42.852686 3217 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.113.20:45453
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765832506145157-29124-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765832506145157-29124-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765832506145157-29124-0/krb5kdc/kudu.keytab
--principal=kudu/127.28.113.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.113.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765832506145157-29124-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.113.3
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.28.113.62:36713
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3217
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.113.3
--log_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765832506145157-29124-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision dc741732bb6a40213e11a13869fad06a49a209a3
build type DEBUG
built by None at 15 Dec 2025 20:43:14 UTC on 24a791456cd2
build id 9530
I20251215 21:02:42.853660 3217 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251215 21:02:42.854463 3217 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251215 21:02:42.860589 3225 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251215 21:02:42.860584 3223 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251215 21:02:42.860584 3222 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251215 21:02:42.860960 3217 server_base.cc:1047] running on GCE node
I20251215 21:02:42.861274 3217 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251215 21:02:42.861730 3217 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251215 21:02:42.862861 3217 hybrid_clock.cc:648] HybridClock initialized: now 1765832562862842 us; error 35 us; skew 500 ppm
Dec 15 21:02:42 dist-test-slave-bjsf krb5kdc[2833](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765832562, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.113.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251215 21:02:42.865332 3217 init.cc:377] Logged in from keytab as kudu/127.28.113.3@KRBTEST.COM (short username kudu)
I20251215 21:02:42.866292 3217 webserver.cc:492] Webserver started at http://127.28.113.3:39057/ using document root <none> and password file <none>
I20251215 21:02:42.866779 3217 fs_manager.cc:362] Metadata directory not provided
I20251215 21:02:42.866833 3217 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251215 21:02:42.866998 3217 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251215 21:02:42.868628 3217 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765832506145157-29124-0/minicluster-data/ts-2/data/instance:
uuid: "2f327e9b8ac347cf93a7661a3a2617e9"
format_stamp: "Formatted at 2025-12-15 21:02:42 on dist-test-slave-bjsf"
server_key: "37df0fcba4d4716bd42a7062424f8ba6"
server_key_iv: "a9521a6a2f5487f73d09f38ffe820d44"
server_key_version: "encryptionkey@0"
I20251215 21:02:42.869057 3217 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765832506145157-29124-0/minicluster-data/ts-2/wal/instance:
uuid: "2f327e9b8ac347cf93a7661a3a2617e9"
format_stamp: "Formatted at 2025-12-15 21:02:42 on dist-test-slave-bjsf"
server_key: "37df0fcba4d4716bd42a7062424f8ba6"
server_key_iv: "a9521a6a2f5487f73d09f38ffe820d44"
server_key_version: "encryptionkey@0"
I20251215 21:02:42.872177 3217 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20251215 21:02:42.874107 3232 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251215 21:02:42.874989 3217 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20251215 21:02:42.875080 3217 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765832506145157-29124-0/minicluster-data/ts-2/data,/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765832506145157-29124-0/minicluster-data/ts-2/wal
uuid: "2f327e9b8ac347cf93a7661a3a2617e9"
format_stamp: "Formatted at 2025-12-15 21:02:42 on dist-test-slave-bjsf"
server_key: "37df0fcba4d4716bd42a7062424f8ba6"
server_key_iv: "a9521a6a2f5487f73d09f38ffe820d44"
server_key_version: "encryptionkey@0"
I20251215 21:02:42.875157 3217 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765832506145157-29124-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765832506145157-29124-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765832506145157-29124-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251215 21:02:42.886688 3217 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251215 21:02:42.889148 3217 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251215 21:02:42.889299 3217 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251215 21:02:42.889758 3217 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251215 21:02:42.890550 3217 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251215 21:02:42.890599 3217 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251215 21:02:42.890648 3217 ts_tablet_manager.cc:616] Registered 0 tablets
I20251215 21:02:42.890674 3217 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251215 21:02:42.899044 3217 rpc_server.cc:307] RPC server started. Bound to: 127.28.113.3:46637
I20251215 21:02:42.899116 3345 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.28.113.3:46637 every 8 connection(s)
I20251215 21:02:42.899890 3217 server_base.cc:1179] Dumped server information to /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765832506145157-29124-0/minicluster-data/ts-2/data/info.pb
I20251215 21:02:42.900795 29124 external_mini_cluster.cc:1428] Started /tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu as pid 3217
I20251215 21:02:42.900902 29124 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765832506145157-29124-0/minicluster-data/ts-2/wal/instance
I20251215 21:02:42.901113 29124 external_mini_cluster.cc:1468] Setting key 1df525e18efe5b41fe005a486865a18c
Dec 15 21:02:42 dist-test-slave-bjsf krb5kdc[2833](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765832562, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.113.3@KRBTEST.COM for kudu/127.28.113.62@KRBTEST.COM
I20251215 21:02:42.911216 2920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:02:42.901532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.113.3:57177 (local address 127.28.113.62:36713)
1215 21:02:42.901684 (+ 152us) server_negotiation.cc:207] Beginning negotiation
1215 21:02:42.901689 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1215 21:02:42.902369 (+ 680us) server_negotiation.cc:408] Connection header received
1215 21:02:42.903329 (+ 960us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1215 21:02:42.903332 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1215 21:02:42.903381 (+ 49us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1215 21:02:42.903449 (+ 68us) server_negotiation.cc:227] Negotiated authn=SASL
1215 21:02:42.904976 (+ 1527us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:02:42.905485 (+ 509us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1215 21:02:42.906066 (+ 581us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:02:42.906199 (+ 133us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:02:42.908211 (+ 2012us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1215 21:02:42.908228 (+ 17us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1215 21:02:42.908231 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1215 21:02:42.908256 (+ 25us) server_negotiation.cc:923] calling sasl_server_start()
1215 21:02:42.909510 (+ 1254us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1215 21:02:42.909953 (+ 443us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1215 21:02:42.909956 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1215 21:02:42.909957 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1215 21:02:42.910007 (+ 50us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1215 21:02:42.910248 (+ 241us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1215 21:02:42.910251 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1215 21:02:42.910253 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1215 21:02:42.910398 (+ 145us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1215 21:02:42.910480 (+ 82us) server_negotiation.cc:1036] Waiting for connection context
1215 21:02:42.910938 (+ 458us) server_negotiation.cc:300] Negotiation successful
1215 21:02:42.911061 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20251215 21:02:42.911808 3348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:02:42.901820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.113.62:36713 (local address 127.28.113.3:57177)
1215 21:02:42.902267 (+ 447us) negotiation.cc:107] Waiting for socket to connect
1215 21:02:42.902300 (+ 33us) client_negotiation.cc:174] Beginning negotiation
1215 21:02:42.903165 (+ 865us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1215 21:02:42.903573 (+ 408us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1215 21:02:42.903580 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
1215 21:02:42.903937 (+ 357us) client_negotiation.cc:189] Negotiated authn=SASL
1215 21:02:42.904828 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:02:42.904841 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:02:42.905570 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1215 21:02:42.905572 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1215 21:02:42.905966 (+ 394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:02:42.905972 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:02:42.906123 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:02:42.906624 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1215 21:02:42.906643 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
1215 21:02:42.908052 (+ 1409us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1215 21:02:42.909619 (+ 1567us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1215 21:02:42.909625 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1215 21:02:42.909636 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
1215 21:02:42.909863 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1215 21:02:42.910083 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1215 21:02:42.910086 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1215 21:02:42.910088 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1215 21:02:42.910168 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1215 21:02:42.910490 (+ 322us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1215 21:02:42.910498 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1215 21:02:42.910707 (+ 209us) client_negotiation.cc:769] Sending connection context
1215 21:02:42.910891 (+ 184us) client_negotiation.cc:240] Negotiation successful
1215 21:02:42.911142 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":138,"threads_started":1}
I20251215 21:02:42.912765 3346 heartbeater.cc:344] Connected to a master server at 127.28.113.62:36713
I20251215 21:02:42.912990 3346 heartbeater.cc:461] Registering TS with master...
I20251215 21:02:42.913483 3346 heartbeater.cc:507] Master 127.28.113.62:36713 requested a full tablet report, sending...
I20251215 21:02:42.914424 2881 ts_manager.cc:194] Registered new tserver with Master: 2f327e9b8ac347cf93a7661a3a2617e9 (127.28.113.3:46637)
I20251215 21:02:42.914965 2881 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.113.3@KRBTEST.COM'} at 127.28.113.3:57177
I20251215 21:02:42.926293 29124 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765832506145157-29124-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765832506145157-29124-0/krb5kdc/test-admin.keytab.
Dec 15 21:02:42 dist-test-slave-bjsf krb5kdc[2833](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765832562, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251215 21:02:42.939890 29124 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20251215 21:02:42.942287 3353 negotiation.cc:337] Failed RPC negotiation. Trace:
1215 21:02:42.941141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41950 (local address 127.28.113.1:40603)
1215 21:02:42.941464 (+ 323us) server_negotiation.cc:207] Beginning negotiation
1215 21:02:42.941470 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1215 21:02:42.941492 (+ 22us) server_negotiation.cc:408] Connection header received
1215 21:02:42.941541 (+ 49us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1215 21:02:42.941546 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1215 21:02:42.941703 (+ 157us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1215 21:02:42.941824 (+ 121us) server_negotiation.cc:227] Negotiated authn=SASL
1215 21:02:42.942173 (+ 349us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41950: BlockingRecv error: recv got EOF from 127.0.0.1:41950 (error 108)
Metrics: {"server-negotiator.queue_time_us":226,"thread_start_us":135,"threads_started":1}
W20251215 21:02:42.942416 29124 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765832506145157-29124-0/krb5kdc/krb5cc' of type FILE
Dec 15 21:02:42 dist-test-slave-bjsf krb5kdc[2833](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765832562, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251215 21:02:42.943325 29124 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Dec 15 21:02:42 dist-test-slave-bjsf krb5kdc[2833](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765832562, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251215 21:02:42.944198 29124 init.cc:295] Successfully reacquired a new kerberos TGT
Dec 15 21:02:42 dist-test-slave-bjsf krb5kdc[2833](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765832562, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.113.1@KRBTEST.COM
I20251215 21:02:42.951874 3353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:02:42.944917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41962 (local address 127.28.113.1:40603)
1215 21:02:42.945032 (+ 115us) server_negotiation.cc:207] Beginning negotiation
1215 21:02:42.945036 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1215 21:02:42.945053 (+ 17us) server_negotiation.cc:408] Connection header received
1215 21:02:42.945171 (+ 118us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1215 21:02:42.945174 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1215 21:02:42.945219 (+ 45us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1215 21:02:42.945281 (+ 62us) server_negotiation.cc:227] Negotiated authn=SASL
1215 21:02:42.945864 (+ 583us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:02:42.946555 (+ 691us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1215 21:02:42.947186 (+ 631us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:02:42.947414 (+ 228us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:02:42.949169 (+ 1755us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1215 21:02:42.949187 (+ 18us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1215 21:02:42.949198 (+ 11us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1215 21:02:42.949222 (+ 24us) server_negotiation.cc:923] calling sasl_server_start()
1215 21:02:42.950636 (+ 1414us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1215 21:02:42.950974 (+ 338us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1215 21:02:42.950978 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1215 21:02:42.950979 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1215 21:02:42.951023 (+ 44us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1215 21:02:42.951163 (+ 140us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1215 21:02:42.951165 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1215 21:02:42.951166 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1215 21:02:42.951326 (+ 160us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1215 21:02:42.951476 (+ 150us) server_negotiation.cc:1036] Waiting for connection context
1215 21:02:42.951586 (+ 110us) server_negotiation.cc:300] Negotiation successful
1215 21:02:42.951720 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
W20251215 21:02:42.955039 3353 negotiation.cc:337] Failed RPC negotiation. Trace:
1215 21:02:42.954235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41968 (local address 127.28.113.1:40603)
1215 21:02:42.954344 (+ 109us) server_negotiation.cc:207] Beginning negotiation
1215 21:02:42.954348 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1215 21:02:42.954409 (+ 61us) server_negotiation.cc:408] Connection header received
1215 21:02:42.954509 (+ 100us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1215 21:02:42.954512 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1215 21:02:42.954554 (+ 42us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1215 21:02:42.954614 (+ 60us) server_negotiation.cc:227] Negotiated authn=SASL
1215 21:02:42.954951 (+ 337us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41968: BlockingRecv error: recv got EOF from 127.0.0.1:41968 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
W20251215 21:02:42.955132 29124 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765832506145157-29124-0/krb5kdc/krb5cc' of type FILE
Dec 15 21:02:42 dist-test-slave-bjsf krb5kdc[2833](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765832562, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251215 21:02:42.956044 29124 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Dec 15 21:02:42 dist-test-slave-bjsf krb5kdc[2833](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765832562, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251215 21:02:42.956775 29124 init.cc:295] Successfully reacquired a new kerberos TGT
Dec 15 21:02:42 dist-test-slave-bjsf krb5kdc[2833](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765832562, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.113.1@KRBTEST.COM
I20251215 21:02:42.963292 3353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:02:42.957489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41980 (local address 127.28.113.1:40603)
1215 21:02:42.957595 (+ 106us) server_negotiation.cc:207] Beginning negotiation
1215 21:02:42.957599 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1215 21:02:42.957648 (+ 49us) server_negotiation.cc:408] Connection header received
1215 21:02:42.957728 (+ 80us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1215 21:02:42.957731 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1215 21:02:42.957767 (+ 36us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1215 21:02:42.957826 (+ 59us) server_negotiation.cc:227] Negotiated authn=SASL
1215 21:02:42.958310 (+ 484us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:02:42.958757 (+ 447us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1215 21:02:42.959275 (+ 518us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:02:42.959411 (+ 136us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:02:42.961056 (+ 1645us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1215 21:02:42.961070 (+ 14us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1215 21:02:42.961072 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1215 21:02:42.961093 (+ 21us) server_negotiation.cc:923] calling sasl_server_start()
1215 21:02:42.962306 (+ 1213us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1215 21:02:42.962585 (+ 279us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1215 21:02:42.962588 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1215 21:02:42.962589 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1215 21:02:42.962622 (+ 33us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1215 21:02:42.962756 (+ 134us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1215 21:02:42.962758 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1215 21:02:42.962759 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1215 21:02:42.962861 (+ 102us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1215 21:02:42.962945 (+ 84us) server_negotiation.cc:1036] Waiting for connection context
1215 21:02:42.963079 (+ 134us) server_negotiation.cc:300] Negotiation successful
1215 21:02:42.963157 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
W20251215 21:02:42.965584 3353 negotiation.cc:337] Failed RPC negotiation. Trace:
1215 21:02:42.964831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41986 (local address 127.28.113.1:40603)
1215 21:02:42.964953 (+ 122us) server_negotiation.cc:207] Beginning negotiation
1215 21:02:42.964956 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1215 21:02:42.964999 (+ 43us) server_negotiation.cc:408] Connection header received
1215 21:02:42.965071 (+ 72us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1215 21:02:42.965074 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1215 21:02:42.965107 (+ 33us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1215 21:02:42.965165 (+ 58us) server_negotiation.cc:227] Negotiated authn=SASL
1215 21:02:42.965498 (+ 333us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41986: BlockingRecv error: recv got EOF from 127.0.0.1:41986 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
W20251215 21:02:42.965662 29124 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765832506145157-29124-0/krb5kdc/krb5cc' of type FILE
Dec 15 21:02:42 dist-test-slave-bjsf krb5kdc[2833](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765832562, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251215 21:02:42.966568 29124 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Dec 15 21:02:42 dist-test-slave-bjsf krb5kdc[2833](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765832562, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251215 21:02:42.967307 29124 init.cc:295] Successfully reacquired a new kerberos TGT
Dec 15 21:02:42 dist-test-slave-bjsf krb5kdc[2833](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765832562, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.113.1@KRBTEST.COM
I20251215 21:02:42.974037 3353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:02:42.968036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41998 (local address 127.28.113.1:40603)
1215 21:02:42.968172 (+ 136us) server_negotiation.cc:207] Beginning negotiation
1215 21:02:42.968176 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1215 21:02:42.968240 (+ 64us) server_negotiation.cc:408] Connection header received
1215 21:02:42.968323 (+ 83us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1215 21:02:42.968325 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1215 21:02:42.968360 (+ 35us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1215 21:02:42.968423 (+ 63us) server_negotiation.cc:227] Negotiated authn=SASL
1215 21:02:42.968926 (+ 503us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:02:42.969386 (+ 460us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1215 21:02:42.969897 (+ 511us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:02:42.970019 (+ 122us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:02:42.971670 (+ 1651us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1215 21:02:42.971685 (+ 15us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1215 21:02:42.971687 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1215 21:02:42.971709 (+ 22us) server_negotiation.cc:923] calling sasl_server_start()
1215 21:02:42.973020 (+ 1311us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1215 21:02:42.973300 (+ 280us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1215 21:02:42.973302 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1215 21:02:42.973304 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1215 21:02:42.973338 (+ 34us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1215 21:02:42.973468 (+ 130us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1215 21:02:42.973470 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1215 21:02:42.973471 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1215 21:02:42.973578 (+ 107us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1215 21:02:42.973661 (+ 83us) server_negotiation.cc:1036] Waiting for connection context
1215 21:02:42.973798 (+ 137us) server_negotiation.cc:300] Negotiation successful
1215 21:02:42.973879 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20251215 21:02:42.974920 29124 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu with pid 2945
I20251215 21:02:42.980374 29124 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu with pid 3081
I20251215 21:02:42.985071 29124 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu with pid 3217
I20251215 21:02:42.989497 29124 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu with pid 2849
2025-12-15T21:02:42Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (2889 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765832506145157-29124-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Dec 15 21:02:43 dist-test-slave-bjsf krb5kdc[3366](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Dec 15 21:02:43 dist-test-slave-bjsf krb5kdc[3366](info): set up 2 sockets
Dec 15 21:02:43 dist-test-slave-bjsf krb5kdc[3366](info): commencing operation
krb5kdc: starting...
W20251215 21:02:44.816273 29124 mini_kdc.cc:121] Time spent starting KDC: real 1.807s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765832506145157-29124-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765832506145157-29124-0/krb5kdc/test-user.keytab.
Dec 15 21:02:44 dist-test-slave-bjsf krb5kdc[3366](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765832564, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-12-15T21:02:44Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-12-15T21:02:44Z Disabled control of system clock
WARNING: no policy specified for oryx/127.28.113.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.28.113.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.28.113.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765832506145157-29124-0/krb5kdc/oryx.keytab.
Entry for principal oryx/127.28.113.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765832506145157-29124-0/krb5kdc/oryx.keytab.
WARNING: no policy specified for HTTP/127.28.113.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.28.113.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.28.113.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765832506145157-29124-0/krb5kdc/oryx.keytab.
Entry for principal HTTP/127.28.113.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765832506145157-29124-0/krb5kdc/oryx.keytab.
I20251215 21:02:44.940459 29124 external_mini_cluster.cc:1366] Running /tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu
/tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765832506145157-29124-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765832506145157-29124-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765832506145157-29124-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765832506145157-29124-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.113.62:41229
--webserver_interface=127.28.113.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765832506145157-29124-0/krb5kdc/oryx.keytab
--principal=oryx/127.28.113.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.28.113.20:39895
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.113.62:41229
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765832506145157-29124-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765832506145157-29124-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765832506145157-29124-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251215 21:02:45.040212 3382 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251215 21:02:45.040463 3382 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251215 21:02:45.040521 3382 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251215 21:02:45.043846 3382 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251215 21:02:45.043924 3382 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251215 21:02:45.043946 3382 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251215 21:02:45.043984 3382 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251215 21:02:45.044003 3382 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251215 21:02:45.048323 3382 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.113.20:39895
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765832506145157-29124-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765832506145157-29124-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.113.62:41229
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765832506145157-29124-0/krb5kdc/oryx.keytab
--principal=oryx/127.28.113.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.113.62:41229
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765832506145157-29124-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.113.62
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3382
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765832506145157-29124-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision dc741732bb6a40213e11a13869fad06a49a209a3
build type DEBUG
built by None at 15 Dec 2025 20:43:14 UTC on 24a791456cd2
build id 9530
I20251215 21:02:45.049388 3382 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251215 21:02:45.050206 3382 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251215 21:02:45.055480 3387 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251215 21:02:45.055543 3388 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251215 21:02:45.055480 3390 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251215 21:02:45.055605 3382 server_base.cc:1047] running on GCE node
I20251215 21:02:45.056288 3382 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251215 21:02:45.057153 3382 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251215 21:02:45.058311 3382 hybrid_clock.cc:648] HybridClock initialized: now 1765832565058296 us; error 32 us; skew 500 ppm
Dec 15 21:02:45 dist-test-slave-bjsf krb5kdc[3366](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765832565, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.113.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251215 21:02:45.060760 3382 init.cc:377] Logged in from keytab as oryx/127.28.113.62@KRBTEST.COM (short username oryx)
I20251215 21:02:45.061746 3382 webserver.cc:492] Webserver started at http://127.28.113.62:37249/ using document root <none> and password file <none>
I20251215 21:02:45.062251 3382 fs_manager.cc:362] Metadata directory not provided
I20251215 21:02:45.062302 3382 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251215 21:02:45.062472 3382 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251215 21:02:45.064150 3382 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765832506145157-29124-0/minicluster-data/master-0/data/instance:
uuid: "8c9267b46ebc4bae9a13928cf23fa8d3"
format_stamp: "Formatted at 2025-12-15 21:02:45 on dist-test-slave-bjsf"
server_key: "48fd820e5bcc4c0d9310df03b0351405"
server_key_iv: "e33b4a9ddb98abf95c00392cd847fac5"
server_key_version: "encryptionkey@0"
I20251215 21:02:45.064596 3382 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765832506145157-29124-0/minicluster-data/master-0/wal/instance:
uuid: "8c9267b46ebc4bae9a13928cf23fa8d3"
format_stamp: "Formatted at 2025-12-15 21:02:45 on dist-test-slave-bjsf"
server_key: "48fd820e5bcc4c0d9310df03b0351405"
server_key_iv: "e33b4a9ddb98abf95c00392cd847fac5"
server_key_version: "encryptionkey@0"
I20251215 21:02:45.067687 3382 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20251215 21:02:45.069782 3397 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251215 21:02:45.070753 3382 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20251215 21:02:45.070856 3382 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765832506145157-29124-0/minicluster-data/master-0/data,/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765832506145157-29124-0/minicluster-data/master-0/wal
uuid: "8c9267b46ebc4bae9a13928cf23fa8d3"
format_stamp: "Formatted at 2025-12-15 21:02:45 on dist-test-slave-bjsf"
server_key: "48fd820e5bcc4c0d9310df03b0351405"
server_key_iv: "e33b4a9ddb98abf95c00392cd847fac5"
server_key_version: "encryptionkey@0"
I20251215 21:02:45.070937 3382 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765832506145157-29124-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765832506145157-29124-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765832506145157-29124-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251215 21:02:45.086589 3382 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251215 21:02:45.091558 3382 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251215 21:02:45.091744 3382 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251215 21:02:45.099349 3449 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.28.113.62:41229 every 8 connection(s)
I20251215 21:02:45.099359 3382 rpc_server.cc:307] RPC server started. Bound to: 127.28.113.62:41229
I20251215 21:02:45.100420 3382 server_base.cc:1179] Dumped server information to /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765832506145157-29124-0/minicluster-data/master-0/data/info.pb
I20251215 21:02:45.102975 3450 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 21:02:45.105176 29124 external_mini_cluster.cc:1428] Started /tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu as pid 3382
I20251215 21:02:45.105302 29124 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765832506145157-29124-0/minicluster-data/master-0/wal/instance
I20251215 21:02:45.105549 29124 external_mini_cluster.cc:1468] Setting key 62d7a82471e66627b93af5299a1f3e2f
I20251215 21:02:45.108654 3450 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8c9267b46ebc4bae9a13928cf23fa8d3: Bootstrap starting.
I20251215 21:02:45.110858 3450 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8c9267b46ebc4bae9a13928cf23fa8d3: Neither blocks nor log segments found. Creating new log.
I20251215 21:02:45.111631 3450 log.cc:826] T 00000000000000000000000000000000 P 8c9267b46ebc4bae9a13928cf23fa8d3: Log is configured to *not* fsync() on all Append() calls
Dec 15 21:02:45 dist-test-slave-bjsf krb5kdc[3366](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765832564, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.28.113.62@KRBTEST.COM
I20251215 21:02:45.113380 3450 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8c9267b46ebc4bae9a13928cf23fa8d3: No bootstrap required, opened a new log
I20251215 21:02:45.116403 3450 raft_consensus.cc:359] T 00000000000000000000000000000000 P 8c9267b46ebc4bae9a13928cf23fa8d3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c9267b46ebc4bae9a13928cf23fa8d3" member_type: VOTER last_known_addr { host: "127.28.113.62" port: 41229 } }
I20251215 21:02:45.116613 3450 raft_consensus.cc:385] T 00000000000000000000000000000000 P 8c9267b46ebc4bae9a13928cf23fa8d3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251215 21:02:45.116677 3450 raft_consensus.cc:740] T 00000000000000000000000000000000 P 8c9267b46ebc4bae9a13928cf23fa8d3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c9267b46ebc4bae9a13928cf23fa8d3, State: Initialized, Role: FOLLOWER
I20251215 21:02:45.117126 3450 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8c9267b46ebc4bae9a13928cf23fa8d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c9267b46ebc4bae9a13928cf23fa8d3" member_type: VOTER last_known_addr { host: "127.28.113.62" port: 41229 } }
I20251215 21:02:45.117265 3450 raft_consensus.cc:399] T 00000000000000000000000000000000 P 8c9267b46ebc4bae9a13928cf23fa8d3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251215 21:02:45.117323 3450 raft_consensus.cc:493] T 00000000000000000000000000000000 P 8c9267b46ebc4bae9a13928cf23fa8d3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251215 21:02:45.117406 3450 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 8c9267b46ebc4bae9a13928cf23fa8d3 [term 0 FOLLOWER]: Advancing to term 1
I20251215 21:02:45.117421 3453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:02:45.106692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42416 (local address 127.28.113.62:41229)
1215 21:02:45.107072 (+ 380us) server_negotiation.cc:207] Beginning negotiation
1215 21:02:45.107082 (+ 10us) server_negotiation.cc:400] Waiting for connection header
1215 21:02:45.107113 (+ 31us) server_negotiation.cc:408] Connection header received
1215 21:02:45.107665 (+ 552us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1215 21:02:45.107681 (+ 16us) server_negotiation.cc:462] Received NEGOTIATE request from client
1215 21:02:45.107953 (+ 272us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1215 21:02:45.108252 (+ 299us) server_negotiation.cc:227] Negotiated authn=SASL
1215 21:02:45.109202 (+ 950us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:02:45.109955 (+ 753us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1215 21:02:45.110748 (+ 793us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:02:45.110957 (+ 209us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:02:45.113208 (+ 2251us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1215 21:02:45.113227 (+ 19us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1215 21:02:45.113239 (+ 12us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1215 21:02:45.113275 (+ 36us) server_negotiation.cc:923] calling sasl_server_start()
1215 21:02:45.115088 (+ 1813us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1215 21:02:45.115433 (+ 345us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1215 21:02:45.115439 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1215 21:02:45.115444 (+ 5us) server_negotiation.cc:968] calling sasl_server_step()
1215 21:02:45.115527 (+ 83us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1215 21:02:45.115685 (+ 158us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1215 21:02:45.115688 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1215 21:02:45.115690 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1215 21:02:45.116102 (+ 412us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1215 21:02:45.116288 (+ 186us) server_negotiation.cc:1036] Waiting for connection context
1215 21:02:45.116467 (+ 179us) server_negotiation.cc:300] Negotiation successful
1215 21:02:45.116703 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":249,"thread_start_us":102,"threads_started":1}
I20251215 21:02:45.118218 3450 raft_consensus.cc:515] T 00000000000000000000000000000000 P 8c9267b46ebc4bae9a13928cf23fa8d3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c9267b46ebc4bae9a13928cf23fa8d3" member_type: VOTER last_known_addr { host: "127.28.113.62" port: 41229 } }
I20251215 21:02:45.118527 3450 leader_election.cc:304] T 00000000000000000000000000000000 P 8c9267b46ebc4bae9a13928cf23fa8d3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8c9267b46ebc4bae9a13928cf23fa8d3; no voters:
I20251215 21:02:45.118794 3450 leader_election.cc:290] T 00000000000000000000000000000000 P 8c9267b46ebc4bae9a13928cf23fa8d3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20251215 21:02:45.118884 3455 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 8c9267b46ebc4bae9a13928cf23fa8d3 [term 1 FOLLOWER]: Leader election won for term 1
I20251215 21:02:45.119073 3455 raft_consensus.cc:697] T 00000000000000000000000000000000 P 8c9267b46ebc4bae9a13928cf23fa8d3 [term 1 LEADER]: Becoming Leader. State: Replica: 8c9267b46ebc4bae9a13928cf23fa8d3, State: Running, Role: LEADER
I20251215 21:02:45.119596 3455 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8c9267b46ebc4bae9a13928cf23fa8d3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c9267b46ebc4bae9a13928cf23fa8d3" member_type: VOTER last_known_addr { host: "127.28.113.62" port: 41229 } }
I20251215 21:02:45.119845 3450 sys_catalog.cc:565] T 00000000000000000000000000000000 P 8c9267b46ebc4bae9a13928cf23fa8d3 [sys.catalog]: configured and running, proceeding with master startup.
I20251215 21:02:45.121469 3457 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c9267b46ebc4bae9a13928cf23fa8d3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8c9267b46ebc4bae9a13928cf23fa8d3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c9267b46ebc4bae9a13928cf23fa8d3" member_type: VOTER last_known_addr { host: "127.28.113.62" port: 41229 } } }
I20251215 21:02:45.121603 3457 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c9267b46ebc4bae9a13928cf23fa8d3 [sys.catalog]: This master's current role is: LEADER
I20251215 21:02:45.121922 3469 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251215 21:02:45.121482 3456 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c9267b46ebc4bae9a13928cf23fa8d3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8c9267b46ebc4bae9a13928cf23fa8d3. Latest consensus state: current_term: 1 leader_uuid: "8c9267b46ebc4bae9a13928cf23fa8d3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c9267b46ebc4bae9a13928cf23fa8d3" member_type: VOTER last_known_addr { host: "127.28.113.62" port: 41229 } } }
I20251215 21:02:45.122534 3456 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c9267b46ebc4bae9a13928cf23fa8d3 [sys.catalog]: This master's current role is: LEADER
I20251215 21:02:45.125242 3469 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251215 21:02:45.129742 3469 catalog_manager.cc:1357] Generated new cluster ID: 587ee523bb434fd8a7a37f28e6cc4cb1
I20251215 21:02:45.129807 3469 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251215 21:02:45.142966 3469 catalog_manager.cc:1380] Generated new certificate authority record
I20251215 21:02:45.144176 3469 catalog_manager.cc:1514] Loading token signing keys...
I20251215 21:02:45.156701 3469 catalog_manager.cc:6027] T 00000000000000000000000000000000 P 8c9267b46ebc4bae9a13928cf23fa8d3: Generated new TSK 0
I20251215 21:02:45.157584 3469 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.28.113.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.28.113.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.28.113.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765832506145157-29124-0/krb5kdc/oryx.keytab.
Entry for principal oryx/127.28.113.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765832506145157-29124-0/krb5kdc/oryx.keytab.
WARNING: no policy specified for HTTP/127.28.113.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.28.113.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.28.113.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765832506145157-29124-0/krb5kdc/oryx.keytab.
Entry for principal HTTP/127.28.113.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765832506145157-29124-0/krb5kdc/oryx.keytab.
I20251215 21:02:45.204425 29124 external_mini_cluster.cc:1366] Running /tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu
/tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765832506145157-29124-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765832506145157-29124-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765832506145157-29124-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765832506145157-29124-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.113.1:0
--local_ip_for_outbound_sockets=127.28.113.1
--webserver_interface=127.28.113.1
--webserver_port=0
--tserver_master_addrs=127.28.113.62:41229
--keytab_file=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765832506145157-29124-0/krb5kdc/oryx.keytab
--principal=oryx/127.28.113.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.28.113.20:39895
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765832506145157-29124-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765832506145157-29124-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765832506145157-29124-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251215 21:02:45.310909 3478 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251215 21:02:45.311136 3478 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251215 21:02:45.311190 3478 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251215 21:02:45.314395 3478 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251215 21:02:45.314452 3478 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251215 21:02:45.314527 3478 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.113.1
I20251215 21:02:45.318544 3478 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.113.20:39895
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765832506145157-29124-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765832506145157-29124-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765832506145157-29124-0/krb5kdc/oryx.keytab
--principal=oryx/127.28.113.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.113.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765832506145157-29124-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.113.1
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.28.113.62:41229
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3478
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.113.1
--log_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765832506145157-29124-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision dc741732bb6a40213e11a13869fad06a49a209a3
build type DEBUG
built by None at 15 Dec 2025 20:43:14 UTC on 24a791456cd2
build id 9530
I20251215 21:02:45.319473 3478 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251215 21:02:45.320271 3478 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251215 21:02:45.326519 3486 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251215 21:02:45.326515 3484 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251215 21:02:45.326570 3483 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251215 21:02:45.326912 3478 server_base.cc:1047] running on GCE node
I20251215 21:02:45.327244 3478 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251215 21:02:45.327730 3478 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251215 21:02:45.328873 3478 hybrid_clock.cc:648] HybridClock initialized: now 1765832565328853 us; error 38 us; skew 500 ppm
Dec 15 21:02:45 dist-test-slave-bjsf krb5kdc[3366](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765832565, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.113.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251215 21:02:45.331229 3478 init.cc:377] Logged in from keytab as oryx/127.28.113.1@KRBTEST.COM (short username oryx)
I20251215 21:02:45.332240 3478 webserver.cc:492] Webserver started at http://127.28.113.1:37473/ using document root <none> and password file <none>
I20251215 21:02:45.332724 3478 fs_manager.cc:362] Metadata directory not provided
I20251215 21:02:45.332767 3478 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251215 21:02:45.332921 3478 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251215 21:02:45.334501 3478 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765832506145157-29124-0/minicluster-data/ts-0/data/instance:
uuid: "6956a9660a3b4d42867878501794f2d5"
format_stamp: "Formatted at 2025-12-15 21:02:45 on dist-test-slave-bjsf"
server_key: "ca85dbf5eba3ad512963247bd0af2e0b"
server_key_iv: "f11137c2b59bf2ba6e9b515bc4fbb292"
server_key_version: "encryptionkey@0"
I20251215 21:02:45.334910 3478 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765832506145157-29124-0/minicluster-data/ts-0/wal/instance:
uuid: "6956a9660a3b4d42867878501794f2d5"
format_stamp: "Formatted at 2025-12-15 21:02:45 on dist-test-slave-bjsf"
server_key: "ca85dbf5eba3ad512963247bd0af2e0b"
server_key_iv: "f11137c2b59bf2ba6e9b515bc4fbb292"
server_key_version: "encryptionkey@0"
I20251215 21:02:45.338053 3478 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20251215 21:02:45.340111 3493 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251215 21:02:45.341064 3478 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20251215 21:02:45.341158 3478 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765832506145157-29124-0/minicluster-data/ts-0/data,/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765832506145157-29124-0/minicluster-data/ts-0/wal
uuid: "6956a9660a3b4d42867878501794f2d5"
format_stamp: "Formatted at 2025-12-15 21:02:45 on dist-test-slave-bjsf"
server_key: "ca85dbf5eba3ad512963247bd0af2e0b"
server_key_iv: "f11137c2b59bf2ba6e9b515bc4fbb292"
server_key_version: "encryptionkey@0"
I20251215 21:02:45.341250 3478 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765832506145157-29124-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765832506145157-29124-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765832506145157-29124-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251215 21:02:45.378618 3478 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251215 21:02:45.381459 3478 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251215 21:02:45.381628 3478 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251215 21:02:45.382171 3478 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251215 21:02:45.383024 3478 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251215 21:02:45.383069 3478 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251215 21:02:45.383114 3478 ts_tablet_manager.cc:616] Registered 0 tablets
I20251215 21:02:45.383142 3478 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251215 21:02:45.391672 3478 rpc_server.cc:307] RPC server started. Bound to: 127.28.113.1:35255
I20251215 21:02:45.391708 3606 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.28.113.1:35255 every 8 connection(s)
I20251215 21:02:45.392572 3478 server_base.cc:1179] Dumped server information to /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765832506145157-29124-0/minicluster-data/ts-0/data/info.pb
I20251215 21:02:45.400041 29124 external_mini_cluster.cc:1428] Started /tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu as pid 3478
I20251215 21:02:45.400167 29124 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765832506145157-29124-0/minicluster-data/ts-0/wal/instance
I20251215 21:02:45.400379 29124 external_mini_cluster.cc:1468] Setting key e0aff1dfc189877b03490e51fa850421
Dec 15 21:02:45 dist-test-slave-bjsf krb5kdc[3366](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765832565, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.113.1@KRBTEST.COM for oryx/127.28.113.62@KRBTEST.COM
I20251215 21:02:45.404729 3453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:02:45.394200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.113.1:55993 (local address 127.28.113.62:41229)
1215 21:02:45.394336 (+ 136us) server_negotiation.cc:207] Beginning negotiation
1215 21:02:45.394339 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1215 21:02:45.394970 (+ 631us) server_negotiation.cc:408] Connection header received
1215 21:02:45.395803 (+ 833us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1215 21:02:45.395807 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1215 21:02:45.395865 (+ 58us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1215 21:02:45.395939 (+ 74us) server_negotiation.cc:227] Negotiated authn=SASL
1215 21:02:45.397602 (+ 1663us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:02:45.398120 (+ 518us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1215 21:02:45.398725 (+ 605us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:02:45.398899 (+ 174us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:02:45.401112 (+ 2213us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1215 21:02:45.401125 (+ 13us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1215 21:02:45.401128 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1215 21:02:45.401152 (+ 24us) server_negotiation.cc:923] calling sasl_server_start()
1215 21:02:45.402667 (+ 1515us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1215 21:02:45.403246 (+ 579us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1215 21:02:45.403250 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1215 21:02:45.403253 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
1215 21:02:45.403322 (+ 69us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1215 21:02:45.403649 (+ 327us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1215 21:02:45.403653 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1215 21:02:45.403655 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1215 21:02:45.403831 (+ 176us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1215 21:02:45.403917 (+ 86us) server_negotiation.cc:1036] Waiting for connection context
1215 21:02:45.404439 (+ 522us) server_negotiation.cc:300] Negotiation successful
1215 21:02:45.404571 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20251215 21:02:45.405350 3609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:02:45.394426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.113.62:41229 (local address 127.28.113.1:55993)
1215 21:02:45.394833 (+ 407us) negotiation.cc:107] Waiting for socket to connect
1215 21:02:45.394869 (+ 36us) client_negotiation.cc:174] Beginning negotiation
1215 21:02:45.395604 (+ 735us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1215 21:02:45.396131 (+ 527us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1215 21:02:45.396139 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
1215 21:02:45.396518 (+ 379us) client_negotiation.cc:189] Negotiated authn=SASL
1215 21:02:45.397446 (+ 928us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:02:45.397461 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:02:45.398236 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1215 21:02:45.398239 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1215 21:02:45.398618 (+ 379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:02:45.398625 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:02:45.398846 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:02:45.399403 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1215 21:02:45.399427 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
1215 21:02:45.400982 (+ 1555us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1215 21:02:45.402824 (+ 1842us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1215 21:02:45.402830 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1215 21:02:45.402842 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
1215 21:02:45.403131 (+ 289us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1215 21:02:45.403424 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1215 21:02:45.403427 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1215 21:02:45.403430 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
1215 21:02:45.403559 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1215 21:02:45.403929 (+ 370us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1215 21:02:45.403936 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1215 21:02:45.404208 (+ 272us) client_negotiation.cc:769] Sending connection context
1215 21:02:45.404396 (+ 188us) client_negotiation.cc:240] Negotiation successful
1215 21:02:45.404661 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":99,"threads_started":1}
I20251215 21:02:45.406454 3607 heartbeater.cc:344] Connected to a master server at 127.28.113.62:41229
I20251215 21:02:45.406680 3607 heartbeater.cc:461] Registering TS with master...
I20251215 21:02:45.407169 3607 heartbeater.cc:507] Master 127.28.113.62:41229 requested a full tablet report, sending...
I20251215 21:02:45.408704 3414 ts_manager.cc:194] Registered new tserver with Master: 6956a9660a3b4d42867878501794f2d5 (127.28.113.1:35255)
WARNING: no policy specified for oryx/127.28.113.2@KRBTEST.COM; defaulting to no policy
I20251215 21:02:45.410043 3414 master_service.cc:502] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.28.113.1@KRBTEST.COM'} at 127.28.113.1:55993
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.28.113.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.28.113.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765832506145157-29124-0/krb5kdc/oryx.keytab.
Entry for principal oryx/127.28.113.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765832506145157-29124-0/krb5kdc/oryx.keytab.
WARNING: no policy specified for HTTP/127.28.113.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.28.113.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.28.113.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765832506145157-29124-0/krb5kdc/oryx.keytab.
Entry for principal HTTP/127.28.113.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765832506145157-29124-0/krb5kdc/oryx.keytab.
I20251215 21:02:45.444623 29124 external_mini_cluster.cc:1366] Running /tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu
/tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765832506145157-29124-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765832506145157-29124-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765832506145157-29124-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765832506145157-29124-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.113.2:0
--local_ip_for_outbound_sockets=127.28.113.2
--webserver_interface=127.28.113.2
--webserver_port=0
--tserver_master_addrs=127.28.113.62:41229
--keytab_file=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765832506145157-29124-0/krb5kdc/oryx.keytab
--principal=oryx/127.28.113.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.28.113.20:39895
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765832506145157-29124-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765832506145157-29124-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765832506145157-29124-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251215 21:02:45.548434 3614 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251215 21:02:45.548692 3614 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251215 21:02:45.548794 3614 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251215 21:02:45.552156 3614 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251215 21:02:45.552227 3614 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251215 21:02:45.552331 3614 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.113.2
I20251215 21:02:45.556609 3614 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.113.20:39895
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765832506145157-29124-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765832506145157-29124-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765832506145157-29124-0/krb5kdc/oryx.keytab
--principal=oryx/127.28.113.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.113.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765832506145157-29124-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.113.2
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.28.113.62:41229
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3614
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.113.2
--log_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765832506145157-29124-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision dc741732bb6a40213e11a13869fad06a49a209a3
build type DEBUG
built by None at 15 Dec 2025 20:43:14 UTC on 24a791456cd2
build id 9530
I20251215 21:02:45.557636 3614 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251215 21:02:45.558463 3614 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251215 21:02:45.564675 3619 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251215 21:02:45.564739 3622 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251215 21:02:45.564692 3620 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251215 21:02:45.564921 3614 server_base.cc:1047] running on GCE node
I20251215 21:02:45.565359 3614 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251215 21:02:45.565867 3614 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251215 21:02:45.567009 3614 hybrid_clock.cc:648] HybridClock initialized: now 1765832565566993 us; error 31 us; skew 500 ppm
Dec 15 21:02:45 dist-test-slave-bjsf krb5kdc[3366](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765832565, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.113.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251215 21:02:45.569599 3614 init.cc:377] Logged in from keytab as oryx/127.28.113.2@KRBTEST.COM (short username oryx)
I20251215 21:02:45.570556 3614 webserver.cc:492] Webserver started at http://127.28.113.2:41927/ using document root <none> and password file <none>
I20251215 21:02:45.571072 3614 fs_manager.cc:362] Metadata directory not provided
I20251215 21:02:45.571120 3614 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251215 21:02:45.571285 3614 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251215 21:02:45.572964 3614 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765832506145157-29124-0/minicluster-data/ts-1/data/instance:
uuid: "f17cfa3c3e5041c092728f920f98dec6"
format_stamp: "Formatted at 2025-12-15 21:02:45 on dist-test-slave-bjsf"
server_key: "1ce2ba8517fd183ff29eb281e5234123"
server_key_iv: "1557c0dc72887429de987e9aea01bad3"
server_key_version: "encryptionkey@0"
I20251215 21:02:45.573426 3614 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765832506145157-29124-0/minicluster-data/ts-1/wal/instance:
uuid: "f17cfa3c3e5041c092728f920f98dec6"
format_stamp: "Formatted at 2025-12-15 21:02:45 on dist-test-slave-bjsf"
server_key: "1ce2ba8517fd183ff29eb281e5234123"
server_key_iv: "1557c0dc72887429de987e9aea01bad3"
server_key_version: "encryptionkey@0"
I20251215 21:02:45.576735 3614 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20251215 21:02:45.578842 3629 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251215 21:02:45.579807 3614 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20251215 21:02:45.579911 3614 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765832506145157-29124-0/minicluster-data/ts-1/data,/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765832506145157-29124-0/minicluster-data/ts-1/wal
uuid: "f17cfa3c3e5041c092728f920f98dec6"
format_stamp: "Formatted at 2025-12-15 21:02:45 on dist-test-slave-bjsf"
server_key: "1ce2ba8517fd183ff29eb281e5234123"
server_key_iv: "1557c0dc72887429de987e9aea01bad3"
server_key_version: "encryptionkey@0"
I20251215 21:02:45.580020 3614 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765832506145157-29124-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765832506145157-29124-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765832506145157-29124-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251215 21:02:45.603374 3614 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251215 21:02:45.606132 3614 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251215 21:02:45.606303 3614 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251215 21:02:45.606820 3614 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251215 21:02:45.607661 3614 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251215 21:02:45.607708 3614 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251215 21:02:45.607759 3614 ts_tablet_manager.cc:616] Registered 0 tablets
I20251215 21:02:45.607785 3614 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251215 21:02:45.616303 3614 rpc_server.cc:307] RPC server started. Bound to: 127.28.113.2:32787
I20251215 21:02:45.616338 3742 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.28.113.2:32787 every 8 connection(s)
I20251215 21:02:45.617233 3614 server_base.cc:1179] Dumped server information to /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765832506145157-29124-0/minicluster-data/ts-1/data/info.pb
I20251215 21:02:45.619298 29124 external_mini_cluster.cc:1428] Started /tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu as pid 3614
I20251215 21:02:45.619391 29124 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765832506145157-29124-0/minicluster-data/ts-1/wal/instance
I20251215 21:02:45.619637 29124 external_mini_cluster.cc:1468] Setting key 36c890af3dd73215d8b498abcf096b09
Dec 15 21:02:45 dist-test-slave-bjsf krb5kdc[3366](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765832565, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.113.2@KRBTEST.COM for oryx/127.28.113.62@KRBTEST.COM
WARNING: no policy specified for oryx/127.28.113.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.28.113.3@KRBTEST.COM" created.
I20251215 21:02:45.630182 3453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:02:45.618900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.113.2:39651 (local address 127.28.113.62:41229)
1215 21:02:45.619022 (+ 122us) server_negotiation.cc:207] Beginning negotiation
1215 21:02:45.619026 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1215 21:02:45.619739 (+ 713us) server_negotiation.cc:408] Connection header received
1215 21:02:45.620728 (+ 989us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1215 21:02:45.620750 (+ 22us) server_negotiation.cc:462] Received NEGOTIATE request from client
1215 21:02:45.620798 (+ 48us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1215 21:02:45.620881 (+ 83us) server_negotiation.cc:227] Negotiated authn=SASL
1215 21:02:45.622701 (+ 1820us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:02:45.623199 (+ 498us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1215 21:02:45.624005 (+ 806us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:02:45.624130 (+ 125us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:02:45.626720 (+ 2590us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1215 21:02:45.626755 (+ 35us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1215 21:02:45.626758 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1215 21:02:45.626784 (+ 26us) server_negotiation.cc:923] calling sasl_server_start()
1215 21:02:45.628330 (+ 1546us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1215 21:02:45.628792 (+ 462us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1215 21:02:45.628796 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1215 21:02:45.628798 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1215 21:02:45.628852 (+ 54us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1215 21:02:45.629172 (+ 320us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1215 21:02:45.629175 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1215 21:02:45.629177 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1215 21:02:45.629333 (+ 156us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1215 21:02:45.629420 (+ 87us) server_negotiation.cc:1036] Waiting for connection context
1215 21:02:45.629902 (+ 482us) server_negotiation.cc:300] Negotiation successful
1215 21:02:45.630031 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20251215 21:02:45.630970 3745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:02:45.619141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.113.62:41229 (local address 127.28.113.2:39651)
1215 21:02:45.619596 (+ 455us) negotiation.cc:107] Waiting for socket to connect
1215 21:02:45.619642 (+ 46us) client_negotiation.cc:174] Beginning negotiation
1215 21:02:45.620521 (+ 879us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1215 21:02:45.621055 (+ 534us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1215 21:02:45.621073 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
1215 21:02:45.621543 (+ 470us) client_negotiation.cc:189] Negotiated authn=SASL
1215 21:02:45.622531 (+ 988us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:02:45.622549 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:02:45.623316 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1215 21:02:45.623320 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1215 21:02:45.623875 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:02:45.623884 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:02:45.624138 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:02:45.624782 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1215 21:02:45.624812 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
1215 21:02:45.626566 (+ 1754us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1215 21:02:45.628439 (+ 1873us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1215 21:02:45.628443 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1215 21:02:45.628453 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
1215 21:02:45.628665 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1215 21:02:45.628926 (+ 261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1215 21:02:45.628928 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1215 21:02:45.628930 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1215 21:02:45.629019 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1215 21:02:45.629418 (+ 399us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1215 21:02:45.629423 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1215 21:02:45.629698 (+ 275us) client_negotiation.cc:769] Sending connection context
1215 21:02:45.629896 (+ 198us) client_negotiation.cc:240] Negotiation successful
1215 21:02:45.630170 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":102,"threads_started":1}
I20251215 21:02:45.632184 3743 heartbeater.cc:344] Connected to a master server at 127.28.113.62:41229
I20251215 21:02:45.632421 3743 heartbeater.cc:461] Registering TS with master...
I20251215 21:02:45.632937 3743 heartbeater.cc:507] Master 127.28.113.62:41229 requested a full tablet report, sending...
I20251215 21:02:45.634021 3414 ts_manager.cc:194] Registered new tserver with Master: f17cfa3c3e5041c092728f920f98dec6 (127.28.113.2:32787)
I20251215 21:02:45.634652 3414 master_service.cc:502] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.28.113.2@KRBTEST.COM'} at 127.28.113.2:39651
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.28.113.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765832506145157-29124-0/krb5kdc/oryx.keytab.
Entry for principal oryx/127.28.113.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765832506145157-29124-0/krb5kdc/oryx.keytab.
WARNING: no policy specified for HTTP/127.28.113.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.28.113.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.28.113.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765832506145157-29124-0/krb5kdc/oryx.keytab.
Entry for principal HTTP/127.28.113.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765832506145157-29124-0/krb5kdc/oryx.keytab.
I20251215 21:02:45.664186 29124 external_mini_cluster.cc:1366] Running /tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu
/tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765832506145157-29124-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765832506145157-29124-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765832506145157-29124-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765832506145157-29124-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.113.3:0
--local_ip_for_outbound_sockets=127.28.113.3
--webserver_interface=127.28.113.3
--webserver_port=0
--tserver_master_addrs=127.28.113.62:41229
--keytab_file=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765832506145157-29124-0/krb5kdc/oryx.keytab
--principal=oryx/127.28.113.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.28.113.20:39895
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765832506145157-29124-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765832506145157-29124-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765832506145157-29124-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251215 21:02:45.764626 3750 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251215 21:02:45.764828 3750 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251215 21:02:45.764886 3750 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251215 21:02:45.767997 3750 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251215 21:02:45.768047 3750 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251215 21:02:45.768131 3750 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.113.3
I20251215 21:02:45.772087 3750 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.113.20:39895
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765832506145157-29124-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765832506145157-29124-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765832506145157-29124-0/krb5kdc/oryx.keytab
--principal=oryx/127.28.113.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.113.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765832506145157-29124-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.113.3
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.28.113.62:41229
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3750
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.113.3
--log_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765832506145157-29124-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision dc741732bb6a40213e11a13869fad06a49a209a3
build type DEBUG
built by None at 15 Dec 2025 20:43:14 UTC on 24a791456cd2
build id 9530
I20251215 21:02:45.773006 3750 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251215 21:02:45.773779 3750 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251215 21:02:45.779893 3758 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251215 21:02:45.779893 3756 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251215 21:02:45.780087 3750 server_base.cc:1047] running on GCE node
W20251215 21:02:45.780158 3755 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251215 21:02:45.780553 3750 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251215 21:02:45.781024 3750 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251215 21:02:45.782156 3750 hybrid_clock.cc:648] HybridClock initialized: now 1765832565782141 us; error 33 us; skew 500 ppm
Dec 15 21:02:45 dist-test-slave-bjsf krb5kdc[3366](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765832565, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.113.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251215 21:02:45.784654 3750 init.cc:377] Logged in from keytab as oryx/127.28.113.3@KRBTEST.COM (short username oryx)
I20251215 21:02:45.785606 3750 webserver.cc:492] Webserver started at http://127.28.113.3:42413/ using document root <none> and password file <none>
I20251215 21:02:45.786109 3750 fs_manager.cc:362] Metadata directory not provided
I20251215 21:02:45.786152 3750 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251215 21:02:45.786320 3750 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251215 21:02:45.787873 3750 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765832506145157-29124-0/minicluster-data/ts-2/data/instance:
uuid: "0a2363aab21d43f68f26fb68238a29f3"
format_stamp: "Formatted at 2025-12-15 21:02:45 on dist-test-slave-bjsf"
server_key: "93cec74eff54b765f102832ee805e7db"
server_key_iv: "b5d3a23f47d2d65786e8bb9f48a52513"
server_key_version: "encryptionkey@0"
I20251215 21:02:45.788363 3750 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765832506145157-29124-0/minicluster-data/ts-2/wal/instance:
uuid: "0a2363aab21d43f68f26fb68238a29f3"
format_stamp: "Formatted at 2025-12-15 21:02:45 on dist-test-slave-bjsf"
server_key: "93cec74eff54b765f102832ee805e7db"
server_key_iv: "b5d3a23f47d2d65786e8bb9f48a52513"
server_key_version: "encryptionkey@0"
I20251215 21:02:45.791419 3750 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20251215 21:02:45.793390 3765 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251215 21:02:45.794315 3750 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20251215 21:02:45.794404 3750 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765832506145157-29124-0/minicluster-data/ts-2/data,/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765832506145157-29124-0/minicluster-data/ts-2/wal
uuid: "0a2363aab21d43f68f26fb68238a29f3"
format_stamp: "Formatted at 2025-12-15 21:02:45 on dist-test-slave-bjsf"
server_key: "93cec74eff54b765f102832ee805e7db"
server_key_iv: "b5d3a23f47d2d65786e8bb9f48a52513"
server_key_version: "encryptionkey@0"
I20251215 21:02:45.794485 3750 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765832506145157-29124-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765832506145157-29124-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765832506145157-29124-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251215 21:02:45.805606 3750 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251215 21:02:45.808112 3750 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251215 21:02:45.808267 3750 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251215 21:02:45.808759 3750 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251215 21:02:45.809587 3750 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251215 21:02:45.809633 3750 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251215 21:02:45.809672 3750 ts_tablet_manager.cc:616] Registered 0 tablets
I20251215 21:02:45.809700 3750 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251215 21:02:45.818024 3750 rpc_server.cc:307] RPC server started. Bound to: 127.28.113.3:35071
I20251215 21:02:45.818061 3878 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.28.113.3:35071 every 8 connection(s)
I20251215 21:02:45.818903 3750 server_base.cc:1179] Dumped server information to /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765832506145157-29124-0/minicluster-data/ts-2/data/info.pb
Dec 15 21:02:45 dist-test-slave-bjsf krb5kdc[3366](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765832565, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.113.3@KRBTEST.COM for oryx/127.28.113.62@KRBTEST.COM
I20251215 21:02:45.829191 29124 external_mini_cluster.cc:1428] Started /tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu as pid 3750
I20251215 21:02:45.829319 29124 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765832506145157-29124-0/minicluster-data/ts-2/wal/instance
I20251215 21:02:45.829546 29124 external_mini_cluster.cc:1468] Setting key b9e4ed64d57e9d4fdb28a904c22fcdf1
I20251215 21:02:45.830031 3453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:02:45.820515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.113.3:55007 (local address 127.28.113.62:41229)
1215 21:02:45.820654 (+ 139us) server_negotiation.cc:207] Beginning negotiation
1215 21:02:45.820659 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1215 21:02:45.821308 (+ 649us) server_negotiation.cc:408] Connection header received
1215 21:02:45.822117 (+ 809us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1215 21:02:45.822120 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1215 21:02:45.822170 (+ 50us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1215 21:02:45.822242 (+ 72us) server_negotiation.cc:227] Negotiated authn=SASL
1215 21:02:45.823811 (+ 1569us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:02:45.824338 (+ 527us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1215 21:02:45.824902 (+ 564us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:02:45.825058 (+ 156us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:02:45.827132 (+ 2074us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1215 21:02:45.827151 (+ 19us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1215 21:02:45.827153 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1215 21:02:45.827179 (+ 26us) server_negotiation.cc:923] calling sasl_server_start()
1215 21:02:45.828325 (+ 1146us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1215 21:02:45.828737 (+ 412us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1215 21:02:45.828740 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1215 21:02:45.828741 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1215 21:02:45.828775 (+ 34us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1215 21:02:45.829010 (+ 235us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1215 21:02:45.829013 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1215 21:02:45.829015 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1215 21:02:45.829198 (+ 183us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1215 21:02:45.829294 (+ 96us) server_negotiation.cc:1036] Waiting for connection context
1215 21:02:45.829793 (+ 499us) server_negotiation.cc:300] Negotiation successful
1215 21:02:45.829903 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20251215 21:02:45.830742 3881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:02:45.820771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.113.62:41229 (local address 127.28.113.3:55007)
1215 21:02:45.821169 (+ 398us) negotiation.cc:107] Waiting for socket to connect
1215 21:02:45.821204 (+ 35us) client_negotiation.cc:174] Beginning negotiation
1215 21:02:45.821940 (+ 736us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1215 21:02:45.822376 (+ 436us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1215 21:02:45.822383 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
1215 21:02:45.822777 (+ 394us) client_negotiation.cc:189] Negotiated authn=SASL
1215 21:02:45.823684 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:02:45.823698 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:02:45.824421 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1215 21:02:45.824424 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1215 21:02:45.824816 (+ 392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:02:45.824823 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:02:45.825011 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:02:45.825566 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1215 21:02:45.825589 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
1215 21:02:45.826990 (+ 1401us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1215 21:02:45.828432 (+ 1442us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1215 21:02:45.828437 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1215 21:02:45.828446 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
1215 21:02:45.828661 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1215 21:02:45.828833 (+ 172us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1215 21:02:45.828836 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1215 21:02:45.828837 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1215 21:02:45.828943 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1215 21:02:45.829288 (+ 345us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1215 21:02:45.829295 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1215 21:02:45.829549 (+ 254us) client_negotiation.cc:769] Sending connection context
1215 21:02:45.829735 (+ 186us) client_negotiation.cc:240] Negotiation successful
1215 21:02:45.829983 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":97,"threads_started":1}
I20251215 21:02:45.831794 3879 heartbeater.cc:344] Connected to a master server at 127.28.113.62:41229
I20251215 21:02:45.832062 3879 heartbeater.cc:461] Registering TS with master...
I20251215 21:02:45.832556 3879 heartbeater.cc:507] Master 127.28.113.62:41229 requested a full tablet report, sending...
I20251215 21:02:45.833518 3414 ts_manager.cc:194] Registered new tserver with Master: 0a2363aab21d43f68f26fb68238a29f3 (127.28.113.3:35071)
I20251215 21:02:45.834049 3414 master_service.cc:502] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.28.113.3@KRBTEST.COM'} at 127.28.113.3:55007
I20251215 21:02:45.843055 29124 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.28.113.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.28.113.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.28.113.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765832506145157-29124-0/krb5kdc/oryx.keytab.
Entry for principal oryx/127.28.113.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765832506145157-29124-0/krb5kdc/oryx.keytab.
WARNING: no policy specified for HTTP/127.28.113.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.28.113.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.28.113.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765832506145157-29124-0/krb5kdc/oryx.keytab.
Entry for principal HTTP/127.28.113.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765832506145157-29124-0/krb5kdc/oryx.keytab.
I20251215 21:02:45.882458 29124 external_mini_cluster.cc:1366] Running /tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu
/tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765832506145157-29124-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765832506145157-29124-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765832506145157-29124-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765832506145157-29124-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.113.61:39207
--webserver_interface=127.28.113.61
--webserver_port=0
--keytab_file=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765832506145157-29124-0/krb5kdc/oryx.keytab
--principal=oryx/127.28.113.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.28.113.20:39895
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.113.62:41229,127.28.113.61:39207
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765832506145157-29124-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765832506145157-29124-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765832506145157-29124-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251215 21:02:45.983877 3886 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251215 21:02:45.984135 3886 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251215 21:02:45.984215 3886 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251215 21:02:45.987414 3886 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251215 21:02:45.987478 3886 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251215 21:02:45.987502 3886 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251215 21:02:45.987534 3886 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251215 21:02:45.987563 3886 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251215 21:02:45.991761 3886 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.113.20:39895
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765832506145157-29124-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765832506145157-29124-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.28.113.62:41229,127.28.113.61:39207
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765832506145157-29124-0/krb5kdc/oryx.keytab
--principal=oryx/127.28.113.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.113.61:39207
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765832506145157-29124-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.28.113.61
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3886
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765832506145157-29124-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision dc741732bb6a40213e11a13869fad06a49a209a3
build type DEBUG
built by None at 15 Dec 2025 20:43:14 UTC on 24a791456cd2
build id 9530
W20251215 21:02:45.991972 3886 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.28.113.62:41229,127.28.113.61:39207'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20251215 21:02:45.992825 3886 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251215 21:02:45.993620 3886 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251215 21:02:45.998931 3891 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251215 21:02:45.998934 3892 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251215 21:02:45.999058 3886 server_base.cc:1047] running on GCE node
W20251215 21:02:45.998939 3894 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251215 21:02:45.999562 3886 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251215 21:02:46.000428 3886 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251215 21:02:46.001569 3886 hybrid_clock.cc:648] HybridClock initialized: now 1765832566001553 us; error 33 us; skew 500 ppm
Dec 15 21:02:45 dist-test-slave-bjsf krb5kdc[3366](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765832565, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.113.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251215 21:02:46.004114 3886 init.cc:377] Logged in from keytab as oryx/127.28.113.61@KRBTEST.COM (short username oryx)
I20251215 21:02:46.005103 3886 webserver.cc:492] Webserver started at http://127.28.113.61:36661/ using document root <none> and password file <none>
I20251215 21:02:46.005604 3886 fs_manager.cc:362] Metadata directory not provided
I20251215 21:02:46.005650 3886 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251215 21:02:46.005836 3886 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251215 21:02:46.007474 3886 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765832506145157-29124-0/minicluster-data/master-1/data/instance:
uuid: "de9d22b45210451f997093f481e4dfc6"
format_stamp: "Formatted at 2025-12-15 21:02:46 on dist-test-slave-bjsf"
server_key: "802a0ba8fcedee52556d977987b287f7"
server_key_iv: "e374525312c5cc54d2234ab16636ef84"
server_key_version: "encryptionkey@0"
I20251215 21:02:46.007925 3886 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765832506145157-29124-0/minicluster-data/master-1/wal/instance:
uuid: "de9d22b45210451f997093f481e4dfc6"
format_stamp: "Formatted at 2025-12-15 21:02:46 on dist-test-slave-bjsf"
server_key: "802a0ba8fcedee52556d977987b287f7"
server_key_iv: "e374525312c5cc54d2234ab16636ef84"
server_key_version: "encryptionkey@0"
I20251215 21:02:46.011139 3886 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20251215 21:02:46.013209 3901 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251215 21:02:46.014218 3886 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20251215 21:02:46.014322 3886 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765832506145157-29124-0/minicluster-data/master-1/data,/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765832506145157-29124-0/minicluster-data/master-1/wal
uuid: "de9d22b45210451f997093f481e4dfc6"
format_stamp: "Formatted at 2025-12-15 21:02:46 on dist-test-slave-bjsf"
server_key: "802a0ba8fcedee52556d977987b287f7"
server_key_iv: "e374525312c5cc54d2234ab16636ef84"
server_key_version: "encryptionkey@0"
I20251215 21:02:46.014413 3886 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765832506145157-29124-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765832506145157-29124-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765832506145157-29124-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251215 21:02:46.028539 3886 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251215 21:02:46.031549 3886 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251215 21:02:46.031718 3886 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251215 21:02:46.038754 3953 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.28.113.61:39207 every 8 connection(s)
I20251215 21:02:46.038756 3886 rpc_server.cc:307] RPC server started. Bound to: 127.28.113.61:39207
I20251215 21:02:46.039806 3886 server_base.cc:1179] Dumped server information to /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765832506145157-29124-0/minicluster-data/master-1/data/info.pb
I20251215 21:02:46.042680 3954 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 21:02:46.046614 3954 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.113.62" port: 41229 } has no permanent_uuid. Determining permanent_uuid...
I20251215 21:02:46.046911 29124 external_mini_cluster.cc:1428] Started /tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu as pid 3886
Dec 15 21:02:46 dist-test-slave-bjsf krb5kdc[3366](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765832565, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.113.61@KRBTEST.COM for oryx/127.28.113.62@KRBTEST.COM
Dec 15 21:02:46 dist-test-slave-bjsf krb5kdc[3366](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765832564, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.28.113.61@KRBTEST.COM
I20251215 21:02:46.058003 3453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:02:46.048879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42432 (local address 127.28.113.62:41229)
1215 21:02:46.049036 (+ 157us) server_negotiation.cc:207] Beginning negotiation
1215 21:02:46.049041 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1215 21:02:46.049207 (+ 166us) server_negotiation.cc:408] Connection header received
1215 21:02:46.049529 (+ 322us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1215 21:02:46.049533 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1215 21:02:46.049589 (+ 56us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1215 21:02:46.049672 (+ 83us) server_negotiation.cc:227] Negotiated authn=SASL
1215 21:02:46.050751 (+ 1079us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:02:46.051507 (+ 756us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1215 21:02:46.052229 (+ 722us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:02:46.052366 (+ 137us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:02:46.054657 (+ 2291us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1215 21:02:46.054679 (+ 22us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1215 21:02:46.054683 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1215 21:02:46.054712 (+ 29us) server_negotiation.cc:923] calling sasl_server_start()
1215 21:02:46.056241 (+ 1529us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1215 21:02:46.056675 (+ 434us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1215 21:02:46.056678 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1215 21:02:46.056680 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1215 21:02:46.056721 (+ 41us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1215 21:02:46.056969 (+ 248us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1215 21:02:46.056972 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1215 21:02:46.056974 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1215 21:02:46.057151 (+ 177us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1215 21:02:46.057247 (+ 96us) server_negotiation.cc:1036] Waiting for connection context
1215 21:02:46.057724 (+ 477us) server_negotiation.cc:300] Negotiation successful
1215 21:02:46.057844 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20251215 21:02:46.058682 3958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:02:46.048756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.113.62:41229 (local address 127.0.0.1:42432)
1215 21:02:46.049087 (+ 331us) negotiation.cc:107] Waiting for socket to connect
1215 21:02:46.049124 (+ 37us) client_negotiation.cc:174] Beginning negotiation
1215 21:02:46.049371 (+ 247us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1215 21:02:46.049729 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1215 21:02:46.049734 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
1215 21:02:46.050214 (+ 480us) client_negotiation.cc:189] Negotiated authn=SASL
1215 21:02:46.050604 (+ 390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:02:46.050618 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:02:46.051602 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1215 21:02:46.051604 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1215 21:02:46.052129 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:02:46.052138 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:02:46.052326 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:02:46.052952 (+ 626us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1215 21:02:46.052984 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
1215 21:02:46.054526 (+ 1542us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1215 21:02:46.056339 (+ 1813us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1215 21:02:46.056347 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1215 21:02:46.056355 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
1215 21:02:46.056579 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1215 21:02:46.056807 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1215 21:02:46.056809 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1215 21:02:46.056811 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1215 21:02:46.056888 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1215 21:02:46.057267 (+ 379us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1215 21:02:46.057274 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1215 21:02:46.057503 (+ 229us) client_negotiation.cc:769] Sending connection context
1215 21:02:46.057677 (+ 174us) client_negotiation.cc:240] Negotiation successful
1215 21:02:46.057930 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":96,"threads_started":1}
I20251215 21:02:46.058682 3957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:02:46.048237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55186 (local address 127.28.113.61:39207)
1215 21:02:46.048599 (+ 362us) server_negotiation.cc:207] Beginning negotiation
1215 21:02:46.048608 (+ 9us) server_negotiation.cc:400] Waiting for connection header
1215 21:02:46.048637 (+ 29us) server_negotiation.cc:408] Connection header received
1215 21:02:46.049228 (+ 591us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1215 21:02:46.049252 (+ 24us) server_negotiation.cc:462] Received NEGOTIATE request from client
1215 21:02:46.049442 (+ 190us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1215 21:02:46.049616 (+ 174us) server_negotiation.cc:227] Negotiated authn=SASL
1215 21:02:46.050620 (+ 1004us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:02:46.051680 (+ 1060us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1215 21:02:46.052484 (+ 804us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:02:46.052663 (+ 179us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:02:46.055088 (+ 2425us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1215 21:02:46.055108 (+ 20us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1215 21:02:46.055118 (+ 10us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1215 21:02:46.055143 (+ 25us) server_negotiation.cc:923] calling sasl_server_start()
1215 21:02:46.056835 (+ 1692us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1215 21:02:46.057228 (+ 393us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1215 21:02:46.057233 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1215 21:02:46.057234 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1215 21:02:46.057289 (+ 55us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1215 21:02:46.057500 (+ 211us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1215 21:02:46.057503 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1215 21:02:46.057505 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1215 21:02:46.057691 (+ 186us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1215 21:02:46.057856 (+ 165us) server_negotiation.cc:1036] Waiting for connection context
1215 21:02:46.058054 (+ 198us) server_negotiation.cc:300] Negotiation successful
1215 21:02:46.058172 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":237,"spinlock_wait_cycles":896,"thread_start_us":103,"threads_started":1}
I20251215 21:02:46.059998 3954 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.113.61" port: 39207 } has no permanent_uuid. Determining permanent_uuid...
Dec 15 21:02:46 dist-test-slave-bjsf krb5kdc[3366](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765832565, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.113.61@KRBTEST.COM for oryx/127.28.113.61@KRBTEST.COM
I20251215 21:02:46.066056 3958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:02:46.060442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.113.61:39207 (local address 127.0.0.1:55192)
1215 21:02:46.060538 (+ 96us) negotiation.cc:107] Waiting for socket to connect
1215 21:02:46.060551 (+ 13us) client_negotiation.cc:174] Beginning negotiation
1215 21:02:46.060622 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1215 21:02:46.060857 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1215 21:02:46.060859 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1215 21:02:46.060976 (+ 117us) client_negotiation.cc:189] Negotiated authn=SASL
1215 21:02:46.061149 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:02:46.061153 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:02:46.061768 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1215 21:02:46.061771 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1215 21:02:46.062096 (+ 325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:02:46.062103 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:02:46.062182 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:02:46.062597 (+ 415us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1215 21:02:46.062610 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
1215 21:02:46.063698 (+ 1088us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1215 21:02:46.065125 (+ 1427us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1215 21:02:46.065128 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1215 21:02:46.065129 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1215 21:02:46.065268 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1215 21:02:46.065431 (+ 163us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1215 21:02:46.065434 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1215 21:02:46.065435 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1215 21:02:46.065471 (+ 36us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1215 21:02:46.065706 (+ 235us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1215 21:02:46.065708 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1215 21:02:46.065774 (+ 66us) client_negotiation.cc:769] Sending connection context
1215 21:02:46.065841 (+ 67us) client_negotiation.cc:240] Negotiation successful
1215 21:02:46.065928 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20251215 21:02:46.066084 3957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:02:46.060606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55192 (local address 127.28.113.61:39207)
1215 21:02:46.060701 (+ 95us) server_negotiation.cc:207] Beginning negotiation
1215 21:02:46.060704 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1215 21:02:46.060714 (+ 10us) server_negotiation.cc:408] Connection header received
1215 21:02:46.060739 (+ 25us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1215 21:02:46.060741 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1215 21:02:46.060772 (+ 31us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1215 21:02:46.060835 (+ 63us) server_negotiation.cc:227] Negotiated authn=SASL
1215 21:02:46.061228 (+ 393us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:02:46.061653 (+ 425us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1215 21:02:46.062173 (+ 520us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:02:46.062295 (+ 122us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:02:46.063817 (+ 1522us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1215 21:02:46.063830 (+ 13us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1215 21:02:46.063832 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1215 21:02:46.063850 (+ 18us) server_negotiation.cc:923] calling sasl_server_start()
1215 21:02:46.065032 (+ 1182us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1215 21:02:46.065340 (+ 308us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1215 21:02:46.065343 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1215 21:02:46.065344 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1215 21:02:46.065376 (+ 32us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1215 21:02:46.065531 (+ 155us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1215 21:02:46.065533 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1215 21:02:46.065535 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1215 21:02:46.065645 (+ 110us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1215 21:02:46.065712 (+ 67us) server_negotiation.cc:1036] Waiting for connection context
1215 21:02:46.065879 (+ 167us) server_negotiation.cc:300] Negotiation successful
1215 21:02:46.065976 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":16,"server-negotiator.queue_time_us":34}
I20251215 21:02:46.069017 3954 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P de9d22b45210451f997093f481e4dfc6: Bootstrap starting.
I20251215 21:02:46.071242 3954 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P de9d22b45210451f997093f481e4dfc6: Neither blocks nor log segments found. Creating new log.
I20251215 21:02:46.071908 3954 log.cc:826] T 00000000000000000000000000000000 P de9d22b45210451f997093f481e4dfc6: Log is configured to *not* fsync() on all Append() calls
I20251215 21:02:46.073501 3954 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P de9d22b45210451f997093f481e4dfc6: No bootstrap required, opened a new log
I20251215 21:02:46.076081 3954 raft_consensus.cc:359] T 00000000000000000000000000000000 P de9d22b45210451f997093f481e4dfc6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c9267b46ebc4bae9a13928cf23fa8d3" member_type: VOTER last_known_addr { host: "127.28.113.62" port: 41229 } } peers { permanent_uuid: "de9d22b45210451f997093f481e4dfc6" member_type: VOTER last_known_addr { host: "127.28.113.61" port: 39207 } }
I20251215 21:02:46.076241 3954 raft_consensus.cc:385] T 00000000000000000000000000000000 P de9d22b45210451f997093f481e4dfc6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251215 21:02:46.076292 3954 raft_consensus.cc:740] T 00000000000000000000000000000000 P de9d22b45210451f997093f481e4dfc6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de9d22b45210451f997093f481e4dfc6, State: Initialized, Role: FOLLOWER
I20251215 21:02:46.076659 3954 consensus_queue.cc:260] T 00000000000000000000000000000000 P de9d22b45210451f997093f481e4dfc6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c9267b46ebc4bae9a13928cf23fa8d3" member_type: VOTER last_known_addr { host: "127.28.113.62" port: 41229 } } peers { permanent_uuid: "de9d22b45210451f997093f481e4dfc6" member_type: VOTER last_known_addr { host: "127.28.113.61" port: 39207 } }
I20251215 21:02:46.077059 3960 sys_catalog.cc:455] T 00000000000000000000000000000000 P de9d22b45210451f997093f481e4dfc6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c9267b46ebc4bae9a13928cf23fa8d3" member_type: VOTER last_known_addr { host: "127.28.113.62" port: 41229 } } peers { permanent_uuid: "de9d22b45210451f997093f481e4dfc6" member_type: VOTER last_known_addr { host: "127.28.113.61" port: 39207 } } }
I20251215 21:02:46.077162 3960 sys_catalog.cc:458] T 00000000000000000000000000000000 P de9d22b45210451f997093f481e4dfc6 [sys.catalog]: This master's current role is: FOLLOWER
I20251215 21:02:46.077627 3954 sys_catalog.cc:565] T 00000000000000000000000000000000 P de9d22b45210451f997093f481e4dfc6 [sys.catalog]: configured and running, proceeding with master startup.
W20251215 21:02:46.082077 3971 catalog_manager.cc:1568] T 00000000000000000000000000000000 P de9d22b45210451f997093f481e4dfc6: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20251215 21:02:46.082139 3971 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20251215 21:02:46.083926 3886 master_runner.cc:428] Detected that this master de9d22b45210451f997093f481e4dfc6 is joining an existing cluster
I20251215 21:02:46.084007 3886 master_runner.cc:432] Initiating AddMaster RPC to add 127.28.113.61:39207
I20251215 21:02:46.092768 3978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:02:46.085686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.113.62:41229 (local address 127.0.0.1:42440)
1215 21:02:46.085977 (+ 291us) negotiation.cc:107] Waiting for socket to connect
1215 21:02:46.085991 (+ 14us) client_negotiation.cc:174] Beginning negotiation
1215 21:02:46.086079 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1215 21:02:46.086516 (+ 437us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1215 21:02:46.086520 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
1215 21:02:46.086735 (+ 215us) client_negotiation.cc:189] Negotiated authn=SASL
1215 21:02:46.087064 (+ 329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:02:46.087082 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:02:46.088128 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1215 21:02:46.088131 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1215 21:02:46.088957 (+ 826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:02:46.088966 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:02:46.089080 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:02:46.089510 (+ 430us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1215 21:02:46.089523 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
1215 21:02:46.089769 (+ 246us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1215 21:02:46.091489 (+ 1720us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1215 21:02:46.091493 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1215 21:02:46.091494 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1215 21:02:46.091645 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1215 21:02:46.091873 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1215 21:02:46.091875 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1215 21:02:46.091876 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1215 21:02:46.091915 (+ 39us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1215 21:02:46.092361 (+ 446us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1215 21:02:46.092364 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1215 21:02:46.092440 (+ 76us) client_negotiation.cc:769] Sending connection context
1215 21:02:46.092531 (+ 91us) client_negotiation.cc:240] Negotiation successful
1215 21:02:46.092623 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":130,"threads_started":1}
I20251215 21:02:46.093034 3453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:02:46.086029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42440 (local address 127.28.113.62:41229)
1215 21:02:46.086246 (+ 217us) server_negotiation.cc:207] Beginning negotiation
1215 21:02:46.086251 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1215 21:02:46.086265 (+ 14us) server_negotiation.cc:408] Connection header received
1215 21:02:46.086304 (+ 39us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1215 21:02:46.086307 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1215 21:02:46.086357 (+ 50us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1215 21:02:46.086436 (+ 79us) server_negotiation.cc:227] Negotiated authn=SASL
1215 21:02:46.087193 (+ 757us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:02:46.088003 (+ 810us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1215 21:02:46.089073 (+ 1070us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:02:46.089269 (+ 196us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:02:46.089872 (+ 603us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1215 21:02:46.089889 (+ 17us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1215 21:02:46.089893 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1215 21:02:46.089916 (+ 23us) server_negotiation.cc:923] calling sasl_server_start()
1215 21:02:46.091375 (+ 1459us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1215 21:02:46.091735 (+ 360us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1215 21:02:46.091739 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1215 21:02:46.091741 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1215 21:02:46.091791 (+ 50us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1215 21:02:46.092047 (+ 256us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1215 21:02:46.092052 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1215 21:02:46.092054 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1215 21:02:46.092237 (+ 183us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1215 21:02:46.092350 (+ 113us) server_negotiation.cc:1036] Waiting for connection context
1215 21:02:46.092596 (+ 246us) server_negotiation.cc:300] Negotiation successful
1215 21:02:46.092710 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136}
Dec 15 21:02:46 dist-test-slave-bjsf krb5kdc[3366](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765832565, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.113.62@KRBTEST.COM for oryx/127.28.113.61@KRBTEST.COM
I20251215 21:02:46.107692 3957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:02:46.086128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55204 (local address 127.28.113.61:39207)
1215 21:02:46.086303 (+ 175us) server_negotiation.cc:207] Beginning negotiation
1215 21:02:46.086305 (+ 2us) server_negotiation.cc:400] Waiting for connection header
1215 21:02:46.086312 (+ 7us) server_negotiation.cc:408] Connection header received
1215 21:02:46.086331 (+ 19us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1215 21:02:46.086333 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1215 21:02:46.086358 (+ 25us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1215 21:02:46.086412 (+ 54us) server_negotiation.cc:227] Negotiated authn=SASL
1215 21:02:46.087357 (+ 945us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:02:46.087758 (+ 401us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1215 21:02:46.088443 (+ 685us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:02:46.088634 (+ 191us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:02:46.104352 (+ 15718us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1215 21:02:46.104366 (+ 14us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1215 21:02:46.104369 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1215 21:02:46.104389 (+ 20us) server_negotiation.cc:923] calling sasl_server_start()
1215 21:02:46.106285 (+ 1896us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1215 21:02:46.106725 (+ 440us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1215 21:02:46.106728 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1215 21:02:46.106730 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1215 21:02:46.106763 (+ 33us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1215 21:02:46.106961 (+ 198us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1215 21:02:46.106964 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1215 21:02:46.106966 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1215 21:02:46.107120 (+ 154us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1215 21:02:46.107181 (+ 61us) server_negotiation.cc:1036] Waiting for connection context
1215 21:02:46.107431 (+ 250us) server_negotiation.cc:300] Negotiation successful
1215 21:02:46.107536 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129}
I20251215 21:02:46.108346 3979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:02:46.097324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.113.61:39207 (local address 127.0.0.1:55216)
1215 21:02:46.097637 (+ 313us) negotiation.cc:107] Waiting for socket to connect
1215 21:02:46.097672 (+ 35us) client_negotiation.cc:174] Beginning negotiation
1215 21:02:46.097807 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1215 21:02:46.098150 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1215 21:02:46.098155 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
1215 21:02:46.098487 (+ 332us) client_negotiation.cc:189] Negotiated authn=SASL
1215 21:02:46.098685 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:02:46.098691 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:02:46.099651 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1215 21:02:46.099655 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1215 21:02:46.100164 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:02:46.100173 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:02:46.100280 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:02:46.100803 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1215 21:02:46.100817 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
1215 21:02:46.102474 (+ 1657us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1215 21:02:46.106765 (+ 4291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1215 21:02:46.106774 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1215 21:02:46.106780 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
1215 21:02:46.107053 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1215 21:02:46.107298 (+ 245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1215 21:02:46.107302 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1215 21:02:46.107304 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1215 21:02:46.107353 (+ 49us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1215 21:02:46.107711 (+ 358us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1215 21:02:46.107718 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1215 21:02:46.107843 (+ 125us) client_negotiation.cc:769] Sending connection context
1215 21:02:46.108015 (+ 172us) client_negotiation.cc:240] Negotiation successful
1215 21:02:46.108176 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":80,"threads_started":1}
I20251215 21:02:46.108362 3977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:02:46.085695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.113.61:39207 (local address 127.0.0.1:55204)
1215 21:02:46.086135 (+ 440us) negotiation.cc:107] Waiting for socket to connect
1215 21:02:46.086146 (+ 11us) client_negotiation.cc:174] Beginning negotiation
1215 21:02:46.086206 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1215 21:02:46.086448 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1215 21:02:46.086451 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1215 21:02:46.086741 (+ 290us) client_negotiation.cc:189] Negotiated authn=SASL
1215 21:02:46.087269 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:02:46.087275 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:02:46.087863 (+ 588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1215 21:02:46.087865 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1215 21:02:46.088334 (+ 469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:02:46.088342 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:02:46.088442 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:02:46.088941 (+ 499us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1215 21:02:46.088957 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
1215 21:02:46.104241 (+ 15284us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1215 21:02:46.106384 (+ 2143us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1215 21:02:46.106387 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1215 21:02:46.106389 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1215 21:02:46.106646 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1215 21:02:46.106846 (+ 200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1215 21:02:46.106848 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1215 21:02:46.106850 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1215 21:02:46.106890 (+ 40us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1215 21:02:46.107233 (+ 343us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1215 21:02:46.107236 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1215 21:02:46.107300 (+ 64us) client_negotiation.cc:769] Sending connection context
1215 21:02:46.108094 (+ 794us) client_negotiation.cc:240] Negotiation successful
1215 21:02:46.108211 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":379,"spinlock_wait_cycles":14080,"thread_start_us":111,"threads_started":1}
I20251215 21:02:46.108986 3980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:02:46.097367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55216 (local address 127.28.113.61:39207)
1215 21:02:46.097635 (+ 268us) server_negotiation.cc:207] Beginning negotiation
1215 21:02:46.097639 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1215 21:02:46.097750 (+ 111us) server_negotiation.cc:408] Connection header received
1215 21:02:46.097949 (+ 199us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1215 21:02:46.097952 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1215 21:02:46.098009 (+ 57us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1215 21:02:46.098094 (+ 85us) server_negotiation.cc:227] Negotiated authn=SASL
1215 21:02:46.098791 (+ 697us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:02:46.099545 (+ 754us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1215 21:02:46.100268 (+ 723us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:02:46.100788 (+ 520us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:02:46.102620 (+ 1832us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1215 21:02:46.102638 (+ 18us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1215 21:02:46.102642 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1215 21:02:46.102667 (+ 25us) server_negotiation.cc:923] calling sasl_server_start()
1215 21:02:46.106636 (+ 3969us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1215 21:02:46.107161 (+ 525us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1215 21:02:46.107164 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1215 21:02:46.107166 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1215 21:02:46.107213 (+ 47us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1215 21:02:46.107434 (+ 221us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1215 21:02:46.107437 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1215 21:02:46.107439 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1215 21:02:46.107583 (+ 144us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1215 21:02:46.107659 (+ 76us) server_negotiation.cc:1036] Waiting for connection context
1215 21:02:46.108715 (+ 1056us) server_negotiation.cc:300] Negotiation successful
1215 21:02:46.108831 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":197,"thread_start_us":99,"threads_started":1}
I20251215 21:02:46.110139 3414 catalog_manager.cc:7141] Initiating ChangeConfig request to add master 127.28.113.61:39207: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "de9d22b45210451f997093f481e4dfc6"
member_type: NON_VOTER
last_known_addr {
host: "127.28.113.61"
port: 39207
}
attrs {
promote: true
}
}
dest_uuid: "8c9267b46ebc4bae9a13928cf23fa8d3"
cas_config_opid_index: -1
I20251215 21:02:46.110730 3414 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8c9267b46ebc4bae9a13928cf23fa8d3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "8c9267b46ebc4bae9a13928cf23fa8d3" member_type: VOTER last_known_addr { host: "127.28.113.62" port: 41229 } } peers { permanent_uuid: "de9d22b45210451f997093f481e4dfc6" member_type: NON_VOTER last_known_addr { host: "127.28.113.61" port: 39207 } attrs { promote: true } }
I20251215 21:02:46.113224 3928 raft_consensus.cc:3060] T 00000000000000000000000000000000 P de9d22b45210451f997093f481e4dfc6 [term 0 FOLLOWER]: Advancing to term 1
I20251215 21:02:46.114391 3928 raft_consensus.cc:1275] T 00000000000000000000000000000000 P de9d22b45210451f997093f481e4dfc6 [term 1 FOLLOWER]: Refusing update from remote peer 8c9267b46ebc4bae9a13928cf23fa8d3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20251215 21:02:46.112848 3982 raft_consensus.cc:2955] T 00000000000000000000000000000000 P 8c9267b46ebc4bae9a13928cf23fa8d3 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER de9d22b45210451f997093f481e4dfc6 (127.28.113.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "8c9267b46ebc4bae9a13928cf23fa8d3" member_type: VOTER last_known_addr { host: "127.28.113.62" port: 41229 } } peers { permanent_uuid: "de9d22b45210451f997093f481e4dfc6" member_type: NON_VOTER last_known_addr { host: "127.28.113.61" port: 39207 } attrs { promote: true } } }
I20251215 21:02:46.114936 3981 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 8c9267b46ebc4bae9a13928cf23fa8d3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "de9d22b45210451f997093f481e4dfc6" member_type: NON_VOTER last_known_addr { host: "127.28.113.61" port: 39207 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
I20251215 21:02:46.116082 3982 catalog_manager.cc:7145] Successfully completed master ChangeConfig request to add master 127.28.113.61:39207
I20251215 21:02:46.116302 3981 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c9267b46ebc4bae9a13928cf23fa8d3 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "8c9267b46ebc4bae9a13928cf23fa8d3" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "8c9267b46ebc4bae9a13928cf23fa8d3" member_type: VOTER last_known_addr { host: "127.28.113.62" port: 41229 } } peers { permanent_uuid: "de9d22b45210451f997093f481e4dfc6" member_type: NON_VOTER last_known_addr { host: "127.28.113.61" port: 39207 } attrs { promote: true } } }
I20251215 21:02:46.116384 3981 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c9267b46ebc4bae9a13928cf23fa8d3 [sys.catalog]: This master's current role is: LEADER
I20251215 21:02:46.116494 3886 master.cc:561] Master@127.28.113.61:39207 shutting down...
I20251215 21:02:46.121959 3928 raft_consensus.cc:2955] T 00000000000000000000000000000000 P de9d22b45210451f997093f481e4dfc6 [term 1 LEARNER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER de9d22b45210451f997093f481e4dfc6 (127.28.113.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "8c9267b46ebc4bae9a13928cf23fa8d3" member_type: VOTER last_known_addr { host: "127.28.113.62" port: 41229 } } peers { permanent_uuid: "de9d22b45210451f997093f481e4dfc6" member_type: NON_VOTER last_known_addr { host: "127.28.113.61" port: 39207 } attrs { promote: true } } }
I20251215 21:02:46.123045 3986 mvcc.cc:204] Tried to move back new op lower bound from 7232850186877702144 to 7232850186730635264. Current Snapshot: MvccSnapshot[applied={T|T < 7232850186763186176}]
WARNING: no policy specified for oryx/127.28.113.60@KRBTEST.COM; defaulting to no policy
I20251215 21:02:46.124047 3960 sys_catalog.cc:455] T 00000000000000000000000000000000 P de9d22b45210451f997093f481e4dfc6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8c9267b46ebc4bae9a13928cf23fa8d3. Latest consensus state: current_term: 1 leader_uuid: "8c9267b46ebc4bae9a13928cf23fa8d3" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "8c9267b46ebc4bae9a13928cf23fa8d3" member_type: VOTER last_known_addr { host: "127.28.113.62" port: 41229 } } peers { permanent_uuid: "de9d22b45210451f997093f481e4dfc6" member_type: NON_VOTER last_known_addr { host: "127.28.113.61" port: 39207 } attrs { promote: true } } }
I20251215 21:02:46.124148 3960 sys_catalog.cc:458] T 00000000000000000000000000000000 P de9d22b45210451f997093f481e4dfc6 [sys.catalog]: This master's current role is: LEARNER
I20251215 21:02:46.124236 3960 sys_catalog.cc:455] T 00000000000000000000000000000000 P de9d22b45210451f997093f481e4dfc6 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8c9267b46ebc4bae9a13928cf23fa8d3" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "8c9267b46ebc4bae9a13928cf23fa8d3" member_type: VOTER last_known_addr { host: "127.28.113.62" port: 41229 } } peers { permanent_uuid: "de9d22b45210451f997093f481e4dfc6" member_type: NON_VOTER last_known_addr { host: "127.28.113.61" port: 39207 } attrs { promote: true } } }
I20251215 21:02:46.124298 3960 sys_catalog.cc:458] T 00000000000000000000000000000000 P de9d22b45210451f997093f481e4dfc6 [sys.catalog]: This master's current role is: LEARNER
I20251215 21:02:46.124357 3960 sys_catalog.cc:455] T 00000000000000000000000000000000 P de9d22b45210451f997093f481e4dfc6 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8c9267b46ebc4bae9a13928cf23fa8d3" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "8c9267b46ebc4bae9a13928cf23fa8d3" member_type: VOTER last_known_addr { host: "127.28.113.62" port: 41229 } } peers { permanent_uuid: "de9d22b45210451f997093f481e4dfc6" member_type: NON_VOTER last_known_addr { host: "127.28.113.61" port: 39207 } attrs { promote: true } } }
I20251215 21:02:46.124413 3960 sys_catalog.cc:458] T 00000000000000000000000000000000 P de9d22b45210451f997093f481e4dfc6 [sys.catalog]: This master's current role is: LEARNER
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.28.113.60@KRBTEST.COM" created.
I20251215 21:02:46.128386 3886 raft_consensus.cc:2243] T 00000000000000000000000000000000 P de9d22b45210451f997093f481e4dfc6 [term 1 LEARNER]: Raft consensus shutting down.
I20251215 21:02:46.128687 3886 raft_consensus.cc:2272] T 00000000000000000000000000000000 P de9d22b45210451f997093f481e4dfc6 [term 1 LEARNER]: Raft consensus is shut down!
I20251215 21:02:46.128769 3886 tablet_replica.cc:333] T 00000000000000000000000000000000 P de9d22b45210451f997093f481e4dfc6: stopping tablet replica
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.28.113.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765832506145157-29124-0/krb5kdc/oryx.keytab.
Entry for principal oryx/127.28.113.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765832506145157-29124-0/krb5kdc/oryx.keytab.
I20251215 21:02:46.143748 3886 master.cc:583] Master@127.28.113.61:39207 shutdown complete.
I20251215 21:02:46.143810 3886 master_runner.cc:305] Clearing existing system tablet
I20251215 21:02:46.144075 3886 fs_manager.cc:362] Metadata directory not provided
I20251215 21:02:46.144130 3886 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251215 21:02:46.145354 3886 fs_manager.cc:714] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20251215 21:02:46.146353 3996 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
WARNING: no policy specified for HTTP/127.28.113.60@KRBTEST.COM; defaulting to no policy
I20251215 21:02:46.146584 3886 fs_report.cc:389] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765832506145157-29124-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251215 21:02:46.146704 3886 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20251215 21:02:46.146742 3886 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765832506145157-29124-0/minicluster-data/master-1/data,/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765832506145157-29124-0/minicluster-data/master-1/wal
uuid: "de9d22b45210451f997093f481e4dfc6"
format_stamp: "Formatted at 2025-12-15 21:02:46 on dist-test-slave-bjsf"
server_key: "802a0ba8fcedee52556d977987b287f7"
server_key_iv: "e374525312c5cc54d2234ab16636ef84"
server_key_version: "encryptionkey@0"
I20251215 21:02:46.148326 3886 ts_tablet_manager.cc:1916] T 00000000000000000000000000000000 P de9d22b45210451f997093f481e4dfc6: Deleting tablet data with delete state TABLET_DATA_DELETED
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.28.113.60@KRBTEST.COM" created.
I20251215 21:02:46.150053 3886 ts_tablet_manager.cc:1929] T 00000000000000000000000000000000 P de9d22b45210451f997093f481e4dfc6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20251215 21:02:46.150137 3886 log.cc:1199] T 00000000000000000000000000000000 P de9d22b45210451f997093f481e4dfc6: Deleting WAL directory at /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765832506145157-29124-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20251215 21:02:46.150398 3886 ts_tablet_manager.cc:1950] T 00000000000000000000000000000000 P de9d22b45210451f997093f481e4dfc6: Deleting consensus metadata
I20251215 21:02:46.150544 3886 master_runner.cc:315] Copying system tablet from 127.28.113.62:41229
I20251215 21:02:46.151664 3886 tablet_copy_client.cc:323] T 00000000000000000000000000000000 P de9d22b45210451f997093f481e4dfc6: tablet copy: Beginning tablet copy session from remote peer at address 127.28.113.62:41229
I20251215 21:02:46.159351 3453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:02:46.152571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42448 (local address 127.28.113.62:41229)
1215 21:02:46.152706 (+ 135us) server_negotiation.cc:207] Beginning negotiation
1215 21:02:46.152711 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1215 21:02:46.152726 (+ 15us) server_negotiation.cc:408] Connection header received
1215 21:02:46.152774 (+ 48us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1215 21:02:46.152777 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1215 21:02:46.152833 (+ 56us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1215 21:02:46.152923 (+ 90us) server_negotiation.cc:227] Negotiated authn=SASL
1215 21:02:46.153579 (+ 656us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:02:46.154346 (+ 767us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1215 21:02:46.155184 (+ 838us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:02:46.155376 (+ 192us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:02:46.156393 (+ 1017us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1215 21:02:46.156411 (+ 18us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1215 21:02:46.156413 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1215 21:02:46.156435 (+ 22us) server_negotiation.cc:923] calling sasl_server_start()
1215 21:02:46.157875 (+ 1440us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1215 21:02:46.158272 (+ 397us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1215 21:02:46.158275 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1215 21:02:46.158276 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1215 21:02:46.158317 (+ 41us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1215 21:02:46.158552 (+ 235us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1215 21:02:46.158554 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1215 21:02:46.158556 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1215 21:02:46.158684 (+ 128us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1215 21:02:46.158751 (+ 67us) server_negotiation.cc:1036] Waiting for connection context
1215 21:02:46.159087 (+ 336us) server_negotiation.cc:300] Negotiation successful
1215 21:02:46.159175 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20251215 21:02:46.159705 4003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:02:46.152203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.113.62:41229 (local address 127.0.0.1:42448)
1215 21:02:46.152450 (+ 247us) negotiation.cc:107] Waiting for socket to connect
1215 21:02:46.152466 (+ 16us) client_negotiation.cc:174] Beginning negotiation
1215 21:02:46.152575 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1215 21:02:46.152942 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1215 21:02:46.152945 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1215 21:02:46.153162 (+ 217us) client_negotiation.cc:189] Negotiated authn=SASL
1215 21:02:46.153446 (+ 284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:02:46.153454 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:02:46.154457 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1215 21:02:46.154461 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1215 21:02:46.155063 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:02:46.155072 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:02:46.155177 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:02:46.155844 (+ 667us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1215 21:02:46.155860 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
1215 21:02:46.156254 (+ 394us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1215 21:02:46.157971 (+ 1717us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1215 21:02:46.157981 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1215 21:02:46.157984 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
1215 21:02:46.158182 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1215 21:02:46.158383 (+ 201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1215 21:02:46.158386 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1215 21:02:46.158389 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
1215 21:02:46.158438 (+ 49us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1215 21:02:46.158791 (+ 353us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1215 21:02:46.158795 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1215 21:02:46.158904 (+ 109us) client_negotiation.cc:769] Sending connection context
1215 21:02:46.159035 (+ 131us) client_negotiation.cc:240] Negotiation successful
1215 21:02:46.159168 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":95,"threads_started":1}
I20251215 21:02:46.160331 3434 tablet_copy_service.cc:140] P 8c9267b46ebc4bae9a13928cf23fa8d3: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer de9d22b45210451f997093f481e4dfc6 ({username='oryx', principal='oryx/127.28.113.61@KRBTEST.COM'} at 127.0.0.1:42448)
I20251215 21:02:46.160475 3434 tablet_copy_service.cc:161] P 8c9267b46ebc4bae9a13928cf23fa8d3: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer de9d22b45210451f997093f481e4dfc6 at {username='oryx', principal='oryx/127.28.113.61@KRBTEST.COM'} at 127.0.0.1:42448: session id = de9d22b45210451f997093f481e4dfc6-00000000000000000000000000000000
I20251215 21:02:46.161921 3434 tablet_copy_source_session.cc:215] T 00000000000000000000000000000000 P 8c9267b46ebc4bae9a13928cf23fa8d3: Tablet Copy: opened 0 blocks and 1 log segments
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.28.113.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765832506145157-29124-0/krb5kdc/oryx.keytab.
Entry for principal HTTP/127.28.113.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765832506145157-29124-0/krb5kdc/oryx.keytab.
I20251215 21:02:46.162985 29124 external_mini_cluster.cc:1366] Running /tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu
/tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765832506145157-29124-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765832506145157-29124-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765832506145157-29124-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765832506145157-29124-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.113.60:34031
--webserver_interface=127.28.113.60
--webserver_port=0
--keytab_file=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765832506145157-29124-0/krb5kdc/oryx.keytab
--principal=oryx/127.28.113.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.28.113.20:39895
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.113.62:41229,127.28.113.61:39207,127.28.113.60:34031
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765832506145157-29124-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765832506145157-29124-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765832506145157-29124-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20251215 21:02:46.163439 3886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20251215 21:02:46.165560 3886 tablet_copy_client.cc:806] T 00000000000000000000000000000000 P de9d22b45210451f997093f481e4dfc6: tablet copy: Starting download of 0 data blocks...
I20251215 21:02:46.165712 3886 tablet_copy_client.cc:670] T 00000000000000000000000000000000 P de9d22b45210451f997093f481e4dfc6: tablet copy: Starting download of 1 WAL segments...
I20251215 21:02:46.167688 3886 tablet_copy_client.cc:538] T 00000000000000000000000000000000 P de9d22b45210451f997093f481e4dfc6: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20251215 21:02:46.169600 3434 tablet_copy_service.cc:342] P 8c9267b46ebc4bae9a13928cf23fa8d3: Request end of tablet copy session de9d22b45210451f997093f481e4dfc6-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.28.113.61@KRBTEST.COM'} at 127.0.0.1:42448
I20251215 21:02:46.169726 3434 tablet_copy_service.cc:434] P 8c9267b46ebc4bae9a13928cf23fa8d3: ending tablet copy session de9d22b45210451f997093f481e4dfc6-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer de9d22b45210451f997093f481e4dfc6
W20251215 21:02:46.175217 3886 builtin_ntp.cc:688] coult not shutdown socket: Network error: shutdown error: Transport endpoint is not connected (error 107)
I20251215 21:02:46.177819 3886 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251215 21:02:46.179487 4008 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251215 21:02:46.179730 4005 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251215 21:02:46.180084 4006 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251215 21:02:46.180150 3886 server_base.cc:1047] running on GCE node
I20251215 21:02:46.180356 3886 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251215 21:02:46.180635 3886 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251215 21:02:46.181777 3886 hybrid_clock.cc:648] HybridClock initialized: now 1765832566181757 us; error 36 us; skew 500 ppm
Dec 15 21:02:46 dist-test-slave-bjsf krb5kdc[3366](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765832566, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.113.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251215 21:02:46.183470 3886 init.cc:377] Logged in from keytab as oryx/127.28.113.61@KRBTEST.COM (short username oryx)
I20251215 21:02:46.183935 3886 webserver.cc:492] Webserver started at http://127.28.113.61:34757/ using document root <none> and password file <none>
I20251215 21:02:46.184203 3886 fs_manager.cc:362] Metadata directory not provided
I20251215 21:02:46.184243 3886 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251215 21:02:46.185554 3886 fs_manager.cc:714] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20251215 21:02:46.186618 4015 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251215 21:02:46.188920 3886 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20251215 21:02:46.189002 3886 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765832506145157-29124-0/minicluster-data/master-1/data,/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765832506145157-29124-0/minicluster-data/master-1/wal
uuid: "de9d22b45210451f997093f481e4dfc6"
format_stamp: "Formatted at 2025-12-15 21:02:46 on dist-test-slave-bjsf"
server_key: "802a0ba8fcedee52556d977987b287f7"
server_key_iv: "e374525312c5cc54d2234ab16636ef84"
server_key_version: "encryptionkey@0"
I20251215 21:02:46.189076 3886 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765832506145157-29124-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765832506145157-29124-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765832506145157-29124-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251215 21:02:46.204545 3886 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251215 21:02:46.206789 3886 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251215 21:02:46.211426 3886 rpc_server.cc:307] RPC server started. Bound to: 127.28.113.61:39207
I20251215 21:02:46.211580 4067 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.28.113.61:39207 every 8 connection(s)
I20251215 21:02:46.212061 3886 server_base.cc:1179] Dumped server information to /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765832506145157-29124-0/minicluster-data/master-1/data/info.pb
I20251215 21:02:46.213804 4068 sys_catalog.cc:263] Verifying existing consensus state
I20251215 21:02:46.214708 4068 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P de9d22b45210451f997093f481e4dfc6: Bootstrap starting.
I20251215 21:02:46.221918 4068 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P de9d22b45210451f997093f481e4dfc6: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251215 21:02:46.222203 4068 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P de9d22b45210451f997093f481e4dfc6: Bootstrap complete.
I20251215 21:02:46.222908 4068 raft_consensus.cc:359] T 00000000000000000000000000000000 P de9d22b45210451f997093f481e4dfc6 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "8c9267b46ebc4bae9a13928cf23fa8d3" member_type: VOTER last_known_addr { host: "127.28.113.62" port: 41229 } } peers { permanent_uuid: "de9d22b45210451f997093f481e4dfc6" member_type: NON_VOTER last_known_addr { host: "127.28.113.61" port: 39207 } attrs { promote: true } }
I20251215 21:02:46.223054 4068 raft_consensus.cc:740] T 00000000000000000000000000000000 P de9d22b45210451f997093f481e4dfc6 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: de9d22b45210451f997093f481e4dfc6, State: Initialized, Role: LEARNER
I20251215 21:02:46.223156 4068 consensus_queue.cc:260] T 00000000000000000000000000000000 P de9d22b45210451f997093f481e4dfc6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "8c9267b46ebc4bae9a13928cf23fa8d3" member_type: VOTER last_known_addr { host: "127.28.113.62" port: 41229 } } peers { permanent_uuid: "de9d22b45210451f997093f481e4dfc6" member_type: NON_VOTER last_known_addr { host: "127.28.113.61" port: 39207 } attrs { promote: true } }
I20251215 21:02:46.223483 4071 sys_catalog.cc:455] T 00000000000000000000000000000000 P de9d22b45210451f997093f481e4dfc6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "8c9267b46ebc4bae9a13928cf23fa8d3" member_type: VOTER last_known_addr { host: "127.28.113.62" port: 41229 } } peers { permanent_uuid: "de9d22b45210451f997093f481e4dfc6" member_type: NON_VOTER last_known_addr { host: "127.28.113.61" port: 39207 } attrs { promote: true } } }
I20251215 21:02:46.223601 4068 sys_catalog.cc:565] T 00000000000000000000000000000000 P de9d22b45210451f997093f481e4dfc6 [sys.catalog]: configured and running, proceeding with master startup.
I20251215 21:02:46.223650 4071 sys_catalog.cc:458] T 00000000000000000000000000000000 P de9d22b45210451f997093f481e4dfc6 [sys.catalog]: This master's current role is: LEARNER
I20251215 21:02:46.226743 4082 catalog_manager.cc:1269] Loaded cluster ID: 587ee523bb434fd8a7a37f28e6cc4cb1
I20251215 21:02:46.226867 4082 catalog_manager.cc:1562] T 00000000000000000000000000000000 P de9d22b45210451f997093f481e4dfc6: loading cluster ID for follower catalog manager: success
I20251215 21:02:46.229166 4082 catalog_manager.cc:1584] T 00000000000000000000000000000000 P de9d22b45210451f997093f481e4dfc6: acquiring CA information for follower catalog manager: success
I20251215 21:02:46.230496 4082 catalog_manager.cc:1612] T 00000000000000000000000000000000 P de9d22b45210451f997093f481e4dfc6: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20251215 21:02:46.279523 4004 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251215 21:02:46.279742 4004 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251215 21:02:46.279820 4004 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251215 21:02:46.283046 4004 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251215 21:02:46.283104 4004 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251215 21:02:46.283133 4004 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251215 21:02:46.283157 4004 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251215 21:02:46.283188 4004 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251215 21:02:46.287318 4004 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.113.20:39895
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765832506145157-29124-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765832506145157-29124-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.28.113.62:41229,127.28.113.61:39207,127.28.113.60:34031
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765832506145157-29124-0/krb5kdc/oryx.keytab
--principal=oryx/127.28.113.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.113.60:34031
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765832506145157-29124-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.28.113.60
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4004
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765832506145157-29124-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision dc741732bb6a40213e11a13869fad06a49a209a3
build type DEBUG
built by None at 15 Dec 2025 20:43:14 UTC on 24a791456cd2
build id 9530
I20251215 21:02:46.288316 4004 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251215 21:02:46.289091 4004 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251215 21:02:46.294153 4088 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251215 21:02:46.294143 4090 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251215 21:02:46.294140 4087 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251215 21:02:46.294497 4004 server_base.cc:1047] running on GCE node
I20251215 21:02:46.294909 4004 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251215 21:02:46.295691 4004 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251215 21:02:46.296826 4004 hybrid_clock.cc:648] HybridClock initialized: now 1765832566296817 us; error 24 us; skew 500 ppm
Dec 15 21:02:46 dist-test-slave-bjsf krb5kdc[3366](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765832566, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.113.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251215 21:02:46.299242 4004 init.cc:377] Logged in from keytab as oryx/127.28.113.60@KRBTEST.COM (short username oryx)
I20251215 21:02:46.300263 4004 webserver.cc:492] Webserver started at http://127.28.113.60:37611/ using document root <none> and password file <none>
I20251215 21:02:46.300747 4004 fs_manager.cc:362] Metadata directory not provided
I20251215 21:02:46.300812 4004 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251215 21:02:46.300974 4004 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251215 21:02:46.302503 4004 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765832506145157-29124-0/minicluster-data/master-2/data/instance:
uuid: "297c0b4eec8b49459719bf6b7429a987"
format_stamp: "Formatted at 2025-12-15 21:02:46 on dist-test-slave-bjsf"
server_key: "7458bcd41abf86350ff9232b33ef2320"
server_key_iv: "60278cc87f7c26862b5bcafc0aeb2694"
server_key_version: "encryptionkey@0"
I20251215 21:02:46.302968 4004 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765832506145157-29124-0/minicluster-data/master-2/wal/instance:
uuid: "297c0b4eec8b49459719bf6b7429a987"
format_stamp: "Formatted at 2025-12-15 21:02:46 on dist-test-slave-bjsf"
server_key: "7458bcd41abf86350ff9232b33ef2320"
server_key_iv: "60278cc87f7c26862b5bcafc0aeb2694"
server_key_version: "encryptionkey@0"
I20251215 21:02:46.306025 4004 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20251215 21:02:46.307932 4097 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251215 21:02:46.308876 4004 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20251215 21:02:46.308971 4004 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765832506145157-29124-0/minicluster-data/master-2/data,/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765832506145157-29124-0/minicluster-data/master-2/wal
uuid: "297c0b4eec8b49459719bf6b7429a987"
format_stamp: "Formatted at 2025-12-15 21:02:46 on dist-test-slave-bjsf"
server_key: "7458bcd41abf86350ff9232b33ef2320"
server_key_iv: "60278cc87f7c26862b5bcafc0aeb2694"
server_key_version: "encryptionkey@0"
I20251215 21:02:46.309049 4004 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765832506145157-29124-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765832506145157-29124-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765832506145157-29124-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251215 21:02:46.330854 4004 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251215 21:02:46.333652 4004 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251215 21:02:46.333823 4004 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251215 21:02:46.340651 4004 rpc_server.cc:307] RPC server started. Bound to: 127.28.113.60:34031
I20251215 21:02:46.340658 4149 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.28.113.60:34031 every 8 connection(s)
I20251215 21:02:46.341584 4004 server_base.cc:1179] Dumped server information to /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765832506145157-29124-0/minicluster-data/master-2/data/info.pb
I20251215 21:02:46.344282 4150 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 21:02:46.347916 4150 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.113.62" port: 41229 } has no permanent_uuid. Determining permanent_uuid...
I20251215 21:02:46.349042 29124 external_mini_cluster.cc:1428] Started /tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu as pid 4004
Dec 15 21:02:46 dist-test-slave-bjsf krb5kdc[3366](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765832566, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.113.60@KRBTEST.COM for oryx/127.28.113.62@KRBTEST.COM
Dec 15 21:02:46 dist-test-slave-bjsf krb5kdc[3366](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765832564, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.28.113.60@KRBTEST.COM
I20251215 21:02:46.359369 3453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:02:46.349919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42454 (local address 127.28.113.62:41229)
1215 21:02:46.350048 (+ 129us) server_negotiation.cc:207] Beginning negotiation
1215 21:02:46.350053 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1215 21:02:46.350769 (+ 716us) server_negotiation.cc:408] Connection header received
1215 21:02:46.351335 (+ 566us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1215 21:02:46.351339 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1215 21:02:46.351388 (+ 49us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1215 21:02:46.351483 (+ 95us) server_negotiation.cc:227] Negotiated authn=SASL
1215 21:02:46.352826 (+ 1343us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:02:46.353338 (+ 512us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1215 21:02:46.353939 (+ 601us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:02:46.354633 (+ 694us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:02:46.356161 (+ 1528us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1215 21:02:46.356184 (+ 23us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1215 21:02:46.356187 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1215 21:02:46.356219 (+ 32us) server_negotiation.cc:923] calling sasl_server_start()
1215 21:02:46.357608 (+ 1389us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1215 21:02:46.358117 (+ 509us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1215 21:02:46.358120 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1215 21:02:46.358122 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1215 21:02:46.358156 (+ 34us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1215 21:02:46.358413 (+ 257us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1215 21:02:46.358415 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1215 21:02:46.358416 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1215 21:02:46.358540 (+ 124us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1215 21:02:46.358622 (+ 82us) server_negotiation.cc:1036] Waiting for connection context
1215 21:02:46.359157 (+ 535us) server_negotiation.cc:300] Negotiation successful
1215 21:02:46.359246 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20251215 21:02:46.360268 4153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:02:46.350164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.113.62:41229 (local address 127.0.0.1:42454)
1215 21:02:46.350630 (+ 466us) negotiation.cc:107] Waiting for socket to connect
1215 21:02:46.350667 (+ 37us) client_negotiation.cc:174] Beginning negotiation
1215 21:02:46.351162 (+ 495us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1215 21:02:46.351918 (+ 756us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1215 21:02:46.351922 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
1215 21:02:46.352311 (+ 389us) client_negotiation.cc:189] Negotiated authn=SASL
1215 21:02:46.352680 (+ 369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:02:46.352692 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:02:46.353446 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1215 21:02:46.353449 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1215 21:02:46.353819 (+ 370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:02:46.353828 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:02:46.354024 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:02:46.354535 (+ 511us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1215 21:02:46.354557 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
1215 21:02:46.355947 (+ 1390us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1215 21:02:46.357714 (+ 1767us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1215 21:02:46.357721 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1215 21:02:46.357729 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
1215 21:02:46.358028 (+ 299us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1215 21:02:46.358230 (+ 202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1215 21:02:46.358233 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1215 21:02:46.358235 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1215 21:02:46.358336 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1215 21:02:46.358639 (+ 303us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1215 21:02:46.358645 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1215 21:02:46.358884 (+ 239us) client_negotiation.cc:769] Sending connection context
1215 21:02:46.359096 (+ 212us) client_negotiation.cc:240] Negotiation successful
1215 21:02:46.359354 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":300,"spinlock_wait_cycles":83712,"thread_start_us":155,"threads_started":1}
I20251215 21:02:46.360268 4152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:02:46.350209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57904 (local address 127.28.113.60:34031)
1215 21:02:46.350571 (+ 362us) server_negotiation.cc:207] Beginning negotiation
1215 21:02:46.350580 (+ 9us) server_negotiation.cc:400] Waiting for connection header
1215 21:02:46.350609 (+ 29us) server_negotiation.cc:408] Connection header received
1215 21:02:46.351143 (+ 534us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1215 21:02:46.351160 (+ 17us) server_negotiation.cc:462] Received NEGOTIATE request from client
1215 21:02:46.351348 (+ 188us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1215 21:02:46.351515 (+ 167us) server_negotiation.cc:227] Negotiated authn=SASL
1215 21:02:46.352310 (+ 795us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:02:46.353213 (+ 903us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1215 21:02:46.353850 (+ 637us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:02:46.354033 (+ 183us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:02:46.356534 (+ 2501us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1215 21:02:46.356555 (+ 21us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1215 21:02:46.356566 (+ 11us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1215 21:02:46.356592 (+ 26us) server_negotiation.cc:923] calling sasl_server_start()
1215 21:02:46.358510 (+ 1918us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1215 21:02:46.358864 (+ 354us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1215 21:02:46.358869 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1215 21:02:46.358870 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1215 21:02:46.358917 (+ 47us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1215 21:02:46.359109 (+ 192us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1215 21:02:46.359112 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1215 21:02:46.359114 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1215 21:02:46.359304 (+ 190us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1215 21:02:46.359705 (+ 401us) server_negotiation.cc:1036] Waiting for connection context
1215 21:02:46.359805 (+ 100us) server_negotiation.cc:300] Negotiation successful
1215 21:02:46.359923 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":256,"spinlock_wait_cycles":896,"thread_start_us":124,"threads_started":1}
I20251215 21:02:46.361445 4150 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.113.61" port: 39207 } has no permanent_uuid. Determining permanent_uuid...
Dec 15 21:02:46 dist-test-slave-bjsf krb5kdc[3366](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765832566, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.113.60@KRBTEST.COM for oryx/127.28.113.61@KRBTEST.COM
I20251215 21:02:46.368201 4153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:02:46.361876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.113.61:39207 (local address 127.0.0.1:55220)
1215 21:02:46.361989 (+ 113us) negotiation.cc:107] Waiting for socket to connect
1215 21:02:46.362001 (+ 12us) client_negotiation.cc:174] Beginning negotiation
1215 21:02:46.362061 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1215 21:02:46.362562 (+ 501us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1215 21:02:46.362565 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1215 21:02:46.362723 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
1215 21:02:46.362894 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:02:46.362900 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:02:46.363780 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1215 21:02:46.363783 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1215 21:02:46.364218 (+ 435us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:02:46.364226 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:02:46.364320 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:02:46.364706 (+ 386us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1215 21:02:46.364717 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
1215 21:02:46.365780 (+ 1063us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1215 21:02:46.367242 (+ 1462us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1215 21:02:46.367245 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1215 21:02:46.367247 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1215 21:02:46.367388 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1215 21:02:46.367554 (+ 166us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1215 21:02:46.367556 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1215 21:02:46.367558 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1215 21:02:46.367592 (+ 34us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1215 21:02:46.367834 (+ 242us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1215 21:02:46.367836 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1215 21:02:46.367899 (+ 63us) client_negotiation.cc:769] Sending connection context
1215 21:02:46.367984 (+ 85us) client_negotiation.cc:240] Negotiation successful
1215 21:02:46.368088 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44}
I20251215 21:02:46.368271 4154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:02:46.362117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55220 (local address 127.28.113.61:39207)
1215 21:02:46.362363 (+ 246us) server_negotiation.cc:207] Beginning negotiation
1215 21:02:46.362366 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1215 21:02:46.362378 (+ 12us) server_negotiation.cc:408] Connection header received
1215 21:02:46.362411 (+ 33us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1215 21:02:46.362413 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1215 21:02:46.362452 (+ 39us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1215 21:02:46.362529 (+ 77us) server_negotiation.cc:227] Negotiated authn=SASL
1215 21:02:46.363010 (+ 481us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:02:46.363674 (+ 664us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1215 21:02:46.364322 (+ 648us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:02:46.364526 (+ 204us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:02:46.365889 (+ 1363us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1215 21:02:46.365906 (+ 17us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1215 21:02:46.365908 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1215 21:02:46.365931 (+ 23us) server_negotiation.cc:923] calling sasl_server_start()
1215 21:02:46.367149 (+ 1218us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1215 21:02:46.367459 (+ 310us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1215 21:02:46.367461 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1215 21:02:46.367463 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1215 21:02:46.367496 (+ 33us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1215 21:02:46.367647 (+ 151us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1215 21:02:46.367649 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1215 21:02:46.367650 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1215 21:02:46.367774 (+ 124us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1215 21:02:46.367841 (+ 67us) server_negotiation.cc:1036] Waiting for connection context
1215 21:02:46.368043 (+ 202us) server_negotiation.cc:300] Negotiation successful
1215 21:02:46.368148 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":176,"thread_start_us":87,"threads_started":1}
I20251215 21:02:46.368876 4150 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.113.60" port: 34031 } has no permanent_uuid. Determining permanent_uuid...
Dec 15 21:02:46 dist-test-slave-bjsf krb5kdc[3366](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765832566, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.113.60@KRBTEST.COM for oryx/127.28.113.60@KRBTEST.COM
I20251215 21:02:46.374650 4153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:02:46.369288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.113.60:34031 (local address 127.0.0.1:57906)
1215 21:02:46.369377 (+ 89us) negotiation.cc:107] Waiting for socket to connect
1215 21:02:46.369390 (+ 13us) client_negotiation.cc:174] Beginning negotiation
1215 21:02:46.369450 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1215 21:02:46.369612 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1215 21:02:46.369614 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1215 21:02:46.369724 (+ 110us) client_negotiation.cc:189] Negotiated authn=SASL
1215 21:02:46.369876 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:02:46.369882 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:02:46.370445 (+ 563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1215 21:02:46.370447 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1215 21:02:46.370770 (+ 323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:02:46.370775 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:02:46.370853 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:02:46.371249 (+ 396us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1215 21:02:46.371261 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
1215 21:02:46.372332 (+ 1071us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1215 21:02:46.373719 (+ 1387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1215 21:02:46.373722 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1215 21:02:46.373723 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1215 21:02:46.373880 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1215 21:02:46.374047 (+ 167us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1215 21:02:46.374049 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1215 21:02:46.374050 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1215 21:02:46.374085 (+ 35us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1215 21:02:46.374311 (+ 226us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1215 21:02:46.374313 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1215 21:02:46.374373 (+ 60us) client_negotiation.cc:769] Sending connection context
1215 21:02:46.374441 (+ 68us) client_negotiation.cc:240] Negotiation successful
1215 21:02:46.374531 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23}
I20251215 21:02:46.374689 4152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:02:46.369351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57906 (local address 127.28.113.60:34031)
1215 21:02:46.369425 (+ 74us) server_negotiation.cc:207] Beginning negotiation
1215 21:02:46.369428 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1215 21:02:46.369438 (+ 10us) server_negotiation.cc:408] Connection header received
1215 21:02:46.369516 (+ 78us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1215 21:02:46.369519 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1215 21:02:46.369551 (+ 32us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1215 21:02:46.369606 (+ 55us) server_negotiation.cc:227] Negotiated authn=SASL
1215 21:02:46.369953 (+ 347us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:02:46.370370 (+ 417us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1215 21:02:46.370847 (+ 477us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:02:46.370969 (+ 122us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:02:46.372432 (+ 1463us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1215 21:02:46.372445 (+ 13us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1215 21:02:46.372448 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1215 21:02:46.372467 (+ 19us) server_negotiation.cc:923] calling sasl_server_start()
1215 21:02:46.373630 (+ 1163us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1215 21:02:46.373952 (+ 322us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1215 21:02:46.373954 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1215 21:02:46.373955 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1215 21:02:46.373987 (+ 32us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1215 21:02:46.374141 (+ 154us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1215 21:02:46.374143 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1215 21:02:46.374144 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1215 21:02:46.374254 (+ 110us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1215 21:02:46.374316 (+ 62us) server_negotiation.cc:1036] Waiting for connection context
1215 21:02:46.374480 (+ 164us) server_negotiation.cc:300] Negotiation successful
1215 21:02:46.374580 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":15,"server-negotiator.queue_time_us":22}
I20251215 21:02:46.377310 4150 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 297c0b4eec8b49459719bf6b7429a987: Bootstrap starting.
I20251215 21:02:46.379576 4150 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 297c0b4eec8b49459719bf6b7429a987: Neither blocks nor log segments found. Creating new log.
I20251215 21:02:46.380314 4150 log.cc:826] T 00000000000000000000000000000000 P 297c0b4eec8b49459719bf6b7429a987: Log is configured to *not* fsync() on all Append() calls
I20251215 21:02:46.381911 4150 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 297c0b4eec8b49459719bf6b7429a987: No bootstrap required, opened a new log
I20251215 21:02:46.384377 4150 raft_consensus.cc:359] T 00000000000000000000000000000000 P 297c0b4eec8b49459719bf6b7429a987 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c9267b46ebc4bae9a13928cf23fa8d3" member_type: VOTER last_known_addr { host: "127.28.113.62" port: 41229 } } peers { permanent_uuid: "de9d22b45210451f997093f481e4dfc6" member_type: VOTER last_known_addr { host: "127.28.113.61" port: 39207 } } peers { permanent_uuid: "297c0b4eec8b49459719bf6b7429a987" member_type: VOTER last_known_addr { host: "127.28.113.60" port: 34031 } }
I20251215 21:02:46.384557 4150 raft_consensus.cc:385] T 00000000000000000000000000000000 P 297c0b4eec8b49459719bf6b7429a987 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251215 21:02:46.384620 4150 raft_consensus.cc:740] T 00000000000000000000000000000000 P 297c0b4eec8b49459719bf6b7429a987 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 297c0b4eec8b49459719bf6b7429a987, State: Initialized, Role: FOLLOWER
I20251215 21:02:46.385043 4150 consensus_queue.cc:260] T 00000000000000000000000000000000 P 297c0b4eec8b49459719bf6b7429a987 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c9267b46ebc4bae9a13928cf23fa8d3" member_type: VOTER last_known_addr { host: "127.28.113.62" port: 41229 } } peers { permanent_uuid: "de9d22b45210451f997093f481e4dfc6" member_type: VOTER last_known_addr { host: "127.28.113.61" port: 39207 } } peers { permanent_uuid: "297c0b4eec8b49459719bf6b7429a987" member_type: VOTER last_known_addr { host: "127.28.113.60" port: 34031 } }
I20251215 21:02:46.385437 4156 sys_catalog.cc:455] T 00000000000000000000000000000000 P 297c0b4eec8b49459719bf6b7429a987 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c9267b46ebc4bae9a13928cf23fa8d3" member_type: VOTER last_known_addr { host: "127.28.113.62" port: 41229 } } peers { permanent_uuid: "de9d22b45210451f997093f481e4dfc6" member_type: VOTER last_known_addr { host: "127.28.113.61" port: 39207 } } peers { permanent_uuid: "297c0b4eec8b49459719bf6b7429a987" member_type: VOTER last_known_addr { host: "127.28.113.60" port: 34031 } } }
I20251215 21:02:46.385531 4156 sys_catalog.cc:458] T 00000000000000000000000000000000 P 297c0b4eec8b49459719bf6b7429a987 [sys.catalog]: This master's current role is: FOLLOWER
I20251215 21:02:46.386029 4150 sys_catalog.cc:565] T 00000000000000000000000000000000 P 297c0b4eec8b49459719bf6b7429a987 [sys.catalog]: configured and running, proceeding with master startup.
W20251215 21:02:46.390271 4167 catalog_manager.cc:1568] T 00000000000000000000000000000000 P 297c0b4eec8b49459719bf6b7429a987: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20251215 21:02:46.390353 4167 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20251215 21:02:46.393733 4004 master_runner.cc:428] Detected that this master 297c0b4eec8b49459719bf6b7429a987 is joining an existing cluster
I20251215 21:02:46.393810 4004 master_runner.cc:432] Initiating AddMaster RPC to add 127.28.113.60:34031
I20251215 21:02:46.412467 3607 heartbeater.cc:499] Master 127.28.113.62:41229 was elected leader, sending a full tablet report...
I20251215 21:02:46.417320 3979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:02:46.414359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.113.62:41229 (local address 127.0.0.1:42474)
1215 21:02:46.414499 (+ 140us) negotiation.cc:107] Waiting for socket to connect
1215 21:02:46.414515 (+ 16us) client_negotiation.cc:174] Beginning negotiation
1215 21:02:46.414594 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1215 21:02:46.414931 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1215 21:02:46.414934 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1215 21:02:46.414952 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1215 21:02:46.415197 (+ 245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:02:46.415205 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:02:46.415920 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1215 21:02:46.415924 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1215 21:02:46.416897 (+ 973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:02:46.416907 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:02:46.417027 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:02:46.417086 (+ 59us) client_negotiation.cc:769] Sending connection context
1215 21:02:46.417134 (+ 48us) client_negotiation.cc:240] Negotiation successful
1215 21:02:46.417194 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58}
I20251215 21:02:46.417567 4176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:02:46.414468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42474 (local address 127.28.113.62:41229)
1215 21:02:46.414726 (+ 258us) server_negotiation.cc:207] Beginning negotiation
1215 21:02:46.414729 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1215 21:02:46.414741 (+ 12us) server_negotiation.cc:408] Connection header received
1215 21:02:46.414781 (+ 40us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1215 21:02:46.414784 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1215 21:02:46.414830 (+ 46us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1215 21:02:46.414895 (+ 65us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1215 21:02:46.415301 (+ 406us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:02:46.415809 (+ 508us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1215 21:02:46.417000 (+ 1191us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:02:46.417302 (+ 302us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:02:46.417353 (+ 51us) server_negotiation.cc:1036] Waiting for connection context
1215 21:02:46.417393 (+ 40us) server_negotiation.cc:300] Negotiation successful
1215 21:02:46.417443 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":203,"thread_start_us":104,"threads_started":1}
I20251215 21:02:46.420955 3979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:02:46.418540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.113.61:39207 (local address 127.0.0.1:55240)
1215 21:02:46.418667 (+ 127us) negotiation.cc:107] Waiting for socket to connect
1215 21:02:46.418678 (+ 11us) client_negotiation.cc:174] Beginning negotiation
1215 21:02:46.418734 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1215 21:02:46.419172 (+ 438us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1215 21:02:46.419175 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1215 21:02:46.419192 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1215 21:02:46.419328 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:02:46.419333 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:02:46.420032 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1215 21:02:46.420036 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1215 21:02:46.420608 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:02:46.420616 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:02:46.420711 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:02:46.420730 (+ 19us) client_negotiation.cc:769] Sending connection context
1215 21:02:46.420776 (+ 46us) client_negotiation.cc:240] Negotiation successful
1215 21:02:46.420838 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":63}
I20251215 21:02:46.421460 4177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:02:46.418689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55240 (local address 127.28.113.61:39207)
1215 21:02:46.418946 (+ 257us) server_negotiation.cc:207] Beginning negotiation
1215 21:02:46.418949 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1215 21:02:46.418961 (+ 12us) server_negotiation.cc:408] Connection header received
1215 21:02:46.419000 (+ 39us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1215 21:02:46.419003 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1215 21:02:46.419047 (+ 44us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1215 21:02:46.419141 (+ 94us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1215 21:02:46.419429 (+ 288us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:02:46.419916 (+ 487us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1215 21:02:46.420717 (+ 801us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:02:46.421172 (+ 455us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:02:46.421279 (+ 107us) server_negotiation.cc:1036] Waiting for connection context
1215 21:02:46.421321 (+ 42us) server_negotiation.cc:300] Negotiation successful
1215 21:02:46.421368 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":191,"thread_start_us":101,"threads_started":1}
I20251215 21:02:46.431705 4175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:02:46.396159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.113.61:39207 (local address 127.0.0.1:55236)
1215 21:02:46.396897 (+ 738us) negotiation.cc:107] Waiting for socket to connect
1215 21:02:46.396912 (+ 15us) client_negotiation.cc:174] Beginning negotiation
1215 21:02:46.397025 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1215 21:02:46.397369 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1215 21:02:46.397372 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1215 21:02:46.424316 (+ 26944us) client_negotiation.cc:189] Negotiated authn=SASL
1215 21:02:46.424657 (+ 341us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:02:46.424666 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:02:46.426252 (+ 1586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1215 21:02:46.426255 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1215 21:02:46.426778 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:02:46.426787 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:02:46.426882 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:02:46.427390 (+ 508us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1215 21:02:46.427405 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
1215 21:02:46.427702 (+ 297us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1215 21:02:46.429908 (+ 2206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1215 21:02:46.429912 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1215 21:02:46.429915 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
1215 21:02:46.430236 (+ 321us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1215 21:02:46.430771 (+ 535us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1215 21:02:46.430774 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1215 21:02:46.430777 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
1215 21:02:46.430830 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1215 21:02:46.431209 (+ 379us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1215 21:02:46.431213 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1215 21:02:46.431290 (+ 77us) client_negotiation.cc:769] Sending connection context
1215 21:02:46.431407 (+ 117us) client_negotiation.cc:240] Negotiation successful
1215 21:02:46.431532 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":680,"thread_start_us":106,"threads_started":1}
I20251215 21:02:46.432469 4154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:02:46.396040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55236 (local address 127.28.113.61:39207)
1215 21:02:46.396175 (+ 135us) server_negotiation.cc:207] Beginning negotiation
1215 21:02:46.396179 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1215 21:02:46.396979 (+ 800us) server_negotiation.cc:408] Connection header received
1215 21:02:46.397133 (+ 154us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1215 21:02:46.397137 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1215 21:02:46.397186 (+ 49us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1215 21:02:46.397237 (+ 51us) server_negotiation.cc:227] Negotiated authn=SASL
1215 21:02:46.424832 (+ 27595us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:02:46.425549 (+ 717us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1215 21:02:46.427068 (+ 1519us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:02:46.427255 (+ 187us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:02:46.427820 (+ 565us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1215 21:02:46.427839 (+ 19us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1215 21:02:46.427842 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1215 21:02:46.427867 (+ 25us) server_negotiation.cc:923] calling sasl_server_start()
1215 21:02:46.429793 (+ 1926us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1215 21:02:46.430349 (+ 556us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1215 21:02:46.430352 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1215 21:02:46.430355 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
1215 21:02:46.430669 (+ 314us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1215 21:02:46.430916 (+ 247us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1215 21:02:46.430919 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1215 21:02:46.430921 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1215 21:02:46.431090 (+ 169us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1215 21:02:46.431170 (+ 80us) server_negotiation.cc:1036] Waiting for connection context
1215 21:02:46.432208 (+ 1038us) server_negotiation.cc:300] Negotiation successful
1215 21:02:46.432320 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20251215 21:02:46.432781 4174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:02:46.396374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.113.60:34031 (local address 127.0.0.1:57910)
1215 21:02:46.397233 (+ 859us) negotiation.cc:107] Waiting for socket to connect
1215 21:02:46.397245 (+ 12us) client_negotiation.cc:174] Beginning negotiation
1215 21:02:46.397334 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1215 21:02:46.397591 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1215 21:02:46.397593 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1215 21:02:46.424828 (+ 27235us) client_negotiation.cc:189] Negotiated authn=SASL
1215 21:02:46.425094 (+ 266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:02:46.425101 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:02:46.426056 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1215 21:02:46.426059 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1215 21:02:46.426945 (+ 886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:02:46.426955 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:02:46.427906 (+ 951us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:02:46.428665 (+ 759us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1215 21:02:46.428679 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
1215 21:02:46.428975 (+ 296us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1215 21:02:46.431521 (+ 2546us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1215 21:02:46.431524 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1215 21:02:46.431526 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1215 21:02:46.431708 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1215 21:02:46.431937 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1215 21:02:46.431940 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1215 21:02:46.431942 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1215 21:02:46.432009 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1215 21:02:46.432346 (+ 337us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1215 21:02:46.432348 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1215 21:02:46.432421 (+ 73us) client_negotiation.cc:769] Sending connection context
1215 21:02:46.432516 (+ 95us) client_negotiation.cc:240] Negotiation successful
1215 21:02:46.432625 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":798,"thread_start_us":234,"threads_started":1}
I20251215 21:02:46.433192 4152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:02:46.396395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57910 (local address 127.28.113.60:34031)
1215 21:02:46.396483 (+ 88us) server_negotiation.cc:207] Beginning negotiation
1215 21:02:46.396487 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1215 21:02:46.397292 (+ 805us) server_negotiation.cc:408] Connection header received
1215 21:02:46.397410 (+ 118us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1215 21:02:46.397413 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1215 21:02:46.397444 (+ 31us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1215 21:02:46.397485 (+ 41us) server_negotiation.cc:227] Negotiated authn=SASL
1215 21:02:46.425211 (+ 27726us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:02:46.425899 (+ 688us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1215 21:02:46.428070 (+ 2171us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:02:46.428230 (+ 160us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:02:46.429084 (+ 854us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1215 21:02:46.429098 (+ 14us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1215 21:02:46.429102 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1215 21:02:46.429123 (+ 21us) server_negotiation.cc:923] calling sasl_server_start()
1215 21:02:46.431424 (+ 2301us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1215 21:02:46.431799 (+ 375us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1215 21:02:46.431802 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1215 21:02:46.431804 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1215 21:02:46.431858 (+ 54us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1215 21:02:46.432090 (+ 232us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1215 21:02:46.432093 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1215 21:02:46.432094 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1215 21:02:46.432235 (+ 141us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1215 21:02:46.432314 (+ 79us) server_negotiation.cc:1036] Waiting for connection context
1215 21:02:46.432979 (+ 665us) server_negotiation.cc:300] Negotiation successful
1215 21:02:46.433081 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20251215 21:02:46.434154 4173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:02:46.395915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.113.62:41229 (local address 127.0.0.1:42470)
1215 21:02:46.396897 (+ 982us) negotiation.cc:107] Waiting for socket to connect
1215 21:02:46.396912 (+ 15us) client_negotiation.cc:174] Beginning negotiation
1215 21:02:46.397015 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1215 21:02:46.397367 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1215 21:02:46.397370 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1215 21:02:46.424336 (+ 26966us) client_negotiation.cc:189] Negotiated authn=SASL
1215 21:02:46.424619 (+ 283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:02:46.424629 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:02:46.425583 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1215 21:02:46.425587 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1215 21:02:46.426102 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:02:46.426112 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:02:46.426206 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:02:46.427028 (+ 822us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1215 21:02:46.427044 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
1215 21:02:46.427454 (+ 410us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1215 21:02:46.429988 (+ 2534us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1215 21:02:46.429991 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1215 21:02:46.429993 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1215 21:02:46.430300 (+ 307us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1215 21:02:46.430561 (+ 261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1215 21:02:46.430565 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1215 21:02:46.430566 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1215 21:02:46.430617 (+ 51us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1215 21:02:46.433732 (+ 3115us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1215 21:02:46.433735 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1215 21:02:46.433806 (+ 71us) client_negotiation.cc:769] Sending connection context
1215 21:02:46.433896 (+ 90us) client_negotiation.cc:240] Negotiation successful
1215 21:02:46.434006 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":887,"thread_start_us":140,"threads_started":1}
I20251215 21:02:46.434193 3453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:02:46.396036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42470 (local address 127.28.113.62:41229)
1215 21:02:46.396280 (+ 244us) server_negotiation.cc:207] Beginning negotiation
1215 21:02:46.396285 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1215 21:02:46.397006 (+ 721us) server_negotiation.cc:408] Connection header received
1215 21:02:46.397109 (+ 103us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1215 21:02:46.397113 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1215 21:02:46.397171 (+ 58us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1215 21:02:46.397251 (+ 80us) server_negotiation.cc:227] Negotiated authn=SASL
1215 21:02:46.424742 (+ 27491us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:02:46.425462 (+ 720us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1215 21:02:46.426429 (+ 967us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:02:46.426605 (+ 176us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:02:46.427582 (+ 977us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1215 21:02:46.427597 (+ 15us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1215 21:02:46.427601 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1215 21:02:46.427624 (+ 23us) server_negotiation.cc:923] calling sasl_server_start()
1215 21:02:46.429884 (+ 2260us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1215 21:02:46.430413 (+ 529us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1215 21:02:46.430416 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1215 21:02:46.430417 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1215 21:02:46.430457 (+ 40us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1215 21:02:46.433464 (+ 3007us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1215 21:02:46.433467 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1215 21:02:46.433469 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1215 21:02:46.433633 (+ 164us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1215 21:02:46.433717 (+ 84us) server_negotiation.cc:1036] Waiting for connection context
1215 21:02:46.433967 (+ 250us) server_negotiation.cc:300] Negotiation successful
1215 21:02:46.434062 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":163}
Dec 15 21:02:46 dist-test-slave-bjsf krb5kdc[3366](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765832565, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.113.62@KRBTEST.COM for oryx/127.28.113.60@KRBTEST.COM
I20251215 21:02:46.444190 3979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:02:46.437089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.113.60:34031 (local address 127.0.0.1:57926)
1215 21:02:46.437188 (+ 99us) negotiation.cc:107] Waiting for socket to connect
1215 21:02:46.437202 (+ 14us) client_negotiation.cc:174] Beginning negotiation
1215 21:02:46.437272 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1215 21:02:46.437499 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1215 21:02:46.437502 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1215 21:02:46.437664 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
1215 21:02:46.437871 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:02:46.437879 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:02:46.438783 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1215 21:02:46.438787 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1215 21:02:46.439324 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:02:46.439333 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:02:46.439437 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:02:46.439996 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1215 21:02:46.440014 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
1215 21:02:46.441232 (+ 1218us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1215 21:02:46.443098 (+ 1866us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1215 21:02:46.443101 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1215 21:02:46.443103 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1215 21:02:46.443248 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1215 21:02:46.443456 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1215 21:02:46.443458 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1215 21:02:46.443459 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1215 21:02:46.443495 (+ 36us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1215 21:02:46.443801 (+ 306us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1215 21:02:46.443802 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1215 21:02:46.443876 (+ 74us) client_negotiation.cc:769] Sending connection context
1215 21:02:46.443945 (+ 69us) client_negotiation.cc:240] Negotiation successful
1215 21:02:46.444078 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20251215 21:02:46.444289 4152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:02:46.437167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57926 (local address 127.28.113.60:34031)
1215 21:02:46.437284 (+ 117us) server_negotiation.cc:207] Beginning negotiation
1215 21:02:46.437288 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1215 21:02:46.437300 (+ 12us) server_negotiation.cc:408] Connection header received
1215 21:02:46.437367 (+ 67us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1215 21:02:46.437370 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1215 21:02:46.437410 (+ 40us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1215 21:02:46.437469 (+ 59us) server_negotiation.cc:227] Negotiated authn=SASL
1215 21:02:46.437984 (+ 515us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:02:46.438657 (+ 673us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1215 21:02:46.439437 (+ 780us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:02:46.439607 (+ 170us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:02:46.441346 (+ 1739us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1215 21:02:46.441362 (+ 16us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1215 21:02:46.441366 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1215 21:02:46.441391 (+ 25us) server_negotiation.cc:923] calling sasl_server_start()
1215 21:02:46.442994 (+ 1603us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1215 21:02:46.443336 (+ 342us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1215 21:02:46.443339 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1215 21:02:46.443341 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1215 21:02:46.443385 (+ 44us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1215 21:02:46.443564 (+ 179us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1215 21:02:46.443567 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1215 21:02:46.443569 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1215 21:02:46.443720 (+ 151us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1215 21:02:46.443802 (+ 82us) server_negotiation.cc:1036] Waiting for connection context
1215 21:02:46.444042 (+ 240us) server_negotiation.cc:300] Negotiation successful
1215 21:02:46.444147 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20251215 21:02:46.445042 3414 catalog_manager.cc:7141] Initiating ChangeConfig request to add master 127.28.113.60:34031: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "297c0b4eec8b49459719bf6b7429a987"
member_type: NON_VOTER
last_known_addr {
host: "127.28.113.60"
port: 34031
}
attrs {
promote: true
}
}
dest_uuid: "8c9267b46ebc4bae9a13928cf23fa8d3"
cas_config_opid_index: 5
I20251215 21:02:46.445636 3414 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8c9267b46ebc4bae9a13928cf23fa8d3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "8c9267b46ebc4bae9a13928cf23fa8d3" member_type: VOTER last_known_addr { host: "127.28.113.62" port: 41229 } } peers { permanent_uuid: "de9d22b45210451f997093f481e4dfc6" member_type: NON_VOTER last_known_addr { host: "127.28.113.61" port: 39207 } attrs { promote: true } } peers { permanent_uuid: "297c0b4eec8b49459719bf6b7429a987" member_type: NON_VOTER last_known_addr { host: "127.28.113.60" port: 34031 } attrs { promote: true } }
I20251215 21:02:46.446609 3984 raft_consensus.cc:2955] T 00000000000000000000000000000000 P 8c9267b46ebc4bae9a13928cf23fa8d3 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 297c0b4eec8b49459719bf6b7429a987 (127.28.113.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "8c9267b46ebc4bae9a13928cf23fa8d3" member_type: VOTER last_known_addr { host: "127.28.113.62" port: 41229 } } peers { permanent_uuid: "de9d22b45210451f997093f481e4dfc6" member_type: NON_VOTER last_known_addr { host: "127.28.113.61" port: 39207 } attrs { promote: true } } peers { permanent_uuid: "297c0b4eec8b49459719bf6b7429a987" member_type: NON_VOTER last_known_addr { host: "127.28.113.60" port: 34031 } attrs { promote: true } } }
I20251215 21:02:46.447116 4124 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 297c0b4eec8b49459719bf6b7429a987 [term 0 FOLLOWER]: Advancing to term 1
I20251215 21:02:46.448009 3984 catalog_manager.cc:7145] Successfully completed master ChangeConfig request to add master 127.28.113.60:34031
I20251215 21:02:46.448165 3984 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c9267b46ebc4bae9a13928cf23fa8d3 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "8c9267b46ebc4bae9a13928cf23fa8d3" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "8c9267b46ebc4bae9a13928cf23fa8d3" member_type: VOTER last_known_addr { host: "127.28.113.62" port: 41229 } } peers { permanent_uuid: "de9d22b45210451f997093f481e4dfc6" member_type: NON_VOTER last_known_addr { host: "127.28.113.61" port: 39207 } attrs { promote: true } } peers { permanent_uuid: "297c0b4eec8b49459719bf6b7429a987" member_type: NON_VOTER last_known_addr { host: "127.28.113.60" port: 34031 } attrs { promote: true } } }
I20251215 21:02:46.448238 3984 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c9267b46ebc4bae9a13928cf23fa8d3 [sys.catalog]: This master's current role is: LEADER
I20251215 21:02:46.448899 4042 raft_consensus.cc:1275] T 00000000000000000000000000000000 P de9d22b45210451f997093f481e4dfc6 [term 1 LEARNER]: Refusing update from remote peer 8c9267b46ebc4bae9a13928cf23fa8d3: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20251215 21:02:46.448997 4124 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 297c0b4eec8b49459719bf6b7429a987 [term 1 FOLLOWER]: Refusing update from remote peer 8c9267b46ebc4bae9a13928cf23fa8d3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20251215 21:02:46.449249 4004 master.cc:561] Master@127.28.113.60:34031 shutting down...
I20251215 21:02:46.449383 3982 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 8c9267b46ebc4bae9a13928cf23fa8d3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "de9d22b45210451f997093f481e4dfc6" member_type: NON_VOTER last_known_addr { host: "127.28.113.61" port: 39207 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20251215 21:02:46.449595 3981 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 8c9267b46ebc4bae9a13928cf23fa8d3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "297c0b4eec8b49459719bf6b7429a987" member_type: NON_VOTER last_known_addr { host: "127.28.113.60" port: 34031 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
I20251215 21:02:46.450459 4042 raft_consensus.cc:2955] T 00000000000000000000000000000000 P de9d22b45210451f997093f481e4dfc6 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 297c0b4eec8b49459719bf6b7429a987 (127.28.113.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "8c9267b46ebc4bae9a13928cf23fa8d3" member_type: VOTER last_known_addr { host: "127.28.113.62" port: 41229 } } peers { permanent_uuid: "de9d22b45210451f997093f481e4dfc6" member_type: NON_VOTER last_known_addr { host: "127.28.113.61" port: 39207 } attrs { promote: true } } peers { permanent_uuid: "297c0b4eec8b49459719bf6b7429a987" member_type: NON_VOTER last_known_addr { host: "127.28.113.60" port: 34031 } attrs { promote: true } } }
W20251215 21:02:46.450920 3400 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20251215 21:02:46.451714 3400 consensus_peers.cc:597] T 00000000000000000000000000000000 P 8c9267b46ebc4bae9a13928cf23fa8d3 -> Peer 297c0b4eec8b49459719bf6b7429a987 (127.28.113.60:34031): Couldn't send request to peer 297c0b4eec8b49459719bf6b7429a987. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20251215 21:02:46.451931 4071 sys_catalog.cc:455] T 00000000000000000000000000000000 P de9d22b45210451f997093f481e4dfc6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8c9267b46ebc4bae9a13928cf23fa8d3. Latest consensus state: current_term: 1 leader_uuid: "8c9267b46ebc4bae9a13928cf23fa8d3" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "8c9267b46ebc4bae9a13928cf23fa8d3" member_type: VOTER last_known_addr { host: "127.28.113.62" port: 41229 } } peers { permanent_uuid: "de9d22b45210451f997093f481e4dfc6" member_type: NON_VOTER last_known_addr { host: "127.28.113.61" port: 39207 } attrs { promote: true } } peers { permanent_uuid: "297c0b4eec8b49459719bf6b7429a987" member_type: NON_VOTER last_known_addr { host: "127.28.113.60" port: 34031 } attrs { promote: true } } }
I20251215 21:02:46.452052 4071 sys_catalog.cc:458] T 00000000000000000000000000000000 P de9d22b45210451f997093f481e4dfc6 [sys.catalog]: This master's current role is: LEARNER
I20251215 21:02:46.452126 4071 sys_catalog.cc:455] T 00000000000000000000000000000000 P de9d22b45210451f997093f481e4dfc6 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8c9267b46ebc4bae9a13928cf23fa8d3" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "8c9267b46ebc4bae9a13928cf23fa8d3" member_type: VOTER last_known_addr { host: "127.28.113.62" port: 41229 } } peers { permanent_uuid: "de9d22b45210451f997093f481e4dfc6" member_type: NON_VOTER last_known_addr { host: "127.28.113.61" port: 39207 } attrs { promote: true } } peers { permanent_uuid: "297c0b4eec8b49459719bf6b7429a987" member_type: NON_VOTER last_known_addr { host: "127.28.113.60" port: 34031 } attrs { promote: true } } }
I20251215 21:02:46.452178 4071 sys_catalog.cc:458] T 00000000000000000000000000000000 P de9d22b45210451f997093f481e4dfc6 [sys.catalog]: This master's current role is: LEARNER
I20251215 21:02:46.452294 3981 raft_consensus.cc:1064] T 00000000000000000000000000000000 P 8c9267b46ebc4bae9a13928cf23fa8d3: attempting to promote NON_VOTER de9d22b45210451f997093f481e4dfc6 to VOTER
I20251215 21:02:46.452606 3981 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8c9267b46ebc4bae9a13928cf23fa8d3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "8c9267b46ebc4bae9a13928cf23fa8d3" member_type: VOTER last_known_addr { host: "127.28.113.62" port: 41229 } } peers { permanent_uuid: "de9d22b45210451f997093f481e4dfc6" member_type: VOTER last_known_addr { host: "127.28.113.61" port: 39207 } attrs { promote: false } } peers { permanent_uuid: "297c0b4eec8b49459719bf6b7429a987" member_type: NON_VOTER last_known_addr { host: "127.28.113.60" port: 34031 } attrs { promote: true } }
I20251215 21:02:46.454895 4042 raft_consensus.cc:1275] T 00000000000000000000000000000000 P de9d22b45210451f997093f481e4dfc6 [term 1 LEARNER]: Refusing update from remote peer 8c9267b46ebc4bae9a13928cf23fa8d3: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20251215 21:02:46.455137 3414 master.cc:679] Remote error: unable to get registration information for peer 297c0b4eec8b49459719bf6b7429a987: Service unavailable: service kudu.master.MasterService not registered on Master
I20251215 21:02:46.455241 3982 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 8c9267b46ebc4bae9a13928cf23fa8d3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "de9d22b45210451f997093f481e4dfc6" member_type: VOTER last_known_addr { host: "127.28.113.61" port: 39207 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20251215 21:02:46.455432 4004 raft_consensus.cc:2243] T 00000000000000000000000000000000 P 297c0b4eec8b49459719bf6b7429a987 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251215 21:02:46.455689 4004 raft_consensus.cc:2272] T 00000000000000000000000000000000 P 297c0b4eec8b49459719bf6b7429a987 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251215 21:02:46.455797 4004 tablet_replica.cc:333] T 00000000000000000000000000000000 P 297c0b4eec8b49459719bf6b7429a987: stopping tablet replica
W20251215 21:02:46.456656 3400 consensus_peers.cc:597] T 00000000000000000000000000000000 P 8c9267b46ebc4bae9a13928cf23fa8d3 -> Peer 297c0b4eec8b49459719bf6b7429a987 (127.28.113.60:34031): Couldn't send request to peer 297c0b4eec8b49459719bf6b7429a987. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20251215 21:02:46.458367 3982 raft_consensus.cc:2955] T 00000000000000000000000000000000 P 8c9267b46ebc4bae9a13928cf23fa8d3 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, de9d22b45210451f997093f481e4dfc6 (127.28.113.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "8c9267b46ebc4bae9a13928cf23fa8d3" member_type: VOTER last_known_addr { host: "127.28.113.62" port: 41229 } } peers { permanent_uuid: "de9d22b45210451f997093f481e4dfc6" member_type: VOTER last_known_addr { host: "127.28.113.61" port: 39207 } attrs { promote: false } } peers { permanent_uuid: "297c0b4eec8b49459719bf6b7429a987" member_type: NON_VOTER last_known_addr { host: "127.28.113.60" port: 34031 } attrs { promote: true } } }
I20251215 21:02:46.459535 3982 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c9267b46ebc4bae9a13928cf23fa8d3 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "8c9267b46ebc4bae9a13928cf23fa8d3" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "8c9267b46ebc4bae9a13928cf23fa8d3" member_type: VOTER last_known_addr { host: "127.28.113.62" port: 41229 } } peers { permanent_uuid: "de9d22b45210451f997093f481e4dfc6" member_type: VOTER last_known_addr { host: "127.28.113.61" port: 39207 } attrs { promote: false } } peers { permanent_uuid: "297c0b4eec8b49459719bf6b7429a987" member_type: NON_VOTER last_known_addr { host: "127.28.113.60" port: 34031 } attrs { promote: true } } }
I20251215 21:02:46.459623 3982 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c9267b46ebc4bae9a13928cf23fa8d3 [sys.catalog]: This master's current role is: LEADER
I20251215 21:02:46.459741 3982 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c9267b46ebc4bae9a13928cf23fa8d3 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8c9267b46ebc4bae9a13928cf23fa8d3" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "8c9267b46ebc4bae9a13928cf23fa8d3" member_type: VOTER last_known_addr { host: "127.28.113.62" port: 41229 } } peers { permanent_uuid: "de9d22b45210451f997093f481e4dfc6" member_type: VOTER last_known_addr { host: "127.28.113.61" port: 39207 } attrs { promote: false } } peers { permanent_uuid: "297c0b4eec8b49459719bf6b7429a987" member_type: NON_VOTER last_known_addr { host: "127.28.113.60" port: 34031 } attrs { promote: true } } }
I20251215 21:02:46.459806 3982 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c9267b46ebc4bae9a13928cf23fa8d3 [sys.catalog]: This master's current role is: LEADER
I20251215 21:02:46.460258 4042 raft_consensus.cc:2955] T 00000000000000000000000000000000 P de9d22b45210451f997093f481e4dfc6 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, de9d22b45210451f997093f481e4dfc6 (127.28.113.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "8c9267b46ebc4bae9a13928cf23fa8d3" member_type: VOTER last_known_addr { host: "127.28.113.62" port: 41229 } } peers { permanent_uuid: "de9d22b45210451f997093f481e4dfc6" member_type: VOTER last_known_addr { host: "127.28.113.61" port: 39207 } attrs { promote: false } } peers { permanent_uuid: "297c0b4eec8b49459719bf6b7429a987" member_type: NON_VOTER last_known_addr { host: "127.28.113.60" port: 34031 } attrs { promote: true } } }
I20251215 21:02:46.460839 4004 master.cc:583] Master@127.28.113.60:34031 shutdown complete.
I20251215 21:02:46.460883 4004 master_runner.cc:305] Clearing existing system tablet
I20251215 21:02:46.461076 4004 fs_manager.cc:362] Metadata directory not provided
I20251215 21:02:46.461133 4004 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251215 21:02:46.461829 4180 sys_catalog.cc:455] T 00000000000000000000000000000000 P de9d22b45210451f997093f481e4dfc6 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8c9267b46ebc4bae9a13928cf23fa8d3" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "8c9267b46ebc4bae9a13928cf23fa8d3" member_type: VOTER last_known_addr { host: "127.28.113.62" port: 41229 } } peers { permanent_uuid: "de9d22b45210451f997093f481e4dfc6" member_type: VOTER last_known_addr { host: "127.28.113.61" port: 39207 } attrs { promote: false } } peers { permanent_uuid: "297c0b4eec8b49459719bf6b7429a987" member_type: NON_VOTER last_known_addr { host: "127.28.113.60" port: 34031 } attrs { promote: true } } }
I20251215 21:02:46.461930 4180 sys_catalog.cc:458] T 00000000000000000000000000000000 P de9d22b45210451f997093f481e4dfc6 [sys.catalog]: This master's current role is: FOLLOWER
I20251215 21:02:46.462337 4004 fs_manager.cc:714] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20251215 21:02:46.463042 4182 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251215 21:02:46.463232 4004 fs_report.cc:389] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765832506145157-29124-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251215 21:02:46.463335 4004 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20251215 21:02:46.463366 4004 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765832506145157-29124-0/minicluster-data/master-2/data,/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765832506145157-29124-0/minicluster-data/master-2/wal
uuid: "297c0b4eec8b49459719bf6b7429a987"
format_stamp: "Formatted at 2025-12-15 21:02:46 on dist-test-slave-bjsf"
server_key: "7458bcd41abf86350ff9232b33ef2320"
server_key_iv: "60278cc87f7c26862b5bcafc0aeb2694"
server_key_version: "encryptionkey@0"
I20251215 21:02:46.464792 4004 ts_tablet_manager.cc:1916] T 00000000000000000000000000000000 P 297c0b4eec8b49459719bf6b7429a987: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251215 21:02:46.466295 4004 ts_tablet_manager.cc:1929] T 00000000000000000000000000000000 P 297c0b4eec8b49459719bf6b7429a987: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20251215 21:02:46.466357 4004 log.cc:1199] T 00000000000000000000000000000000 P 297c0b4eec8b49459719bf6b7429a987: Deleting WAL directory at /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765832506145157-29124-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20251215 21:02:46.466531 4004 ts_tablet_manager.cc:1950] T 00000000000000000000000000000000 P 297c0b4eec8b49459719bf6b7429a987: Deleting consensus metadata
I20251215 21:02:46.466658 4004 master_runner.cc:315] Copying system tablet from 127.28.113.62:41229
I20251215 21:02:46.467597 4004 tablet_copy_client.cc:323] T 00000000000000000000000000000000 P 297c0b4eec8b49459719bf6b7429a987: tablet copy: Beginning tablet copy session from remote peer at address 127.28.113.62:41229
I20251215 21:02:46.473816 3453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:02:46.468099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42480 (local address 127.28.113.62:41229)
1215 21:02:46.468226 (+ 127us) server_negotiation.cc:207] Beginning negotiation
1215 21:02:46.468231 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1215 21:02:46.468417 (+ 186us) server_negotiation.cc:408] Connection header received
1215 21:02:46.468512 (+ 95us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1215 21:02:46.468516 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1215 21:02:46.468561 (+ 45us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1215 21:02:46.468621 (+ 60us) server_negotiation.cc:227] Negotiated authn=SASL
1215 21:02:46.469143 (+ 522us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:02:46.469622 (+ 479us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1215 21:02:46.470209 (+ 587us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:02:46.470674 (+ 465us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:02:46.471133 (+ 459us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1215 21:02:46.471148 (+ 15us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1215 21:02:46.471150 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1215 21:02:46.471171 (+ 21us) server_negotiation.cc:923] calling sasl_server_start()
1215 21:02:46.472519 (+ 1348us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1215 21:02:46.472875 (+ 356us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1215 21:02:46.472880 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1215 21:02:46.472882 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1215 21:02:46.472924 (+ 42us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1215 21:02:46.473127 (+ 203us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1215 21:02:46.473130 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1215 21:02:46.473132 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1215 21:02:46.473265 (+ 133us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1215 21:02:46.473352 (+ 87us) server_negotiation.cc:1036] Waiting for connection context
1215 21:02:46.473591 (+ 239us) server_negotiation.cc:300] Negotiation successful
1215 21:02:46.473690 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20251215 21:02:46.473835 4188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:02:46.468029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.113.62:41229 (local address 127.0.0.1:42480)
1215 21:02:46.468335 (+ 306us) negotiation.cc:107] Waiting for socket to connect
1215 21:02:46.468346 (+ 11us) client_negotiation.cc:174] Beginning negotiation
1215 21:02:46.468439 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1215 21:02:46.468650 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1215 21:02:46.468653 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1215 21:02:46.468845 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
1215 21:02:46.469030 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:02:46.469036 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:02:46.469726 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1215 21:02:46.469729 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1215 21:02:46.470103 (+ 374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:02:46.470110 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:02:46.470204 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:02:46.470704 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1215 21:02:46.470719 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
1215 21:02:46.471007 (+ 288us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1215 21:02:46.472624 (+ 1617us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1215 21:02:46.472628 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1215 21:02:46.472630 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1215 21:02:46.472784 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1215 21:02:46.473002 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1215 21:02:46.473005 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1215 21:02:46.473007 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1215 21:02:46.473048 (+ 41us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1215 21:02:46.473353 (+ 305us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1215 21:02:46.473356 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1215 21:02:46.473438 (+ 82us) client_negotiation.cc:769] Sending connection context
1215 21:02:46.473525 (+ 87us) client_negotiation.cc:240] Negotiation successful
1215 21:02:46.473650 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":79,"threads_started":1}
I20251215 21:02:46.474144 3434 tablet_copy_service.cc:140] P 8c9267b46ebc4bae9a13928cf23fa8d3: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 297c0b4eec8b49459719bf6b7429a987 ({username='oryx', principal='oryx/127.28.113.60@KRBTEST.COM'} at 127.0.0.1:42480)
I20251215 21:02:46.474206 3434 tablet_copy_service.cc:161] P 8c9267b46ebc4bae9a13928cf23fa8d3: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 297c0b4eec8b49459719bf6b7429a987 at {username='oryx', principal='oryx/127.28.113.60@KRBTEST.COM'} at 127.0.0.1:42480: session id = 297c0b4eec8b49459719bf6b7429a987-00000000000000000000000000000000
I20251215 21:02:46.475093 3434 tablet_copy_source_session.cc:215] T 00000000000000000000000000000000 P 8c9267b46ebc4bae9a13928cf23fa8d3: Tablet Copy: opened 0 blocks and 1 log segments
I20251215 21:02:46.476277 3979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:02:46.475981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.113.60:34031 (local address 127.0.0.1:57928)
1215 21:02:46.476144 (+ 163us) negotiation.cc:107] Waiting for socket to connect
1215 21:02:46.476206 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.113.60:34031: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20251215 21:02:46.476378 4004 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
W20251215 21:02:46.476483 3414 master.cc:679] Network error: unable to get registration information for peer 297c0b4eec8b49459719bf6b7429a987: Client connection negotiation failed: client connection to 127.28.113.60:34031: connect: Connection refused (error 111)
I20251215 21:02:46.477995 4004 tablet_copy_client.cc:806] T 00000000000000000000000000000000 P 297c0b4eec8b49459719bf6b7429a987: tablet copy: Starting download of 0 data blocks...
I20251215 21:02:46.478137 4004 tablet_copy_client.cc:670] T 00000000000000000000000000000000 P 297c0b4eec8b49459719bf6b7429a987: tablet copy: Starting download of 1 WAL segments...
I20251215 21:02:46.479629 4004 tablet_copy_client.cc:538] T 00000000000000000000000000000000 P 297c0b4eec8b49459719bf6b7429a987: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20251215 21:02:46.480981 3434 tablet_copy_service.cc:342] P 8c9267b46ebc4bae9a13928cf23fa8d3: Request end of tablet copy session 297c0b4eec8b49459719bf6b7429a987-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.28.113.60@KRBTEST.COM'} at 127.0.0.1:42480
I20251215 21:02:46.481073 3434 tablet_copy_service.cc:434] P 8c9267b46ebc4bae9a13928cf23fa8d3: ending tablet copy session 297c0b4eec8b49459719bf6b7429a987-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 297c0b4eec8b49459719bf6b7429a987
W20251215 21:02:46.484719 4004 builtin_ntp.cc:688] coult not shutdown socket: Network error: shutdown error: Transport endpoint is not connected (error 107)
I20251215 21:02:46.486624 4004 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251215 21:02:46.488111 4192 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251215 21:02:46.488124 4190 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251215 21:02:46.488124 4189 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251215 21:02:46.488266 4004 server_base.cc:1047] running on GCE node
I20251215 21:02:46.488533 4004 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251215 21:02:46.488788 4004 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251215 21:02:46.489918 4004 hybrid_clock.cc:648] HybridClock initialized: now 1765832566489906 us; error 29 us; skew 500 ppm
Dec 15 21:02:46 dist-test-slave-bjsf krb5kdc[3366](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765832566, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.113.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251215 21:02:46.491284 4004 init.cc:377] Logged in from keytab as oryx/127.28.113.60@KRBTEST.COM (short username oryx)
I20251215 21:02:46.491695 4004 webserver.cc:492] Webserver started at http://127.28.113.60:34311/ using document root <none> and password file <none>
I20251215 21:02:46.491855 4004 fs_manager.cc:362] Metadata directory not provided
I20251215 21:02:46.491887 4004 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251215 21:02:46.492861 4004 fs_manager.cc:714] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20251215 21:02:46.493708 4199 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251215 21:02:46.493908 4004 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20251215 21:02:46.493970 4004 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765832506145157-29124-0/minicluster-data/master-2/data,/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765832506145157-29124-0/minicluster-data/master-2/wal
uuid: "297c0b4eec8b49459719bf6b7429a987"
format_stamp: "Formatted at 2025-12-15 21:02:46 on dist-test-slave-bjsf"
server_key: "7458bcd41abf86350ff9232b33ef2320"
server_key_iv: "60278cc87f7c26862b5bcafc0aeb2694"
server_key_version: "encryptionkey@0"
I20251215 21:02:46.494040 4004 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765832506145157-29124-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765832506145157-29124-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765832506145157-29124-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251215 21:02:46.510442 4004 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251215 21:02:46.512326 4004 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251215 21:02:46.512854 3979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:02:46.512551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.113.60:34031 (local address 127.0.0.1:57934)
1215 21:02:46.512698 (+ 147us) negotiation.cc:107] Waiting for socket to connect
1215 21:02:46.512757 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.113.60:34031: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20251215 21:02:46.513065 3414 master.cc:679] Network error: unable to get registration information for peer 297c0b4eec8b49459719bf6b7429a987: Client connection negotiation failed: client connection to 127.28.113.60:34031: connect: Connection refused (error 111)
I20251215 21:02:46.516817 4004 rpc_server.cc:307] RPC server started. Bound to: 127.28.113.60:34031
I20251215 21:02:46.516984 4251 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.28.113.60:34031 every 8 connection(s)
I20251215 21:02:46.517339 4004 server_base.cc:1179] Dumped server information to /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765832506145157-29124-0/minicluster-data/master-2/data/info.pb
I20251215 21:02:46.519069 4252 sys_catalog.cc:263] Verifying existing consensus state
I20251215 21:02:46.520107 4252 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 297c0b4eec8b49459719bf6b7429a987: Bootstrap starting.
I20251215 21:02:46.531234 4252 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 297c0b4eec8b49459719bf6b7429a987: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251215 21:02:46.531527 4252 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 297c0b4eec8b49459719bf6b7429a987: Bootstrap complete.
I20251215 21:02:46.532208 4252 raft_consensus.cc:359] T 00000000000000000000000000000000 P 297c0b4eec8b49459719bf6b7429a987 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "8c9267b46ebc4bae9a13928cf23fa8d3" member_type: VOTER last_known_addr { host: "127.28.113.62" port: 41229 } } peers { permanent_uuid: "de9d22b45210451f997093f481e4dfc6" member_type: VOTER last_known_addr { host: "127.28.113.61" port: 39207 } attrs { promote: false } } peers { permanent_uuid: "297c0b4eec8b49459719bf6b7429a987" member_type: NON_VOTER last_known_addr { host: "127.28.113.60" port: 34031 } attrs { promote: true } }
I20251215 21:02:46.532349 4252 raft_consensus.cc:740] T 00000000000000000000000000000000 P 297c0b4eec8b49459719bf6b7429a987 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 297c0b4eec8b49459719bf6b7429a987, State: Initialized, Role: LEARNER
I20251215 21:02:46.532466 4252 consensus_queue.cc:260] T 00000000000000000000000000000000 P 297c0b4eec8b49459719bf6b7429a987 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "8c9267b46ebc4bae9a13928cf23fa8d3" member_type: VOTER last_known_addr { host: "127.28.113.62" port: 41229 } } peers { permanent_uuid: "de9d22b45210451f997093f481e4dfc6" member_type: VOTER last_known_addr { host: "127.28.113.61" port: 39207 } attrs { promote: false } } peers { permanent_uuid: "297c0b4eec8b49459719bf6b7429a987" member_type: NON_VOTER last_known_addr { host: "127.28.113.60" port: 34031 } attrs { promote: true } }
I20251215 21:02:46.532783 4255 sys_catalog.cc:455] T 00000000000000000000000000000000 P 297c0b4eec8b49459719bf6b7429a987 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "8c9267b46ebc4bae9a13928cf23fa8d3" member_type: VOTER last_known_addr { host: "127.28.113.62" port: 41229 } } peers { permanent_uuid: "de9d22b45210451f997093f481e4dfc6" member_type: VOTER last_known_addr { host: "127.28.113.61" port: 39207 } attrs { promote: false } } peers { permanent_uuid: "297c0b4eec8b49459719bf6b7429a987" member_type: NON_VOTER last_known_addr { host: "127.28.113.60" port: 34031 } attrs { promote: true } } }
I20251215 21:02:46.532883 4255 sys_catalog.cc:458] T 00000000000000000000000000000000 P 297c0b4eec8b49459719bf6b7429a987 [sys.catalog]: This master's current role is: LEARNER
I20251215 21:02:46.532924 4252 sys_catalog.cc:565] T 00000000000000000000000000000000 P 297c0b4eec8b49459719bf6b7429a987 [sys.catalog]: configured and running, proceeding with master startup.
I20251215 21:02:46.536401 4266 catalog_manager.cc:1269] Loaded cluster ID: 587ee523bb434fd8a7a37f28e6cc4cb1
I20251215 21:02:46.536449 4266 catalog_manager.cc:1562] T 00000000000000000000000000000000 P 297c0b4eec8b49459719bf6b7429a987: loading cluster ID for follower catalog manager: success
I20251215 21:02:46.537963 4266 catalog_manager.cc:1584] T 00000000000000000000000000000000 P 297c0b4eec8b49459719bf6b7429a987: acquiring CA information for follower catalog manager: success
I20251215 21:02:46.539022 4266 catalog_manager.cc:1612] T 00000000000000000000000000000000 P 297c0b4eec8b49459719bf6b7429a987: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20251215 21:02:46.582904 3979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:02:46.580458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.113.60:34031 (local address 127.0.0.1:57948)
1215 21:02:46.580611 (+ 153us) negotiation.cc:107] Waiting for socket to connect
1215 21:02:46.580625 (+ 14us) client_negotiation.cc:174] Beginning negotiation
1215 21:02:46.580704 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1215 21:02:46.580996 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1215 21:02:46.580998 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1215 21:02:46.581011 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1215 21:02:46.581186 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:02:46.581190 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:02:46.582009 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1215 21:02:46.582011 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1215 21:02:46.582585 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:02:46.582609 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:02:46.582694 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:02:46.582713 (+ 19us) client_negotiation.cc:769] Sending connection context
1215 21:02:46.582752 (+ 39us) client_negotiation.cc:240] Negotiation successful
1215 21:02:46.582800 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":66}
I20251215 21:02:46.583339 4267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:02:46.580555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57948 (local address 127.28.113.60:34031)
1215 21:02:46.580772 (+ 217us) server_negotiation.cc:207] Beginning negotiation
1215 21:02:46.580775 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1215 21:02:46.580788 (+ 13us) server_negotiation.cc:408] Connection header received
1215 21:02:46.580841 (+ 53us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1215 21:02:46.580844 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1215 21:02:46.580892 (+ 48us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1215 21:02:46.580979 (+ 87us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1215 21:02:46.581265 (+ 286us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:02:46.581930 (+ 665us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1215 21:02:46.582691 (+ 761us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:02:46.583029 (+ 338us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:02:46.583153 (+ 124us) server_negotiation.cc:1036] Waiting for connection context
1215 21:02:46.583196 (+ 43us) server_negotiation.cc:300] Negotiation successful
1215 21:02:46.583244 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":72,"threads_started":1}
I20251215 21:02:46.636715 3743 heartbeater.cc:499] Master 127.28.113.62:41229 was elected leader, sending a full tablet report...
I20251215 21:02:46.836015 3879 heartbeater.cc:499] Master 127.28.113.62:41229 was elected leader, sending a full tablet report...
I20251215 21:02:46.973129 4226 raft_consensus.cc:1217] T 00000000000000000000000000000000 P 297c0b4eec8b49459719bf6b7429a987 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20251215 21:02:46.973547 4255 sys_catalog.cc:455] T 00000000000000000000000000000000 P 297c0b4eec8b49459719bf6b7429a987 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8c9267b46ebc4bae9a13928cf23fa8d3. Latest consensus state: current_term: 1 leader_uuid: "8c9267b46ebc4bae9a13928cf23fa8d3" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "8c9267b46ebc4bae9a13928cf23fa8d3" member_type: VOTER last_known_addr { host: "127.28.113.62" port: 41229 } } peers { permanent_uuid: "de9d22b45210451f997093f481e4dfc6" member_type: VOTER last_known_addr { host: "127.28.113.61" port: 39207 } attrs { promote: false } } peers { permanent_uuid: "297c0b4eec8b49459719bf6b7429a987" member_type: NON_VOTER last_known_addr { host: "127.28.113.60" port: 34031 } attrs { promote: true } } }
I20251215 21:02:46.973690 4255 sys_catalog.cc:458] T 00000000000000000000000000000000 P 297c0b4eec8b49459719bf6b7429a987 [sys.catalog]: This master's current role is: LEARNER
I20251215 21:02:47.515645 4268 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c9267b46ebc4bae9a13928cf23fa8d3 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8c9267b46ebc4bae9a13928cf23fa8d3" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "8c9267b46ebc4bae9a13928cf23fa8d3" member_type: VOTER last_known_addr { host: "127.28.113.62" port: 41229 } } peers { permanent_uuid: "de9d22b45210451f997093f481e4dfc6" member_type: VOTER last_known_addr { host: "127.28.113.61" port: 39207 } attrs { promote: false } } peers { permanent_uuid: "297c0b4eec8b49459719bf6b7429a987" member_type: NON_VOTER last_known_addr { host: "127.28.113.60" port: 34031 } attrs { promote: true } } }
I20251215 21:02:47.515851 4268 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c9267b46ebc4bae9a13928cf23fa8d3 [sys.catalog]: This master's current role is: LEADER
I20251215 21:02:47.516726 4268 raft_consensus.cc:1064] T 00000000000000000000000000000000 P 8c9267b46ebc4bae9a13928cf23fa8d3: attempting to promote NON_VOTER 297c0b4eec8b49459719bf6b7429a987 to VOTER
I20251215 21:02:47.517107 4268 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8c9267b46ebc4bae9a13928cf23fa8d3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "8c9267b46ebc4bae9a13928cf23fa8d3" member_type: VOTER last_known_addr { host: "127.28.113.62" port: 41229 } } peers { permanent_uuid: "de9d22b45210451f997093f481e4dfc6" member_type: VOTER last_known_addr { host: "127.28.113.61" port: 39207 } attrs { promote: false } } peers { permanent_uuid: "297c0b4eec8b49459719bf6b7429a987" member_type: VOTER last_known_addr { host: "127.28.113.60" port: 34031 } attrs { promote: false } }
I20251215 21:02:47.518270 4226 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 297c0b4eec8b49459719bf6b7429a987 [term 1 LEARNER]: Refusing update from remote peer 8c9267b46ebc4bae9a13928cf23fa8d3: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20251215 21:02:47.518306 4042 raft_consensus.cc:1275] T 00000000000000000000000000000000 P de9d22b45210451f997093f481e4dfc6 [term 1 FOLLOWER]: Refusing update from remote peer 8c9267b46ebc4bae9a13928cf23fa8d3: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20251215 21:02:47.518628 4269 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 8c9267b46ebc4bae9a13928cf23fa8d3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "297c0b4eec8b49459719bf6b7429a987" member_type: VOTER last_known_addr { host: "127.28.113.60" port: 34031 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20251215 21:02:47.518850 4270 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 8c9267b46ebc4bae9a13928cf23fa8d3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "de9d22b45210451f997093f481e4dfc6" member_type: VOTER last_known_addr { host: "127.28.113.61" port: 39207 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20251215 21:02:47.521350 4269 raft_consensus.cc:2955] T 00000000000000000000000000000000 P 8c9267b46ebc4bae9a13928cf23fa8d3 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 297c0b4eec8b49459719bf6b7429a987 (127.28.113.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "8c9267b46ebc4bae9a13928cf23fa8d3" member_type: VOTER last_known_addr { host: "127.28.113.62" port: 41229 } } peers { permanent_uuid: "de9d22b45210451f997093f481e4dfc6" member_type: VOTER last_known_addr { host: "127.28.113.61" port: 39207 } attrs { promote: false } } peers { permanent_uuid: "297c0b4eec8b49459719bf6b7429a987" member_type: VOTER last_known_addr { host: "127.28.113.60" port: 34031 } attrs { promote: false } } }
I20251215 21:02:47.521822 4226 raft_consensus.cc:2955] T 00000000000000000000000000000000 P 297c0b4eec8b49459719bf6b7429a987 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 297c0b4eec8b49459719bf6b7429a987 (127.28.113.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "8c9267b46ebc4bae9a13928cf23fa8d3" member_type: VOTER last_known_addr { host: "127.28.113.62" port: 41229 } } peers { permanent_uuid: "de9d22b45210451f997093f481e4dfc6" member_type: VOTER last_known_addr { host: "127.28.113.61" port: 39207 } attrs { promote: false } } peers { permanent_uuid: "297c0b4eec8b49459719bf6b7429a987" member_type: VOTER last_known_addr { host: "127.28.113.60" port: 34031 } attrs { promote: false } } }
I20251215 21:02:47.522706 4270 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c9267b46ebc4bae9a13928cf23fa8d3 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "8c9267b46ebc4bae9a13928cf23fa8d3" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "8c9267b46ebc4bae9a13928cf23fa8d3" member_type: VOTER last_known_addr { host: "127.28.113.62" port: 41229 } } peers { permanent_uuid: "de9d22b45210451f997093f481e4dfc6" member_type: VOTER last_known_addr { host: "127.28.113.61" port: 39207 } attrs { promote: false } } peers { permanent_uuid: "297c0b4eec8b49459719bf6b7429a987" member_type: VOTER last_known_addr { host: "127.28.113.60" port: 34031 } attrs { promote: false } } }
I20251215 21:02:47.522804 4270 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c9267b46ebc4bae9a13928cf23fa8d3 [sys.catalog]: This master's current role is: LEADER
I20251215 21:02:47.522948 4270 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c9267b46ebc4bae9a13928cf23fa8d3 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8c9267b46ebc4bae9a13928cf23fa8d3" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "8c9267b46ebc4bae9a13928cf23fa8d3" member_type: VOTER last_known_addr { host: "127.28.113.62" port: 41229 } } peers { permanent_uuid: "de9d22b45210451f997093f481e4dfc6" member_type: VOTER last_known_addr { host: "127.28.113.61" port: 39207 } attrs { promote: false } } peers { permanent_uuid: "297c0b4eec8b49459719bf6b7429a987" member_type: VOTER last_known_addr { host: "127.28.113.60" port: 34031 } attrs { promote: false } } }
I20251215 21:02:47.523011 4270 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c9267b46ebc4bae9a13928cf23fa8d3 [sys.catalog]: This master's current role is: LEADER
I20251215 21:02:47.524124 4275 sys_catalog.cc:455] T 00000000000000000000000000000000 P 297c0b4eec8b49459719bf6b7429a987 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8c9267b46ebc4bae9a13928cf23fa8d3" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "8c9267b46ebc4bae9a13928cf23fa8d3" member_type: VOTER last_known_addr { host: "127.28.113.62" port: 41229 } } peers { permanent_uuid: "de9d22b45210451f997093f481e4dfc6" member_type: VOTER last_known_addr { host: "127.28.113.61" port: 39207 } attrs { promote: false } } peers { permanent_uuid: "297c0b4eec8b49459719bf6b7429a987" member_type: VOTER last_known_addr { host: "127.28.113.60" port: 34031 } attrs { promote: false } } }
I20251215 21:02:47.524235 4275 sys_catalog.cc:458] T 00000000000000000000000000000000 P 297c0b4eec8b49459719bf6b7429a987 [sys.catalog]: This master's current role is: FOLLOWER
I20251215 21:02:47.524926 4270 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c9267b46ebc4bae9a13928cf23fa8d3 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8c9267b46ebc4bae9a13928cf23fa8d3" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "8c9267b46ebc4bae9a13928cf23fa8d3" member_type: VOTER last_known_addr { host: "127.28.113.62" port: 41229 } } peers { permanent_uuid: "de9d22b45210451f997093f481e4dfc6" member_type: VOTER last_known_addr { host: "127.28.113.61" port: 39207 } attrs { promote: false } } peers { permanent_uuid: "297c0b4eec8b49459719bf6b7429a987" member_type: VOTER last_known_addr { host: "127.28.113.60" port: 34031 } attrs { promote: false } } }
I20251215 21:02:47.525007 4270 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c9267b46ebc4bae9a13928cf23fa8d3 [sys.catalog]: This master's current role is: LEADER
I20251215 21:02:47.525305 4042 raft_consensus.cc:2955] T 00000000000000000000000000000000 P de9d22b45210451f997093f481e4dfc6 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 297c0b4eec8b49459719bf6b7429a987 (127.28.113.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "8c9267b46ebc4bae9a13928cf23fa8d3" member_type: VOTER last_known_addr { host: "127.28.113.62" port: 41229 } } peers { permanent_uuid: "de9d22b45210451f997093f481e4dfc6" member_type: VOTER last_known_addr { host: "127.28.113.61" port: 39207 } attrs { promote: false } } peers { permanent_uuid: "297c0b4eec8b49459719bf6b7429a987" member_type: VOTER last_known_addr { host: "127.28.113.60" port: 34031 } attrs { promote: false } } }
I20251215 21:02:47.526432 4276 sys_catalog.cc:455] T 00000000000000000000000000000000 P de9d22b45210451f997093f481e4dfc6 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8c9267b46ebc4bae9a13928cf23fa8d3" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "8c9267b46ebc4bae9a13928cf23fa8d3" member_type: VOTER last_known_addr { host: "127.28.113.62" port: 41229 } } peers { permanent_uuid: "de9d22b45210451f997093f481e4dfc6" member_type: VOTER last_known_addr { host: "127.28.113.61" port: 39207 } attrs { promote: false } } peers { permanent_uuid: "297c0b4eec8b49459719bf6b7429a987" member_type: VOTER last_known_addr { host: "127.28.113.60" port: 34031 } attrs { promote: false } } }
I20251215 21:02:47.526536 4276 sys_catalog.cc:458] T 00000000000000000000000000000000 P de9d22b45210451f997093f481e4dfc6 [sys.catalog]: This master's current role is: FOLLOWER
I20251215 21:02:48.502736 4278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:02:48.496702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56084 (local address 127.28.113.61:39207)
1215 21:02:48.496931 (+ 229us) server_negotiation.cc:207] Beginning negotiation
1215 21:02:48.496935 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1215 21:02:48.496972 (+ 37us) server_negotiation.cc:408] Connection header received
1215 21:02:48.497092 (+ 120us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1215 21:02:48.497095 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1215 21:02:48.497134 (+ 39us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1215 21:02:48.497203 (+ 69us) server_negotiation.cc:227] Negotiated authn=SASL
1215 21:02:48.497921 (+ 718us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:02:48.498441 (+ 520us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1215 21:02:48.499050 (+ 609us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:02:48.499177 (+ 127us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:02:48.500273 (+ 1096us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1215 21:02:48.500287 (+ 14us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1215 21:02:48.500289 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1215 21:02:48.500304 (+ 15us) server_negotiation.cc:923] calling sasl_server_start()
1215 21:02:48.501590 (+ 1286us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1215 21:02:48.501915 (+ 325us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1215 21:02:48.501917 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1215 21:02:48.501918 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1215 21:02:48.501952 (+ 34us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1215 21:02:48.502109 (+ 157us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1215 21:02:48.502110 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1215 21:02:48.502112 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1215 21:02:48.502237 (+ 125us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1215 21:02:48.502327 (+ 90us) server_negotiation.cc:1036] Waiting for connection context
1215 21:02:48.502465 (+ 138us) server_negotiation.cc:300] Negotiation successful
1215 21:02:48.502554 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":158,"thread_start_us":87,"threads_started":1}
I20251215 21:02:48.505659 4279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:02:48.503439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.113.62:41229 (local address 127.0.0.1:36722)
1215 21:02:48.503643 (+ 204us) negotiation.cc:107] Waiting for socket to connect
1215 21:02:48.503654 (+ 11us) client_negotiation.cc:174] Beginning negotiation
1215 21:02:48.503721 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1215 21:02:48.503937 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1215 21:02:48.503939 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1215 21:02:48.503951 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1215 21:02:48.504154 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:02:48.504160 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:02:48.504815 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1215 21:02:48.504817 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1215 21:02:48.505389 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:02:48.505395 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:02:48.505476 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:02:48.505489 (+ 13us) client_negotiation.cc:769] Sending connection context
1215 21:02:48.505521 (+ 32us) client_negotiation.cc:240] Negotiation successful
1215 21:02:48.505563 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":70,"threads_started":1}
I20251215 21:02:48.506073 4280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:02:48.503561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36722 (local address 127.28.113.62:41229)
1215 21:02:48.503758 (+ 197us) server_negotiation.cc:207] Beginning negotiation
1215 21:02:48.503761 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1215 21:02:48.503772 (+ 11us) server_negotiation.cc:408] Connection header received
1215 21:02:48.503812 (+ 40us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1215 21:02:48.503814 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1215 21:02:48.503854 (+ 40us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1215 21:02:48.503922 (+ 68us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1215 21:02:48.504243 (+ 321us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:02:48.504736 (+ 493us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1215 21:02:48.505478 (+ 742us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:02:48.505825 (+ 347us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:02:48.505865 (+ 40us) server_negotiation.cc:1036] Waiting for connection context
1215 21:02:48.505915 (+ 50us) server_negotiation.cc:300] Negotiation successful
1215 21:02:48.505963 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":57,"threads_started":1}
I20251215 21:02:48.509272 4279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:02:48.507156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.113.61:39207 (local address 127.0.0.1:56088)
1215 21:02:48.507263 (+ 107us) negotiation.cc:107] Waiting for socket to connect
1215 21:02:48.507275 (+ 12us) client_negotiation.cc:174] Beginning negotiation
1215 21:02:48.507343 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1215 21:02:48.507537 (+ 194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1215 21:02:48.507539 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1215 21:02:48.507556 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1215 21:02:48.507701 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:02:48.507706 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:02:48.508409 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1215 21:02:48.508413 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1215 21:02:48.508936 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:02:48.508942 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:02:48.509037 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:02:48.509052 (+ 15us) client_negotiation.cc:769] Sending connection context
1215 21:02:48.509109 (+ 57us) client_negotiation.cc:240] Negotiation successful
1215 21:02:48.509160 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
I20251215 21:02:48.509599 4278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:02:48.507275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56088 (local address 127.28.113.61:39207)
1215 21:02:48.507381 (+ 106us) server_negotiation.cc:207] Beginning negotiation
1215 21:02:48.507385 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1215 21:02:48.507396 (+ 11us) server_negotiation.cc:408] Connection header received
1215 21:02:48.507428 (+ 32us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1215 21:02:48.507431 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1215 21:02:48.507464 (+ 33us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1215 21:02:48.507519 (+ 55us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1215 21:02:48.507798 (+ 279us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:02:48.508314 (+ 516us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1215 21:02:48.509045 (+ 731us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:02:48.509430 (+ 385us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:02:48.509456 (+ 26us) server_negotiation.cc:1036] Waiting for connection context
1215 21:02:48.509491 (+ 35us) server_negotiation.cc:300] Negotiation successful
1215 21:02:48.509530 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46,"spinlock_wait_cycles":12928}
I20251215 21:02:48.512811 4279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:02:48.510553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.113.60:34031 (local address 127.0.0.1:46366)
1215 21:02:48.510653 (+ 100us) negotiation.cc:107] Waiting for socket to connect
1215 21:02:48.510664 (+ 11us) client_negotiation.cc:174] Beginning negotiation
1215 21:02:48.510721 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1215 21:02:48.511149 (+ 428us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1215 21:02:48.511152 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1215 21:02:48.511170 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1215 21:02:48.511307 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:02:48.511312 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:02:48.512003 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1215 21:02:48.512007 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1215 21:02:48.512541 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:02:48.512549 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:02:48.512639 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:02:48.512652 (+ 13us) client_negotiation.cc:769] Sending connection context
1215 21:02:48.512685 (+ 33us) client_negotiation.cc:240] Negotiation successful
1215 21:02:48.512723 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20251215 21:02:48.513226 4281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:02:48.510703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46366 (local address 127.28.113.60:34031)
1215 21:02:48.510949 (+ 246us) server_negotiation.cc:207] Beginning negotiation
1215 21:02:48.510953 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1215 21:02:48.510963 (+ 10us) server_negotiation.cc:408] Connection header received
1215 21:02:48.511002 (+ 39us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1215 21:02:48.511005 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1215 21:02:48.511042 (+ 37us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1215 21:02:48.511119 (+ 77us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1215 21:02:48.511396 (+ 277us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:02:48.511875 (+ 479us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1215 21:02:48.512641 (+ 766us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:02:48.513010 (+ 369us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:02:48.513040 (+ 30us) server_negotiation.cc:1036] Waiting for connection context
1215 21:02:48.513091 (+ 51us) server_negotiation.cc:300] Negotiation successful
1215 21:02:48.513133 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":180,"thread_start_us":96,"threads_started":1}
I20251215 21:02:48.521338 4281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:02:48.514885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46382 (local address 127.28.113.60:34031)
1215 21:02:48.514975 (+ 90us) server_negotiation.cc:207] Beginning negotiation
1215 21:02:48.514978 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1215 21:02:48.514989 (+ 11us) server_negotiation.cc:408] Connection header received
1215 21:02:48.515081 (+ 92us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1215 21:02:48.515084 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1215 21:02:48.515120 (+ 36us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1215 21:02:48.515180 (+ 60us) server_negotiation.cc:227] Negotiated authn=SASL
1215 21:02:48.515905 (+ 725us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:02:48.516384 (+ 479us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1215 21:02:48.516994 (+ 610us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:02:48.517170 (+ 176us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:02:48.518262 (+ 1092us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1215 21:02:48.518276 (+ 14us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1215 21:02:48.518278 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1215 21:02:48.518301 (+ 23us) server_negotiation.cc:923] calling sasl_server_start()
1215 21:02:48.519866 (+ 1565us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1215 21:02:48.520297 (+ 431us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1215 21:02:48.520300 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1215 21:02:48.520301 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1215 21:02:48.520341 (+ 40us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1215 21:02:48.520556 (+ 215us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1215 21:02:48.520559 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1215 21:02:48.520561 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1215 21:02:48.520705 (+ 144us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1215 21:02:48.520844 (+ 139us) server_negotiation.cc:1036] Waiting for connection context
1215 21:02:48.521084 (+ 240us) server_negotiation.cc:300] Negotiation successful
1215 21:02:48.521198 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20251215 21:02:48.524591 4282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:02:48.522152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.113.62:41229 (local address 127.0.0.1:36724)
1215 21:02:48.522400 (+ 248us) negotiation.cc:107] Waiting for socket to connect
1215 21:02:48.522411 (+ 11us) client_negotiation.cc:174] Beginning negotiation
1215 21:02:48.522490 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1215 21:02:48.522706 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1215 21:02:48.522708 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1215 21:02:48.522719 (+ 11us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1215 21:02:48.522894 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:02:48.522899 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:02:48.523605 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1215 21:02:48.523608 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1215 21:02:48.524294 (+ 686us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:02:48.524303 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:02:48.524394 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:02:48.524408 (+ 14us) client_negotiation.cc:769] Sending connection context
1215 21:02:48.524442 (+ 34us) client_negotiation.cc:240] Negotiation successful
1215 21:02:48.524490 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":91,"threads_started":1}
I20251215 21:02:48.525110 4280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:02:48.522252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36724 (local address 127.28.113.62:41229)
1215 21:02:48.522371 (+ 119us) server_negotiation.cc:207] Beginning negotiation
1215 21:02:48.522375 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1215 21:02:48.522468 (+ 93us) server_negotiation.cc:408] Connection header received
1215 21:02:48.522565 (+ 97us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1215 21:02:48.522567 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1215 21:02:48.522625 (+ 58us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1215 21:02:48.522687 (+ 62us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1215 21:02:48.522995 (+ 308us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:02:48.523493 (+ 498us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1215 21:02:48.524431 (+ 938us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:02:48.524852 (+ 421us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:02:48.524906 (+ 54us) server_negotiation.cc:1036] Waiting for connection context
1215 21:02:48.524955 (+ 49us) server_negotiation.cc:300] Negotiation successful
1215 21:02:48.525020 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20251215 21:02:48.528581 4282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:02:48.526241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.113.61:39207 (local address 127.0.0.1:56096)
1215 21:02:48.526477 (+ 236us) negotiation.cc:107] Waiting for socket to connect
1215 21:02:48.526491 (+ 14us) client_negotiation.cc:174] Beginning negotiation
1215 21:02:48.526572 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1215 21:02:48.526802 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1215 21:02:48.526805 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1215 21:02:48.526820 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1215 21:02:48.526976 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:02:48.526982 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:02:48.527654 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1215 21:02:48.527657 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1215 21:02:48.528249 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:02:48.528258 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:02:48.528354 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:02:48.528371 (+ 17us) client_negotiation.cc:769] Sending connection context
1215 21:02:48.528418 (+ 47us) client_negotiation.cc:240] Negotiation successful
1215 21:02:48.528472 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":169}
I20251215 21:02:48.528957 4278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:02:48.526358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56096 (local address 127.28.113.61:39207)
1215 21:02:48.526479 (+ 121us) server_negotiation.cc:207] Beginning negotiation
1215 21:02:48.526483 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1215 21:02:48.526546 (+ 63us) server_negotiation.cc:408] Connection header received
1215 21:02:48.526667 (+ 121us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1215 21:02:48.526670 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1215 21:02:48.526716 (+ 46us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1215 21:02:48.526777 (+ 61us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1215 21:02:48.527089 (+ 312us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:02:48.527548 (+ 459us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1215 21:02:48.528360 (+ 812us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:02:48.528769 (+ 409us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:02:48.528799 (+ 30us) server_negotiation.cc:1036] Waiting for connection context
1215 21:02:48.528839 (+ 40us) server_negotiation.cc:300] Negotiation successful
1215 21:02:48.528878 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20251215 21:02:48.532150 4282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:02:48.530150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.113.60:34031 (local address 127.0.0.1:46386)
1215 21:02:48.530264 (+ 114us) negotiation.cc:107] Waiting for socket to connect
1215 21:02:48.530278 (+ 14us) client_negotiation.cc:174] Beginning negotiation
1215 21:02:48.530341 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1215 21:02:48.530516 (+ 175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1215 21:02:48.530518 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1215 21:02:48.530530 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1215 21:02:48.530666 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:02:48.530670 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:02:48.531298 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1215 21:02:48.531300 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1215 21:02:48.531823 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:02:48.531829 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:02:48.531918 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:02:48.531932 (+ 14us) client_negotiation.cc:769] Sending connection context
1215 21:02:48.531997 (+ 65us) client_negotiation.cc:240] Negotiation successful
1215 21:02:48.532048 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
I20251215 21:02:48.532552 4281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:02:48.530213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46386 (local address 127.28.113.60:34031)
1215 21:02:48.530319 (+ 106us) server_negotiation.cc:207] Beginning negotiation
1215 21:02:48.530322 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1215 21:02:48.530333 (+ 11us) server_negotiation.cc:408] Connection header received
1215 21:02:48.530413 (+ 80us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1215 21:02:48.530416 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1215 21:02:48.530453 (+ 37us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1215 21:02:48.530505 (+ 52us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1215 21:02:48.530738 (+ 233us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:02:48.531219 (+ 481us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1215 21:02:48.531940 (+ 721us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:02:48.532370 (+ 430us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:02:48.532398 (+ 28us) server_negotiation.cc:1036] Waiting for connection context
1215 21:02:48.532434 (+ 36us) server_negotiation.cc:300] Negotiation successful
1215 21:02:48.532473 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43,"spinlock_wait_cycles":2304}
I20251215 21:02:48.533849 29124 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu with pid 3478
I20251215 21:02:48.539728 29124 external_mini_cluster.cc:1366] Running /tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu
/tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765832506145157-29124-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765832506145157-29124-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765832506145157-29124-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765832506145157-29124-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.113.1:35255
--local_ip_for_outbound_sockets=127.28.113.1
--tserver_master_addrs=127.28.113.62:41229
--webserver_port=37473
--webserver_interface=127.28.113.1
--keytab_file=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765832506145157-29124-0/krb5kdc/oryx.keytab
--principal=oryx/127.28.113.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.28.113.20:39895
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.28.113.62:41229,127.28.113.61:39207
--tserver_master_addrs=127.28.113.62:41229,127.28.113.61:39207,127.28.113.60:34031 with env {KRB5CCNAME=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765832506145157-29124-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765832506145157-29124-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765832506145157-29124-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251215 21:02:48.646191 4283 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251215 21:02:48.646414 4283 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251215 21:02:48.646481 4283 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251215 21:02:48.649715 4283 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251215 21:02:48.649784 4283 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251215 21:02:48.649899 4283 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.113.1
I20251215 21:02:48.654062 4283 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.113.20:39895
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765832506145157-29124-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765832506145157-29124-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765832506145157-29124-0/krb5kdc/oryx.keytab
--principal=oryx/127.28.113.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.113.1:35255
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765832506145157-29124-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.113.1
--webserver_port=37473
--webserver_require_spnego=true
--tserver_master_addrs=127.28.113.62:41229,127.28.113.61:39207,127.28.113.60:34031
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4283
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.113.1
--log_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765832506145157-29124-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision dc741732bb6a40213e11a13869fad06a49a209a3
build type DEBUG
built by None at 15 Dec 2025 20:43:14 UTC on 24a791456cd2
build id 9530
I20251215 21:02:48.655107 4283 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251215 21:02:48.655946 4283 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251215 21:02:48.662387 4288 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251215 21:02:48.662388 4291 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251215 21:02:48.662582 4283 server_base.cc:1047] running on GCE node
W20251215 21:02:48.662432 4289 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251215 21:02:48.663175 4283 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251215 21:02:48.663690 4283 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251215 21:02:48.664836 4283 hybrid_clock.cc:648] HybridClock initialized: now 1765832568664824 us; error 29 us; skew 500 ppm
Dec 15 21:02:48 dist-test-slave-bjsf krb5kdc[3366](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765832568, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.113.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251215 21:02:48.667378 4283 init.cc:377] Logged in from keytab as oryx/127.28.113.1@KRBTEST.COM (short username oryx)
I20251215 21:02:48.668376 4283 webserver.cc:492] Webserver started at http://127.28.113.1:37473/ using document root <none> and password file <none>
I20251215 21:02:48.668898 4283 fs_manager.cc:362] Metadata directory not provided
I20251215 21:02:48.668959 4283 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251215 21:02:48.672240 4283 fs_manager.cc:714] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20251215 21:02:48.674002 4298 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251215 21:02:48.675169 4283 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20251215 21:02:48.675276 4283 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765832506145157-29124-0/minicluster-data/ts-0/data,/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765832506145157-29124-0/minicluster-data/ts-0/wal
uuid: "6956a9660a3b4d42867878501794f2d5"
format_stamp: "Formatted at 2025-12-15 21:02:45 on dist-test-slave-bjsf"
server_key: "ca85dbf5eba3ad512963247bd0af2e0b"
server_key_iv: "f11137c2b59bf2ba6e9b515bc4fbb292"
server_key_version: "encryptionkey@0"
I20251215 21:02:48.675720 4283 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765832506145157-29124-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765832506145157-29124-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765832506145157-29124-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251215 21:02:48.689158 4283 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251215 21:02:48.691838 4283 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251215 21:02:48.692035 4283 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251215 21:02:48.692592 4283 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251215 21:02:48.693488 4283 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251215 21:02:48.693533 4283 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251215 21:02:48.693581 4283 ts_tablet_manager.cc:616] Registered 0 tablets
I20251215 21:02:48.693611 4283 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251215 21:02:48.702107 4283 rpc_server.cc:307] RPC server started. Bound to: 127.28.113.1:35255
I20251215 21:02:48.702131 4411 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.28.113.1:35255 every 8 connection(s)
I20251215 21:02:48.703112 4283 server_base.cc:1179] Dumped server information to /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765832506145157-29124-0/minicluster-data/ts-0/data/info.pb
I20251215 21:02:48.705159 29124 external_mini_cluster.cc:1428] Started /tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu as pid 4283
I20251215 21:02:48.705329 29124 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu with pid 3614
I20251215 21:02:48.713037 29124 external_mini_cluster.cc:1366] Running /tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu
/tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765832506145157-29124-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765832506145157-29124-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765832506145157-29124-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765832506145157-29124-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.113.2:32787
--local_ip_for_outbound_sockets=127.28.113.2
--tserver_master_addrs=127.28.113.62:41229
--webserver_port=41927
--webserver_interface=127.28.113.2
--keytab_file=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765832506145157-29124-0/krb5kdc/oryx.keytab
--principal=oryx/127.28.113.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.28.113.20:39895
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.28.113.62:41229,127.28.113.61:39207
--tserver_master_addrs=127.28.113.62:41229,127.28.113.61:39207,127.28.113.60:34031 with env {KRB5CCNAME=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765832506145157-29124-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765832506145157-29124-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765832506145157-29124-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Dec 15 21:02:48 dist-test-slave-bjsf krb5kdc[3366](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765832568, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.113.1@KRBTEST.COM for oryx/127.28.113.62@KRBTEST.COM
Dec 15 21:02:48 dist-test-slave-bjsf krb5kdc[3366](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765832568, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.113.1@KRBTEST.COM for oryx/127.28.113.61@KRBTEST.COM
Dec 15 21:02:48 dist-test-slave-bjsf krb5kdc[3366](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765832568, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.113.1@KRBTEST.COM for oryx/127.28.113.60@KRBTEST.COM
I20251215 21:02:48.718344 4280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:02:48.704859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.113.1:60673 (local address 127.28.113.62:41229)
1215 21:02:48.705112 (+ 253us) server_negotiation.cc:207] Beginning negotiation
1215 21:02:48.705116 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1215 21:02:48.706146 (+ 1030us) server_negotiation.cc:408] Connection header received
1215 21:02:48.707154 (+ 1008us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1215 21:02:48.707158 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1215 21:02:48.707212 (+ 54us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1215 21:02:48.707279 (+ 67us) server_negotiation.cc:227] Negotiated authn=SASL
1215 21:02:48.709008 (+ 1729us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:02:48.709810 (+ 802us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1215 21:02:48.710679 (+ 869us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:02:48.710870 (+ 191us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:02:48.714517 (+ 3647us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1215 21:02:48.714537 (+ 20us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1215 21:02:48.714540 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1215 21:02:48.714570 (+ 30us) server_negotiation.cc:923] calling sasl_server_start()
1215 21:02:48.716106 (+ 1536us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1215 21:02:48.716731 (+ 625us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1215 21:02:48.716734 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1215 21:02:48.716737 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
1215 21:02:48.716791 (+ 54us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1215 21:02:48.717119 (+ 328us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1215 21:02:48.717122 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1215 21:02:48.717124 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1215 21:02:48.717300 (+ 176us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1215 21:02:48.717367 (+ 67us) server_negotiation.cc:1036] Waiting for connection context
1215 21:02:48.718056 (+ 689us) server_negotiation.cc:300] Negotiation successful
1215 21:02:48.718184 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":174}
I20251215 21:02:48.718344 4278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:02:48.704866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.113.1:35235 (local address 127.28.113.61:39207)
1215 21:02:48.705010 (+ 144us) server_negotiation.cc:207] Beginning negotiation
1215 21:02:48.705015 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1215 21:02:48.706936 (+ 1921us) server_negotiation.cc:408] Connection header received
1215 21:02:48.707116 (+ 180us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1215 21:02:48.707119 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1215 21:02:48.707175 (+ 56us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1215 21:02:48.707247 (+ 72us) server_negotiation.cc:227] Negotiated authn=SASL
1215 21:02:48.709015 (+ 1768us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:02:48.709792 (+ 777us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1215 21:02:48.710678 (+ 886us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:02:48.710868 (+ 190us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:02:48.714323 (+ 3455us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1215 21:02:48.714344 (+ 21us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1215 21:02:48.714348 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1215 21:02:48.714376 (+ 28us) server_negotiation.cc:923] calling sasl_server_start()
1215 21:02:48.716066 (+ 1690us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1215 21:02:48.716731 (+ 665us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1215 21:02:48.716734 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1215 21:02:48.716736 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1215 21:02:48.716784 (+ 48us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1215 21:02:48.717119 (+ 335us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1215 21:02:48.717122 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1215 21:02:48.717124 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1215 21:02:48.717311 (+ 187us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1215 21:02:48.717367 (+ 56us) server_negotiation.cc:1036] Waiting for connection context
1215 21:02:48.718062 (+ 695us) server_negotiation.cc:300] Negotiation successful
1215 21:02:48.718183 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20251215 21:02:48.719374 4281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:02:48.705296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.113.1:34019 (local address 127.28.113.60:34031)
1215 21:02:48.705415 (+ 119us) server_negotiation.cc:207] Beginning negotiation
1215 21:02:48.705419 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1215 21:02:48.708699 (+ 3280us) server_negotiation.cc:408] Connection header received
1215 21:02:48.708864 (+ 165us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1215 21:02:48.708867 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1215 21:02:48.708918 (+ 51us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1215 21:02:48.708964 (+ 46us) server_negotiation.cc:227] Negotiated authn=SASL
1215 21:02:48.709679 (+ 715us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:02:48.710401 (+ 722us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1215 21:02:48.711231 (+ 830us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:02:48.711941 (+ 710us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:02:48.715567 (+ 3626us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1215 21:02:48.715588 (+ 21us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1215 21:02:48.715591 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1215 21:02:48.715621 (+ 30us) server_negotiation.cc:923] calling sasl_server_start()
1215 21:02:48.717327 (+ 1706us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1215 21:02:48.717780 (+ 453us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1215 21:02:48.717784 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1215 21:02:48.717786 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1215 21:02:48.717850 (+ 64us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1215 21:02:48.718117 (+ 267us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1215 21:02:48.718120 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1215 21:02:48.718122 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1215 21:02:48.718296 (+ 174us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1215 21:02:48.718366 (+ 70us) server_negotiation.cc:1036] Waiting for connection context
1215 21:02:48.719098 (+ 732us) server_negotiation.cc:300] Negotiation successful
1215 21:02:48.719219 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20251215 21:02:48.719676 4416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:02:48.705199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.113.62:41229 (local address 127.28.113.1:60673)
1215 21:02:48.705993 (+ 794us) negotiation.cc:107] Waiting for socket to connect
1215 21:02:48.706036 (+ 43us) client_negotiation.cc:174] Beginning negotiation
1215 21:02:48.706930 (+ 894us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1215 21:02:48.707482 (+ 552us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1215 21:02:48.707485 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1215 21:02:48.708073 (+ 588us) client_negotiation.cc:189] Negotiated authn=SASL
1215 21:02:48.708838 (+ 765us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:02:48.708853 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:02:48.709942 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1215 21:02:48.709945 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1215 21:02:48.710552 (+ 607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:02:48.710558 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:02:48.711097 (+ 539us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:02:48.711887 (+ 790us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1215 21:02:48.711901 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
1215 21:02:48.714392 (+ 2491us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1215 21:02:48.716206 (+ 1814us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1215 21:02:48.716208 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1215 21:02:48.716214 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
1215 21:02:48.716620 (+ 406us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1215 21:02:48.716891 (+ 271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1215 21:02:48.716894 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1215 21:02:48.716896 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1215 21:02:48.717017 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1215 21:02:48.717448 (+ 431us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1215 21:02:48.717455 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1215 21:02:48.717755 (+ 300us) client_negotiation.cc:769] Sending connection context
1215 21:02:48.718523 (+ 768us) client_negotiation.cc:240] Negotiation successful
1215 21:02:48.718897 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":583,"mutex_wait_us":117,"spinlock_wait_cycles":24448,"thread_start_us":111,"threads_started":1}
I20251215 21:02:48.719684 4417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:02:48.705278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.113.61:39207 (local address 127.28.113.1:35235)
1215 21:02:48.706838 (+ 1560us) negotiation.cc:107] Waiting for socket to connect
1215 21:02:48.706848 (+ 10us) client_negotiation.cc:174] Beginning negotiation
1215 21:02:48.706942 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1215 21:02:48.707413 (+ 471us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1215 21:02:48.707420 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
1215 21:02:48.708415 (+ 995us) client_negotiation.cc:189] Negotiated authn=SASL
1215 21:02:48.708838 (+ 423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:02:48.708853 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:02:48.709942 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1215 21:02:48.709945 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1215 21:02:48.710549 (+ 604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:02:48.710558 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:02:48.711097 (+ 539us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:02:48.711843 (+ 746us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1215 21:02:48.711886 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
1215 21:02:48.714153 (+ 2267us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1215 21:02:48.716196 (+ 2043us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1215 21:02:48.716201 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1215 21:02:48.716214 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
1215 21:02:48.716620 (+ 406us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1215 21:02:48.716891 (+ 271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1215 21:02:48.716894 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1215 21:02:48.716896 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1215 21:02:48.717017 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1215 21:02:48.717448 (+ 431us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1215 21:02:48.717455 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1215 21:02:48.717753 (+ 298us) client_negotiation.cc:769] Sending connection context
1215 21:02:48.718523 (+ 770us) client_negotiation.cc:240] Negotiation successful
1215 21:02:48.718824 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1494,"spinlock_wait_cycles":76416,"thread_start_us":115,"threads_started":1}
I20251215 21:02:48.720096 4418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:02:48.705199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.113.60:34031 (local address 127.28.113.1:34019)
1215 21:02:48.708615 (+ 3416us) negotiation.cc:107] Waiting for socket to connect
1215 21:02:48.708628 (+ 13us) client_negotiation.cc:174] Beginning negotiation
1215 21:02:48.708759 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1215 21:02:48.709081 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1215 21:02:48.709084 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1215 21:02:48.709248 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
1215 21:02:48.709544 (+ 296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:02:48.709550 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:02:48.710536 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1215 21:02:48.710539 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1215 21:02:48.711106 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:02:48.711115 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:02:48.712081 (+ 966us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:02:48.712530 (+ 449us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1215 21:02:48.712544 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
1215 21:02:48.715015 (+ 2471us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1215 21:02:48.717462 (+ 2447us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1215 21:02:48.717466 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1215 21:02:48.717468 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1215 21:02:48.717667 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1215 21:02:48.717961 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1215 21:02:48.717964 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1215 21:02:48.717966 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1215 21:02:48.718019 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1215 21:02:48.718792 (+ 773us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1215 21:02:48.718796 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1215 21:02:48.718899 (+ 103us) client_negotiation.cc:769] Sending connection context
1215 21:02:48.719747 (+ 848us) client_negotiation.cc:240] Negotiation successful
1215 21:02:48.719894 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3327,"mutex_wait_us":795,"thread_start_us":80,"threads_started":1}
I20251215 21:02:48.721025 4413 heartbeater.cc:344] Connected to a master server at 127.28.113.62:41229
I20251215 21:02:48.721029 4414 heartbeater.cc:344] Connected to a master server at 127.28.113.61:39207
I20251215 21:02:48.721225 4412 heartbeater.cc:344] Connected to a master server at 127.28.113.60:34031
I20251215 21:02:48.721320 4413 heartbeater.cc:461] Registering TS with master...
I20251215 21:02:48.721320 4414 heartbeater.cc:461] Registering TS with master...
I20251215 21:02:48.721320 4412 heartbeater.cc:461] Registering TS with master...
I20251215 21:02:48.721988 4412 heartbeater.cc:507] Master 127.28.113.60:34031 requested a full tablet report, sending...
I20251215 21:02:48.721979 4414 heartbeater.cc:507] Master 127.28.113.61:39207 requested a full tablet report, sending...
I20251215 21:02:48.721979 4413 heartbeater.cc:507] Master 127.28.113.62:41229 requested a full tablet report, sending...
I20251215 21:02:48.723171 3414 ts_manager.cc:194] Re-registered known tserver with Master: 6956a9660a3b4d42867878501794f2d5 (127.28.113.1:35255)
I20251215 21:02:48.723413 4032 ts_manager.cc:194] Registered new tserver with Master: 6956a9660a3b4d42867878501794f2d5 (127.28.113.1:35255)
I20251215 21:02:48.723452 4216 ts_manager.cc:194] Registered new tserver with Master: 6956a9660a3b4d42867878501794f2d5 (127.28.113.1:35255)
I20251215 21:02:48.723837 3414 master_service.cc:502] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.28.113.1@KRBTEST.COM'} at 127.28.113.1:60673
W20251215 21:02:48.819797 4419 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251215 21:02:48.820060 4419 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251215 21:02:48.820138 4419 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251215 21:02:48.823529 4419 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251215 21:02:48.823602 4419 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251215 21:02:48.823694 4419 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.113.2
I20251215 21:02:48.827993 4419 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.113.20:39895
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765832506145157-29124-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765832506145157-29124-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765832506145157-29124-0/krb5kdc/oryx.keytab
--principal=oryx/127.28.113.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.113.2:32787
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765832506145157-29124-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.113.2
--webserver_port=41927
--webserver_require_spnego=true
--tserver_master_addrs=127.28.113.62:41229,127.28.113.61:39207,127.28.113.60:34031
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4419
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.113.2
--log_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765832506145157-29124-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision dc741732bb6a40213e11a13869fad06a49a209a3
build type DEBUG
built by None at 15 Dec 2025 20:43:14 UTC on 24a791456cd2
build id 9530
I20251215 21:02:48.829038 4419 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251215 21:02:48.829891 4419 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251215 21:02:48.836249 4424 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251215 21:02:48.836243 4427 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251215 21:02:48.836442 4419 server_base.cc:1047] running on GCE node
W20251215 21:02:48.836236 4425 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251215 21:02:48.836855 4419 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251215 21:02:48.837369 4419 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251215 21:02:48.838512 4419 hybrid_clock.cc:648] HybridClock initialized: now 1765832568838504 us; error 28 us; skew 500 ppm
Dec 15 21:02:48 dist-test-slave-bjsf krb5kdc[3366](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765832568, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.113.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251215 21:02:48.841312 4419 init.cc:377] Logged in from keytab as oryx/127.28.113.2@KRBTEST.COM (short username oryx)
I20251215 21:02:48.842268 4419 webserver.cc:492] Webserver started at http://127.28.113.2:41927/ using document root <none> and password file <none>
I20251215 21:02:48.842794 4419 fs_manager.cc:362] Metadata directory not provided
I20251215 21:02:48.842850 4419 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251215 21:02:48.846203 4419 fs_manager.cc:714] Time spent opening directory manager: real 0.002s user 0.001s sys 0.000s
I20251215 21:02:48.847914 4434 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251215 21:02:48.848881 4419 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20251215 21:02:48.848973 4419 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765832506145157-29124-0/minicluster-data/ts-1/data,/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765832506145157-29124-0/minicluster-data/ts-1/wal
uuid: "f17cfa3c3e5041c092728f920f98dec6"
format_stamp: "Formatted at 2025-12-15 21:02:45 on dist-test-slave-bjsf"
server_key: "1ce2ba8517fd183ff29eb281e5234123"
server_key_iv: "1557c0dc72887429de987e9aea01bad3"
server_key_version: "encryptionkey@0"
I20251215 21:02:48.849404 4419 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765832506145157-29124-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765832506145157-29124-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765832506145157-29124-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251215 21:02:48.874028 4419 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251215 21:02:48.876788 4419 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251215 21:02:48.876981 4419 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251215 21:02:48.877542 4419 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251215 21:02:48.878459 4419 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251215 21:02:48.878510 4419 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251215 21:02:48.878569 4419 ts_tablet_manager.cc:616] Registered 0 tablets
I20251215 21:02:48.878604 4419 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251215 21:02:48.887778 4419 rpc_server.cc:307] RPC server started. Bound to: 127.28.113.2:32787
I20251215 21:02:48.887830 4547 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.28.113.2:32787 every 8 connection(s)
I20251215 21:02:48.888804 4419 server_base.cc:1179] Dumped server information to /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765832506145157-29124-0/minicluster-data/ts-1/data/info.pb
I20251215 21:02:48.890038 29124 external_mini_cluster.cc:1428] Started /tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu as pid 4419
I20251215 21:02:48.890237 29124 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu with pid 3750
I20251215 21:02:48.897115 29124 external_mini_cluster.cc:1366] Running /tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu
/tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765832506145157-29124-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765832506145157-29124-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765832506145157-29124-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765832506145157-29124-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.113.3:35071
--local_ip_for_outbound_sockets=127.28.113.3
--tserver_master_addrs=127.28.113.62:41229
--webserver_port=42413
--webserver_interface=127.28.113.3
--keytab_file=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765832506145157-29124-0/krb5kdc/oryx.keytab
--principal=oryx/127.28.113.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.28.113.20:39895
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.28.113.62:41229,127.28.113.61:39207
--tserver_master_addrs=127.28.113.62:41229,127.28.113.61:39207,127.28.113.60:34031 with env {KRB5CCNAME=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765832506145157-29124-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765832506145157-29124-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765832506145157-29124-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Dec 15 21:02:48 dist-test-slave-bjsf krb5kdc[3366](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765832568, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.113.2@KRBTEST.COM for oryx/127.28.113.60@KRBTEST.COM
Dec 15 21:02:48 dist-test-slave-bjsf krb5kdc[3366](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765832568, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.113.2@KRBTEST.COM for oryx/127.28.113.61@KRBTEST.COM
Dec 15 21:02:48 dist-test-slave-bjsf krb5kdc[3366](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765832568, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.113.2@KRBTEST.COM for oryx/127.28.113.62@KRBTEST.COM
I20251215 21:02:48.903574 4278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:02:48.892470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.113.2:57633 (local address 127.28.113.61:39207)
1215 21:02:48.892588 (+ 118us) server_negotiation.cc:207] Beginning negotiation
1215 21:02:48.892592 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1215 21:02:48.892604 (+ 12us) server_negotiation.cc:408] Connection header received
1215 21:02:48.892641 (+ 37us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1215 21:02:48.892644 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1215 21:02:48.892686 (+ 42us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1215 21:02:48.892760 (+ 74us) server_negotiation.cc:227] Negotiated authn=SASL
1215 21:02:48.894525 (+ 1765us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:02:48.895332 (+ 807us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1215 21:02:48.896269 (+ 937us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:02:48.896447 (+ 178us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:02:48.899594 (+ 3147us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1215 21:02:48.899616 (+ 22us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1215 21:02:48.899620 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1215 21:02:48.899652 (+ 32us) server_negotiation.cc:923] calling sasl_server_start()
1215 21:02:48.901460 (+ 1808us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1215 21:02:48.902028 (+ 568us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1215 21:02:48.902031 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1215 21:02:48.902034 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
1215 21:02:48.902083 (+ 49us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1215 21:02:48.902403 (+ 320us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1215 21:02:48.902407 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1215 21:02:48.902409 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1215 21:02:48.902628 (+ 219us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1215 21:02:48.902753 (+ 125us) server_negotiation.cc:1036] Waiting for connection context
1215 21:02:48.903302 (+ 549us) server_negotiation.cc:300] Negotiation successful
1215 21:02:48.903422 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20251215 21:02:48.903720 4281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:02:48.892093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.113.2:54613 (local address 127.28.113.60:34031)
1215 21:02:48.892223 (+ 130us) server_negotiation.cc:207] Beginning negotiation
1215 21:02:48.892227 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1215 21:02:48.892243 (+ 16us) server_negotiation.cc:408] Connection header received
1215 21:02:48.892556 (+ 313us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1215 21:02:48.892560 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1215 21:02:48.892614 (+ 54us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1215 21:02:48.892664 (+ 50us) server_negotiation.cc:227] Negotiated authn=SASL
1215 21:02:48.894495 (+ 1831us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:02:48.895224 (+ 729us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1215 21:02:48.896110 (+ 886us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:02:48.896297 (+ 187us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:02:48.900008 (+ 3711us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1215 21:02:48.900028 (+ 20us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1215 21:02:48.900031 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1215 21:02:48.900060 (+ 29us) server_negotiation.cc:923] calling sasl_server_start()
1215 21:02:48.902186 (+ 2126us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1215 21:02:48.902655 (+ 469us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1215 21:02:48.902659 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1215 21:02:48.902661 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1215 21:02:48.902708 (+ 47us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1215 21:02:48.902946 (+ 238us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1215 21:02:48.902949 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1215 21:02:48.902951 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1215 21:02:48.903106 (+ 155us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1215 21:02:48.903196 (+ 90us) server_negotiation.cc:1036] Waiting for connection context
1215 21:02:48.903494 (+ 298us) server_negotiation.cc:300] Negotiation successful
1215 21:02:48.903595 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20251215 21:02:48.904649 4552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:02:48.890806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.113.61:39207 (local address 127.28.113.2:57633)
1215 21:02:48.891310 (+ 504us) negotiation.cc:107] Waiting for socket to connect
1215 21:02:48.891354 (+ 44us) client_negotiation.cc:174] Beginning negotiation
1215 21:02:48.892338 (+ 984us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1215 21:02:48.892949 (+ 611us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1215 21:02:48.892960 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
1215 21:02:48.893565 (+ 605us) client_negotiation.cc:189] Negotiated authn=SASL
1215 21:02:48.894335 (+ 770us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:02:48.894352 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:02:48.895543 (+ 1191us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1215 21:02:48.895547 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1215 21:02:48.896128 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:02:48.896139 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:02:48.896662 (+ 523us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:02:48.897501 (+ 839us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1215 21:02:48.897519 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
1215 21:02:48.899436 (+ 1917us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1215 21:02:48.901589 (+ 2153us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1215 21:02:48.901596 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1215 21:02:48.901608 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
1215 21:02:48.901915 (+ 307us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1215 21:02:48.902186 (+ 271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1215 21:02:48.902189 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1215 21:02:48.902191 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1215 21:02:48.902297 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1215 21:02:48.902745 (+ 448us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1215 21:02:48.902752 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1215 21:02:48.903023 (+ 271us) client_negotiation.cc:769] Sending connection context
1215 21:02:48.903759 (+ 736us) client_negotiation.cc:240] Negotiation successful
1215 21:02:48.903885 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":293,"spinlock_wait_cycles":10880,"thread_start_us":112,"threads_started":1}
I20251215 21:02:48.904654 4553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:02:48.891680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.113.60:34031 (local address 127.28.113.2:54613)
1215 21:02:48.891917 (+ 237us) negotiation.cc:107] Waiting for socket to connect
1215 21:02:48.891930 (+ 13us) client_negotiation.cc:174] Beginning negotiation
1215 21:02:48.892350 (+ 420us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1215 21:02:48.893262 (+ 912us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1215 21:02:48.893265 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1215 21:02:48.893566 (+ 301us) client_negotiation.cc:189] Negotiated authn=SASL
1215 21:02:48.894338 (+ 772us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:02:48.894351 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:02:48.895378 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1215 21:02:48.895381 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1215 21:02:48.895982 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:02:48.895991 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:02:48.896661 (+ 670us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:02:48.897230 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1215 21:02:48.897263 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
1215 21:02:48.899853 (+ 2590us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1215 21:02:48.902284 (+ 2431us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1215 21:02:48.902287 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1215 21:02:48.902289 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1215 21:02:48.902560 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1215 21:02:48.902797 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1215 21:02:48.902800 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1215 21:02:48.902802 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1215 21:02:48.902858 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1215 21:02:48.903220 (+ 362us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1215 21:02:48.903224 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1215 21:02:48.903320 (+ 96us) client_negotiation.cc:769] Sending connection context
1215 21:02:48.903475 (+ 155us) client_negotiation.cc:240] Negotiation successful
1215 21:02:48.903759 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":168,"spinlock_wait_cycles":6016,"thread_start_us":81,"threads_started":1}
I20251215 21:02:48.905463 4554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:02:48.893072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.113.62:41229 (local address 127.28.113.2:36137)
1215 21:02:48.894573 (+ 1501us) negotiation.cc:107] Waiting for socket to connect
1215 21:02:48.894585 (+ 12us) client_negotiation.cc:174] Beginning negotiation
1215 21:02:48.894695 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1215 21:02:48.895014 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1215 21:02:48.895017 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1215 21:02:48.895282 (+ 265us) client_negotiation.cc:189] Negotiated authn=SASL
1215 21:02:48.895570 (+ 288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:02:48.895576 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:02:48.896553 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1215 21:02:48.896556 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1215 21:02:48.897547 (+ 991us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:02:48.897557 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:02:48.897982 (+ 425us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:02:48.898501 (+ 519us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1215 21:02:48.898516 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
1215 21:02:48.899882 (+ 1366us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1215 21:02:48.902218 (+ 2336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1215 21:02:48.902222 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1215 21:02:48.902225 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
1215 21:02:48.902507 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1215 21:02:48.903238 (+ 731us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1215 21:02:48.903241 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1215 21:02:48.903244 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
1215 21:02:48.903305 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1215 21:02:48.905027 (+ 1722us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1215 21:02:48.905031 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1215 21:02:48.905111 (+ 80us) client_negotiation.cc:769] Sending connection context
1215 21:02:48.905209 (+ 98us) client_negotiation.cc:240] Negotiation successful
1215 21:02:48.905324 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1439,"spinlock_wait_cycles":3328,"thread_start_us":77,"threads_started":1}
I20251215 21:02:48.905851 4548 heartbeater.cc:344] Connected to a master server at 127.28.113.60:34031
I20251215 21:02:48.905917 4550 heartbeater.cc:344] Connected to a master server at 127.28.113.61:39207
I20251215 21:02:48.906104 4550 heartbeater.cc:461] Registering TS with master...
I20251215 21:02:48.906508 4280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:02:48.893300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.113.2:36137 (local address 127.28.113.62:41229)
1215 21:02:48.894372 (+ 1072us) server_negotiation.cc:207] Beginning negotiation
1215 21:02:48.894376 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1215 21:02:48.894640 (+ 264us) server_negotiation.cc:408] Connection header received
1215 21:02:48.894786 (+ 146us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1215 21:02:48.894789 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1215 21:02:48.894846 (+ 57us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1215 21:02:48.894895 (+ 49us) server_negotiation.cc:227] Negotiated authn=SASL
1215 21:02:48.895684 (+ 789us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:02:48.896425 (+ 741us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1215 21:02:48.897674 (+ 1249us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:02:48.897870 (+ 196us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:02:48.900030 (+ 2160us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1215 21:02:48.900051 (+ 21us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1215 21:02:48.900055 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1215 21:02:48.900085 (+ 30us) server_negotiation.cc:923] calling sasl_server_start()
1215 21:02:48.902115 (+ 2030us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1215 21:02:48.902811 (+ 696us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1215 21:02:48.902816 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1215 21:02:48.902818 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1215 21:02:48.902877 (+ 59us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1215 21:02:48.904708 (+ 1831us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1215 21:02:48.904712 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1215 21:02:48.904715 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
1215 21:02:48.904906 (+ 191us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1215 21:02:48.905034 (+ 128us) server_negotiation.cc:1036] Waiting for connection context
1215 21:02:48.906208 (+ 1174us) server_negotiation.cc:300] Negotiation successful
1215 21:02:48.906339 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":985}
I20251215 21:02:48.906602 4550 heartbeater.cc:507] Master 127.28.113.61:39207 requested a full tablet report, sending...
I20251215 21:02:48.906105 4548 heartbeater.cc:461] Registering TS with master...
I20251215 21:02:48.907097 4548 heartbeater.cc:507] Master 127.28.113.60:34031 requested a full tablet report, sending...
I20251215 21:02:48.907153 4549 heartbeater.cc:344] Connected to a master server at 127.28.113.62:41229
I20251215 21:02:48.907222 4549 heartbeater.cc:461] Registering TS with master...
I20251215 21:02:48.907385 4549 heartbeater.cc:507] Master 127.28.113.62:41229 requested a full tablet report, sending...
I20251215 21:02:48.907740 4032 ts_manager.cc:194] Registered new tserver with Master: f17cfa3c3e5041c092728f920f98dec6 (127.28.113.2:32787)
I20251215 21:02:48.907791 4216 ts_manager.cc:194] Registered new tserver with Master: f17cfa3c3e5041c092728f920f98dec6 (127.28.113.2:32787)
I20251215 21:02:48.908448 3414 ts_manager.cc:194] Re-registered known tserver with Master: f17cfa3c3e5041c092728f920f98dec6 (127.28.113.2:32787)
I20251215 21:02:48.908983 3414 master_service.cc:502] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.28.113.2@KRBTEST.COM'} at 127.28.113.2:36137
W20251215 21:02:49.007018 4555 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251215 21:02:49.007256 4555 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251215 21:02:49.007340 4555 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251215 21:02:49.010591 4555 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251215 21:02:49.010664 4555 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251215 21:02:49.010778 4555 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.113.3
I20251215 21:02:49.014979 4555 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.113.20:39895
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765832506145157-29124-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765832506145157-29124-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765832506145157-29124-0/krb5kdc/oryx.keytab
--principal=oryx/127.28.113.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.113.3:35071
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765832506145157-29124-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.113.3
--webserver_port=42413
--webserver_require_spnego=true
--tserver_master_addrs=127.28.113.62:41229,127.28.113.61:39207,127.28.113.60:34031
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4555
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.113.3
--log_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765832506145157-29124-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision dc741732bb6a40213e11a13869fad06a49a209a3
build type DEBUG
built by None at 15 Dec 2025 20:43:14 UTC on 24a791456cd2
build id 9530
I20251215 21:02:49.016064 4555 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251215 21:02:49.016907 4555 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251215 21:02:49.023275 4561 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251215 21:02:49.023324 4563 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251215 21:02:49.023324 4560 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251215 21:02:49.023890 4555 server_base.cc:1047] running on GCE node
I20251215 21:02:49.024266 4555 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251215 21:02:49.024792 4555 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251215 21:02:49.025966 4555 hybrid_clock.cc:648] HybridClock initialized: now 1765832569025956 us; error 26 us; skew 500 ppm
Dec 15 21:02:49 dist-test-slave-bjsf krb5kdc[3366](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765832569, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.113.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251215 21:02:49.028499 4555 init.cc:377] Logged in from keytab as oryx/127.28.113.3@KRBTEST.COM (short username oryx)
I20251215 21:02:49.029425 4555 webserver.cc:492] Webserver started at http://127.28.113.3:42413/ using document root <none> and password file <none>
I20251215 21:02:49.029906 4555 fs_manager.cc:362] Metadata directory not provided
I20251215 21:02:49.029968 4555 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251215 21:02:49.033499 4555 fs_manager.cc:714] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20251215 21:02:49.035490 4570 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251215 21:02:49.036497 4555 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20251215 21:02:49.036589 4555 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765832506145157-29124-0/minicluster-data/ts-2/data,/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765832506145157-29124-0/minicluster-data/ts-2/wal
uuid: "0a2363aab21d43f68f26fb68238a29f3"
format_stamp: "Formatted at 2025-12-15 21:02:45 on dist-test-slave-bjsf"
server_key: "93cec74eff54b765f102832ee805e7db"
server_key_iv: "b5d3a23f47d2d65786e8bb9f48a52513"
server_key_version: "encryptionkey@0"
I20251215 21:02:49.036981 4555 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765832506145157-29124-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765832506145157-29124-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765832506145157-29124-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251215 21:02:49.063061 4555 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251215 21:02:49.065716 4555 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251215 21:02:49.065874 4555 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251215 21:02:49.066401 4555 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251215 21:02:49.067268 4555 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251215 21:02:49.067312 4555 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251215 21:02:49.067353 4555 ts_tablet_manager.cc:616] Registered 0 tablets
I20251215 21:02:49.067384 4555 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251215 21:02:49.075745 4555 rpc_server.cc:307] RPC server started. Bound to: 127.28.113.3:35071
I20251215 21:02:49.075774 4683 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.28.113.3:35071 every 8 connection(s)
I20251215 21:02:49.076726 4555 server_base.cc:1179] Dumped server information to /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765832506145157-29124-0/minicluster-data/ts-2/data/info.pb
I20251215 21:02:49.083936 29124 external_mini_cluster.cc:1428] Started /tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu as pid 4555
Dec 15 21:02:49 dist-test-slave-bjsf krb5kdc[3366](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765832569, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.113.3@KRBTEST.COM for oryx/127.28.113.60@KRBTEST.COM
Dec 15 21:02:49 dist-test-slave-bjsf krb5kdc[3366](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765832569, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.113.3@KRBTEST.COM for oryx/127.28.113.61@KRBTEST.COM
Dec 15 21:02:49 dist-test-slave-bjsf krb5kdc[3366](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765832569, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.113.3@KRBTEST.COM for oryx/127.28.113.62@KRBTEST.COM
I20251215 21:02:49.091306 4281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:02:49.078342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.113.3:51755 (local address 127.28.113.60:34031)
1215 21:02:49.078473 (+ 131us) server_negotiation.cc:207] Beginning negotiation
1215 21:02:49.078477 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1215 21:02:49.079351 (+ 874us) server_negotiation.cc:408] Connection header received
1215 21:02:49.080324 (+ 973us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1215 21:02:49.080328 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1215 21:02:49.080386 (+ 58us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1215 21:02:49.080458 (+ 72us) server_negotiation.cc:227] Negotiated authn=SASL
1215 21:02:49.082075 (+ 1617us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:02:49.082838 (+ 763us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1215 21:02:49.083451 (+ 613us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:02:49.083640 (+ 189us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:02:49.086551 (+ 2911us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1215 21:02:49.086571 (+ 20us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1215 21:02:49.086575 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1215 21:02:49.086604 (+ 29us) server_negotiation.cc:923] calling sasl_server_start()
1215 21:02:49.088577 (+ 1973us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1215 21:02:49.089392 (+ 815us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1215 21:02:49.089396 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1215 21:02:49.089397 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1215 21:02:49.089468 (+ 71us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1215 21:02:49.089854 (+ 386us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1215 21:02:49.089858 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1215 21:02:49.089860 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1215 21:02:49.090044 (+ 184us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1215 21:02:49.090117 (+ 73us) server_negotiation.cc:1036] Waiting for connection context
1215 21:02:49.091044 (+ 927us) server_negotiation.cc:300] Negotiation successful
1215 21:02:49.091149 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20251215 21:02:49.091812 4690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:02:49.078645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.113.61:39207 (local address 127.28.113.3:36531)
1215 21:02:49.079823 (+ 1178us) negotiation.cc:107] Waiting for socket to connect
1215 21:02:49.079833 (+ 10us) client_negotiation.cc:174] Beginning negotiation
1215 21:02:49.080146 (+ 313us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1215 21:02:49.080835 (+ 689us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1215 21:02:49.080839 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
1215 21:02:49.081254 (+ 415us) client_negotiation.cc:189] Negotiated authn=SASL
1215 21:02:49.081901 (+ 647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:02:49.081910 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:02:49.083558 (+ 1648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1215 21:02:49.083561 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1215 21:02:49.084113 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:02:49.084123 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:02:49.084257 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:02:49.084722 (+ 465us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1215 21:02:49.084737 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
1215 21:02:49.086703 (+ 1966us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1215 21:02:49.089250 (+ 2547us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1215 21:02:49.089254 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1215 21:02:49.089256 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1215 21:02:49.089460 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1215 21:02:49.089696 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1215 21:02:49.089699 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1215 21:02:49.089701 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1215 21:02:49.089761 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1215 21:02:49.090188 (+ 427us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1215 21:02:49.090194 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1215 21:02:49.090471 (+ 277us) client_negotiation.cc:769] Sending connection context
1215 21:02:49.090688 (+ 217us) client_negotiation.cc:240] Negotiation successful
1215 21:02:49.090974 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1108,"mutex_wait_us":50,"spinlock_wait_cycles":87936,"thread_start_us":147,"threads_started":1}
I20251215 21:02:49.092690 4278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:02:49.078339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.113.3:36531 (local address 127.28.113.61:39207)
1215 21:02:49.078473 (+ 134us) server_negotiation.cc:207] Beginning negotiation
1215 21:02:49.078477 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1215 21:02:49.079942 (+ 1465us) server_negotiation.cc:408] Connection header received
1215 21:02:49.080664 (+ 722us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1215 21:02:49.080668 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1215 21:02:49.080718 (+ 50us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1215 21:02:49.080797 (+ 79us) server_negotiation.cc:227] Negotiated authn=SASL
1215 21:02:49.082974 (+ 2177us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:02:49.083439 (+ 465us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1215 21:02:49.084277 (+ 838us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:02:49.084933 (+ 656us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:02:49.087248 (+ 2315us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1215 21:02:49.087263 (+ 15us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1215 21:02:49.087266 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1215 21:02:49.087291 (+ 25us) server_negotiation.cc:923] calling sasl_server_start()
1215 21:02:49.089139 (+ 1848us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1215 21:02:49.089561 (+ 422us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1215 21:02:49.089565 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1215 21:02:49.089567 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1215 21:02:49.089612 (+ 45us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1215 21:02:49.089847 (+ 235us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1215 21:02:49.089850 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1215 21:02:49.089852 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1215 21:02:49.090019 (+ 167us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1215 21:02:49.090116 (+ 97us) server_negotiation.cc:1036] Waiting for connection context
1215 21:02:49.092430 (+ 2314us) server_negotiation.cc:300] Negotiation successful
1215 21:02:49.092544 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20251215 21:02:49.091812 4689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:02:49.078645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.113.60:34031 (local address 127.28.113.3:51755)
1215 21:02:49.079223 (+ 578us) negotiation.cc:107] Waiting for socket to connect
1215 21:02:49.079262 (+ 39us) client_negotiation.cc:174] Beginning negotiation
1215 21:02:49.080131 (+ 869us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1215 21:02:49.080634 (+ 503us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1215 21:02:49.080642 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
1215 21:02:49.081244 (+ 602us) client_negotiation.cc:189] Negotiated authn=SASL
1215 21:02:49.081901 (+ 657us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:02:49.081910 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:02:49.082954 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1215 21:02:49.082956 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1215 21:02:49.083342 (+ 386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:02:49.083348 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:02:49.083516 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:02:49.084558 (+ 1042us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1215 21:02:49.084589 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
1215 21:02:49.086381 (+ 1792us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1215 21:02:49.088950 (+ 2569us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1215 21:02:49.088955 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1215 21:02:49.088967 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
1215 21:02:49.089273 (+ 306us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1215 21:02:49.089616 (+ 343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1215 21:02:49.089620 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1215 21:02:49.089622 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1215 21:02:49.089735 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1215 21:02:49.090189 (+ 454us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1215 21:02:49.090197 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1215 21:02:49.090534 (+ 337us) client_negotiation.cc:769] Sending connection context
1215 21:02:49.091490 (+ 956us) client_negotiation.cc:240] Negotiation successful
1215 21:02:49.091607 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":392,"spinlock_wait_cycles":10496,"thread_start_us":106,"threads_started":1}
I20251215 21:02:49.093089 4688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:02:49.078702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.113.62:41229 (local address 127.28.113.3:60855)
1215 21:02:49.079576 (+ 874us) negotiation.cc:107] Waiting for socket to connect
1215 21:02:49.079587 (+ 11us) client_negotiation.cc:174] Beginning negotiation
1215 21:02:49.080146 (+ 559us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1215 21:02:49.080636 (+ 490us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1215 21:02:49.080643 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
1215 21:02:49.081431 (+ 788us) client_negotiation.cc:189] Negotiated authn=SASL
1215 21:02:49.081874 (+ 443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:02:49.081886 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:02:49.083790 (+ 1904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1215 21:02:49.083793 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1215 21:02:49.084302 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:02:49.084310 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:02:49.084398 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:02:49.085319 (+ 921us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1215 21:02:49.085334 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
1215 21:02:49.088815 (+ 3481us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1215 21:02:49.090686 (+ 1871us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1215 21:02:49.090689 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1215 21:02:49.090691 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1215 21:02:49.090888 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1215 21:02:49.092252 (+ 1364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1215 21:02:49.092256 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1215 21:02:49.092257 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1215 21:02:49.092311 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1215 21:02:49.092645 (+ 334us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1215 21:02:49.092648 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1215 21:02:49.092725 (+ 77us) client_negotiation.cc:769] Sending connection context
1215 21:02:49.092818 (+ 93us) client_negotiation.cc:240] Negotiation successful
1215 21:02:49.092948 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":811,"spinlock_wait_cycles":45312,"thread_start_us":150,"threads_started":1}
I20251215 21:02:49.093370 4280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:02:49.078867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.113.3:60855 (local address 127.28.113.62:41229)
1215 21:02:49.078994 (+ 127us) server_negotiation.cc:207] Beginning negotiation
1215 21:02:49.078999 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1215 21:02:49.079672 (+ 673us) server_negotiation.cc:408] Connection header received
1215 21:02:49.080433 (+ 761us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1215 21:02:49.080436 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1215 21:02:49.080482 (+ 46us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1215 21:02:49.080548 (+ 66us) server_negotiation.cc:227] Negotiated authn=SASL
1215 21:02:49.082029 (+ 1481us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:02:49.082838 (+ 809us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1215 21:02:49.085638 (+ 2800us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:02:49.085840 (+ 202us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:02:49.088948 (+ 3108us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1215 21:02:49.088967 (+ 19us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1215 21:02:49.088970 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1215 21:02:49.088995 (+ 25us) server_negotiation.cc:923] calling sasl_server_start()
1215 21:02:49.090587 (+ 1592us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1215 21:02:49.091606 (+ 1019us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1215 21:02:49.091609 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1215 21:02:49.091611 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1215 21:02:49.092168 (+ 557us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1215 21:02:49.092400 (+ 232us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1215 21:02:49.092403 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1215 21:02:49.092405 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1215 21:02:49.092568 (+ 163us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1215 21:02:49.092654 (+ 86us) server_negotiation.cc:1036] Waiting for connection context
1215 21:02:49.093127 (+ 473us) server_negotiation.cc:300] Negotiation successful
1215 21:02:49.093244 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20251215 21:02:49.094295 4685 heartbeater.cc:344] Connected to a master server at 127.28.113.62:41229
I20251215 21:02:49.094329 4684 heartbeater.cc:344] Connected to a master server at 127.28.113.60:34031
I20251215 21:02:49.094336 4686 heartbeater.cc:344] Connected to a master server at 127.28.113.61:39207
I20251215 21:02:49.094529 4685 heartbeater.cc:461] Registering TS with master...
I20251215 21:02:49.094594 4686 heartbeater.cc:461] Registering TS with master...
I20251215 21:02:49.095144 4686 heartbeater.cc:507] Master 127.28.113.61:39207 requested a full tablet report, sending...
I20251215 21:02:49.094529 4684 heartbeater.cc:461] Registering TS with master...
I20251215 21:02:49.095355 4684 heartbeater.cc:507] Master 127.28.113.60:34031 requested a full tablet report, sending...
I20251215 21:02:49.096266 4032 ts_manager.cc:194] Registered new tserver with Master: 0a2363aab21d43f68f26fb68238a29f3 (127.28.113.3:35071)
I20251215 21:02:49.096279 4216 ts_manager.cc:194] Registered new tserver with Master: 0a2363aab21d43f68f26fb68238a29f3 (127.28.113.3:35071)
I20251215 21:02:49.095144 4685 heartbeater.cc:507] Master 127.28.113.62:41229 requested a full tablet report, sending...
I20251215 21:02:49.097564 4700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:02:49.087089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46400 (local address 127.28.113.60:34031)
1215 21:02:49.087332 (+ 243us) server_negotiation.cc:207] Beginning negotiation
1215 21:02:49.087337 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1215 21:02:49.087877 (+ 540us) server_negotiation.cc:408] Connection header received
1215 21:02:49.087918 (+ 41us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1215 21:02:49.087921 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1215 21:02:49.087990 (+ 69us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1215 21:02:49.088077 (+ 87us) server_negotiation.cc:227] Negotiated authn=SASL
1215 21:02:49.089927 (+ 1850us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:02:49.090613 (+ 686us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1215 21:02:49.091689 (+ 1076us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:02:49.091865 (+ 176us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:02:49.093281 (+ 1416us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1215 21:02:49.093293 (+ 12us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1215 21:02:49.093296 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1215 21:02:49.093313 (+ 17us) server_negotiation.cc:923] calling sasl_server_start()
1215 21:02:49.095774 (+ 2461us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1215 21:02:49.096230 (+ 456us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1215 21:02:49.096234 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1215 21:02:49.096236 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1215 21:02:49.096285 (+ 49us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1215 21:02:49.096532 (+ 247us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1215 21:02:49.096535 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1215 21:02:49.096538 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
1215 21:02:49.096697 (+ 159us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1215 21:02:49.096828 (+ 131us) server_negotiation.cc:1036] Waiting for connection context
1215 21:02:49.097316 (+ 488us) server_negotiation.cc:300] Negotiation successful
1215 21:02:49.097424 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":163,"thread_start_us":62,"threads_started":1}
I20251215 21:02:49.097563 3414 ts_manager.cc:194] Re-registered known tserver with Master: 0a2363aab21d43f68f26fb68238a29f3 (127.28.113.3:35071)
I20251215 21:02:49.098197 3414 master_service.cc:502] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.28.113.3@KRBTEST.COM'} at 127.28.113.3:60855
I20251215 21:02:49.098773 4696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:02:49.085785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36730 (local address 127.28.113.62:41229)
1215 21:02:49.087184 (+ 1399us) server_negotiation.cc:207] Beginning negotiation
1215 21:02:49.087188 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1215 21:02:49.087201 (+ 13us) server_negotiation.cc:408] Connection header received
1215 21:02:49.087237 (+ 36us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1215 21:02:49.087240 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1215 21:02:49.087286 (+ 46us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1215 21:02:49.087379 (+ 93us) server_negotiation.cc:227] Negotiated authn=SASL
1215 21:02:49.089127 (+ 1748us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:02:49.090701 (+ 1574us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1215 21:02:49.092879 (+ 2178us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:02:49.093185 (+ 306us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:02:49.095187 (+ 2002us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1215 21:02:49.095206 (+ 19us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1215 21:02:49.095209 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1215 21:02:49.095234 (+ 25us) server_negotiation.cc:923] calling sasl_server_start()
1215 21:02:49.096856 (+ 1622us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1215 21:02:49.097364 (+ 508us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1215 21:02:49.097367 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1215 21:02:49.097369 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1215 21:02:49.097413 (+ 44us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1215 21:02:49.098064 (+ 651us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1215 21:02:49.098068 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1215 21:02:49.098070 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1215 21:02:49.098224 (+ 154us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1215 21:02:49.098343 (+ 119us) server_negotiation.cc:1036] Waiting for connection context
1215 21:02:49.098526 (+ 183us) server_negotiation.cc:300] Negotiation successful
1215 21:02:49.098628 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1321,"thread_start_us":69,"threads_started":1}
I20251215 21:02:49.100972 3414 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36730:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251215 21:02:49.103137 3414 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251215 21:02:49.112903 4698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:02:49.086653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56098 (local address 127.28.113.61:39207)
1215 21:02:49.087570 (+ 917us) server_negotiation.cc:207] Beginning negotiation
1215 21:02:49.087573 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1215 21:02:49.092157 (+ 4584us) server_negotiation.cc:408] Connection header received
1215 21:02:49.092195 (+ 38us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1215 21:02:49.092198 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1215 21:02:49.092445 (+ 247us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1215 21:02:49.092517 (+ 72us) server_negotiation.cc:227] Negotiated authn=SASL
1215 21:02:49.100059 (+ 7542us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:02:49.104663 (+ 4604us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1215 21:02:49.105436 (+ 773us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:02:49.105619 (+ 183us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:02:49.108722 (+ 3103us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1215 21:02:49.108738 (+ 16us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1215 21:02:49.108741 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1215 21:02:49.108767 (+ 26us) server_negotiation.cc:923] calling sasl_server_start()
1215 21:02:49.110574 (+ 1807us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1215 21:02:49.110980 (+ 406us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1215 21:02:49.110984 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1215 21:02:49.110985 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1215 21:02:49.111035 (+ 50us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1215 21:02:49.111516 (+ 481us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1215 21:02:49.111520 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1215 21:02:49.111521 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1215 21:02:49.111707 (+ 186us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1215 21:02:49.111835 (+ 128us) server_negotiation.cc:1036] Waiting for connection context
1215 21:02:49.112615 (+ 780us) server_negotiation.cc:300] Negotiation successful
1215 21:02:49.112736 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":849,"thread_start_us":65,"threads_started":1}
I20251215 21:02:49.118758 4711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:02:49.114706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.113.3:35071 (local address 127.0.0.1:51486)
1215 21:02:49.114946 (+ 240us) negotiation.cc:107] Waiting for socket to connect
1215 21:02:49.114960 (+ 14us) client_negotiation.cc:174] Beginning negotiation
1215 21:02:49.115043 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1215 21:02:49.116078 (+ 1035us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1215 21:02:49.116081 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1215 21:02:49.116097 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1215 21:02:49.116281 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:02:49.116286 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:02:49.117299 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1215 21:02:49.117302 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1215 21:02:49.118192 (+ 890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:02:49.118201 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:02:49.118408 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:02:49.118427 (+ 19us) client_negotiation.cc:769] Sending connection context
1215 21:02:49.118480 (+ 53us) client_negotiation.cc:240] Negotiation successful
1215 21:02:49.118534 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":94,"threads_started":1}
I20251215 21:02:49.119103 4713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:02:49.115001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.113.1:35255 (local address 127.0.0.1:45060)
1215 21:02:49.115318 (+ 317us) negotiation.cc:107] Waiting for socket to connect
1215 21:02:49.115331 (+ 13us) client_negotiation.cc:174] Beginning negotiation
1215 21:02:49.115980 (+ 649us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1215 21:02:49.116426 (+ 446us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1215 21:02:49.116429 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1215 21:02:49.116447 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1215 21:02:49.116671 (+ 224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:02:49.116676 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:02:49.117968 (+ 1292us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1215 21:02:49.117972 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1215 21:02:49.118792 (+ 820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:02:49.118802 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:02:49.118878 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:02:49.118894 (+ 16us) client_negotiation.cc:769] Sending connection context
1215 21:02:49.118932 (+ 38us) client_negotiation.cc:240] Negotiation successful
1215 21:02:49.118986 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":89,"threads_started":1}
I20251215 21:02:49.119390 4714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:02:49.115308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.113.2:32787 (local address 127.0.0.1:34290)
1215 21:02:49.116156 (+ 848us) negotiation.cc:107] Waiting for socket to connect
1215 21:02:49.116168 (+ 12us) client_negotiation.cc:174] Beginning negotiation
1215 21:02:49.116234 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1215 21:02:49.116929 (+ 695us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1215 21:02:49.116933 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
1215 21:02:49.116949 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1215 21:02:49.117164 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:02:49.117170 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:02:49.118063 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1215 21:02:49.118067 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1215 21:02:49.118844 (+ 777us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:02:49.118854 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:02:49.118957 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:02:49.118972 (+ 15us) client_negotiation.cc:769] Sending connection context
1215 21:02:49.119022 (+ 50us) client_negotiation.cc:240] Negotiation successful
1215 21:02:49.119078 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":788,"thread_start_us":85,"threads_started":1}
I20251215 21:02:49.119920 4716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:02:49.115677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45060 (local address 127.28.113.1:35255)
1215 21:02:49.116011 (+ 334us) server_negotiation.cc:207] Beginning negotiation
1215 21:02:49.116017 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1215 21:02:49.116037 (+ 20us) server_negotiation.cc:408] Connection header received
1215 21:02:49.116126 (+ 89us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1215 21:02:49.116131 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1215 21:02:49.116285 (+ 154us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1215 21:02:49.116443 (+ 158us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1215 21:02:49.116788 (+ 345us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:02:49.117845 (+ 1057us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1215 21:02:49.119078 (+ 1233us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:02:49.119567 (+ 489us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:02:49.119685 (+ 118us) server_negotiation.cc:1036] Waiting for connection context
1215 21:02:49.119744 (+ 59us) server_negotiation.cc:300] Negotiation successful
1215 21:02:49.119810 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":213,"thread_start_us":108,"threads_started":1}
I20251215 21:02:49.119947 4712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:02:49.114841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51486 (local address 127.28.113.3:35071)
1215 21:02:49.115460 (+ 619us) server_negotiation.cc:207] Beginning negotiation
1215 21:02:49.115465 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1215 21:02:49.115484 (+ 19us) server_negotiation.cc:408] Connection header received
1215 21:02:49.115552 (+ 68us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1215 21:02:49.115556 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1215 21:02:49.115738 (+ 182us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1215 21:02:49.115882 (+ 144us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1215 21:02:49.116395 (+ 513us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:02:49.117216 (+ 821us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1215 21:02:49.119155 (+ 1939us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:02:49.119651 (+ 496us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:02:49.119735 (+ 84us) server_negotiation.cc:1036] Waiting for connection context
1215 21:02:49.119786 (+ 51us) server_negotiation.cc:300] Negotiation successful
1215 21:02:49.119846 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":517,"thread_start_us":95,"threads_started":1}
I20251215 21:02:49.121018 4715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:02:49.115472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34290 (local address 127.28.113.2:32787)
1215 21:02:49.116458 (+ 986us) server_negotiation.cc:207] Beginning negotiation
1215 21:02:49.116465 (+ 7us) server_negotiation.cc:400] Waiting for connection header
1215 21:02:49.116484 (+ 19us) server_negotiation.cc:408] Connection header received
1215 21:02:49.116558 (+ 74us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1215 21:02:49.116563 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1215 21:02:49.116678 (+ 115us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1215 21:02:49.116786 (+ 108us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1215 21:02:49.117289 (+ 503us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:02:49.117945 (+ 656us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1215 21:02:49.120107 (+ 2162us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:02:49.120598 (+ 491us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:02:49.120750 (+ 152us) server_negotiation.cc:1036] Waiting for connection context
1215 21:02:49.120807 (+ 57us) server_negotiation.cc:300] Negotiation successful
1215 21:02:49.120895 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":886,"thread_start_us":79,"threads_started":1}
I20251215 21:02:49.122633 4346 tablet_service.cc:1505] Processing CreateTablet for tablet f2bc9cd03afe4441a099e2054894818a (DEFAULT_TABLE table=test-table [id=e56c8a99e1e347eaa991df11a1f728b6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251215 21:02:49.122650 4618 tablet_service.cc:1505] Processing CreateTablet for tablet f2bc9cd03afe4441a099e2054894818a (DEFAULT_TABLE table=test-table [id=e56c8a99e1e347eaa991df11a1f728b6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251215 21:02:49.123433 4346 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f2bc9cd03afe4441a099e2054894818a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 21:02:49.123497 4618 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f2bc9cd03afe4441a099e2054894818a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 21:02:49.123332 4482 tablet_service.cc:1505] Processing CreateTablet for tablet f2bc9cd03afe4441a099e2054894818a (DEFAULT_TABLE table=test-table [id=e56c8a99e1e347eaa991df11a1f728b6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251215 21:02:49.124182 4482 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f2bc9cd03afe4441a099e2054894818a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 21:02:49.129413 4717 tablet_bootstrap.cc:492] T f2bc9cd03afe4441a099e2054894818a P 6956a9660a3b4d42867878501794f2d5: Bootstrap starting.
I20251215 21:02:49.131628 4717 tablet_bootstrap.cc:654] T f2bc9cd03afe4441a099e2054894818a P 6956a9660a3b4d42867878501794f2d5: Neither blocks nor log segments found. Creating new log.
I20251215 21:02:49.132283 4718 tablet_bootstrap.cc:492] T f2bc9cd03afe4441a099e2054894818a P 0a2363aab21d43f68f26fb68238a29f3: Bootstrap starting.
I20251215 21:02:49.132319 4717 log.cc:826] T f2bc9cd03afe4441a099e2054894818a P 6956a9660a3b4d42867878501794f2d5: Log is configured to *not* fsync() on all Append() calls
I20251215 21:02:49.132875 4719 tablet_bootstrap.cc:492] T f2bc9cd03afe4441a099e2054894818a P f17cfa3c3e5041c092728f920f98dec6: Bootstrap starting.
I20251215 21:02:49.134183 4717 tablet_bootstrap.cc:492] T f2bc9cd03afe4441a099e2054894818a P 6956a9660a3b4d42867878501794f2d5: No bootstrap required, opened a new log
I20251215 21:02:49.134378 4718 tablet_bootstrap.cc:654] T f2bc9cd03afe4441a099e2054894818a P 0a2363aab21d43f68f26fb68238a29f3: Neither blocks nor log segments found. Creating new log.
I20251215 21:02:49.134372 4717 ts_tablet_manager.cc:1403] T f2bc9cd03afe4441a099e2054894818a P 6956a9660a3b4d42867878501794f2d5: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20251215 21:02:49.134848 4719 tablet_bootstrap.cc:654] T f2bc9cd03afe4441a099e2054894818a P f17cfa3c3e5041c092728f920f98dec6: Neither blocks nor log segments found. Creating new log.
I20251215 21:02:49.135100 4718 log.cc:826] T f2bc9cd03afe4441a099e2054894818a P 0a2363aab21d43f68f26fb68238a29f3: Log is configured to *not* fsync() on all Append() calls
I20251215 21:02:49.135560 4719 log.cc:826] T f2bc9cd03afe4441a099e2054894818a P f17cfa3c3e5041c092728f920f98dec6: Log is configured to *not* fsync() on all Append() calls
I20251215 21:02:49.136879 4718 tablet_bootstrap.cc:492] T f2bc9cd03afe4441a099e2054894818a P 0a2363aab21d43f68f26fb68238a29f3: No bootstrap required, opened a new log
I20251215 21:02:49.137066 4718 ts_tablet_manager.cc:1403] T f2bc9cd03afe4441a099e2054894818a P 0a2363aab21d43f68f26fb68238a29f3: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20251215 21:02:49.137208 4719 tablet_bootstrap.cc:492] T f2bc9cd03afe4441a099e2054894818a P f17cfa3c3e5041c092728f920f98dec6: No bootstrap required, opened a new log
I20251215 21:02:49.137380 4719 ts_tablet_manager.cc:1403] T f2bc9cd03afe4441a099e2054894818a P f17cfa3c3e5041c092728f920f98dec6: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20251215 21:02:49.137481 4717 raft_consensus.cc:359] T f2bc9cd03afe4441a099e2054894818a P 6956a9660a3b4d42867878501794f2d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a2363aab21d43f68f26fb68238a29f3" member_type: VOTER last_known_addr { host: "127.28.113.3" port: 35071 } } peers { permanent_uuid: "6956a9660a3b4d42867878501794f2d5" member_type: VOTER last_known_addr { host: "127.28.113.1" port: 35255 } } peers { permanent_uuid: "f17cfa3c3e5041c092728f920f98dec6" member_type: VOTER last_known_addr { host: "127.28.113.2" port: 32787 } }
I20251215 21:02:49.137696 4717 raft_consensus.cc:385] T f2bc9cd03afe4441a099e2054894818a P 6956a9660a3b4d42867878501794f2d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251215 21:02:49.137753 4717 raft_consensus.cc:740] T f2bc9cd03afe4441a099e2054894818a P 6956a9660a3b4d42867878501794f2d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6956a9660a3b4d42867878501794f2d5, State: Initialized, Role: FOLLOWER
I20251215 21:02:49.138195 4717 consensus_queue.cc:260] T f2bc9cd03afe4441a099e2054894818a P 6956a9660a3b4d42867878501794f2d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a2363aab21d43f68f26fb68238a29f3" member_type: VOTER last_known_addr { host: "127.28.113.3" port: 35071 } } peers { permanent_uuid: "6956a9660a3b4d42867878501794f2d5" member_type: VOTER last_known_addr { host: "127.28.113.1" port: 35255 } } peers { permanent_uuid: "f17cfa3c3e5041c092728f920f98dec6" member_type: VOTER last_known_addr { host: "127.28.113.2" port: 32787 } }
I20251215 21:02:49.139055 4717 ts_tablet_manager.cc:1434] T f2bc9cd03afe4441a099e2054894818a P 6956a9660a3b4d42867878501794f2d5: Time spent starting tablet: real 0.005s user 0.001s sys 0.004s
I20251215 21:02:49.139276 4413 heartbeater.cc:499] Master 127.28.113.62:41229 was elected leader, sending a full tablet report...
I20251215 21:02:49.139766 4718 raft_consensus.cc:359] T f2bc9cd03afe4441a099e2054894818a P 0a2363aab21d43f68f26fb68238a29f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a2363aab21d43f68f26fb68238a29f3" member_type: VOTER last_known_addr { host: "127.28.113.3" port: 35071 } } peers { permanent_uuid: "6956a9660a3b4d42867878501794f2d5" member_type: VOTER last_known_addr { host: "127.28.113.1" port: 35255 } } peers { permanent_uuid: "f17cfa3c3e5041c092728f920f98dec6" member_type: VOTER last_known_addr { host: "127.28.113.2" port: 32787 } }
I20251215 21:02:49.139978 4718 raft_consensus.cc:385] T f2bc9cd03afe4441a099e2054894818a P 0a2363aab21d43f68f26fb68238a29f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251215 21:02:49.140034 4718 raft_consensus.cc:740] T f2bc9cd03afe4441a099e2054894818a P 0a2363aab21d43f68f26fb68238a29f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a2363aab21d43f68f26fb68238a29f3, State: Initialized, Role: FOLLOWER
I20251215 21:02:49.140164 4719 raft_consensus.cc:359] T f2bc9cd03afe4441a099e2054894818a P f17cfa3c3e5041c092728f920f98dec6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a2363aab21d43f68f26fb68238a29f3" member_type: VOTER last_known_addr { host: "127.28.113.3" port: 35071 } } peers { permanent_uuid: "6956a9660a3b4d42867878501794f2d5" member_type: VOTER last_known_addr { host: "127.28.113.1" port: 35255 } } peers { permanent_uuid: "f17cfa3c3e5041c092728f920f98dec6" member_type: VOTER last_known_addr { host: "127.28.113.2" port: 32787 } }
I20251215 21:02:49.140337 4719 raft_consensus.cc:385] T f2bc9cd03afe4441a099e2054894818a P f17cfa3c3e5041c092728f920f98dec6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251215 21:02:49.140388 4719 raft_consensus.cc:740] T f2bc9cd03afe4441a099e2054894818a P f17cfa3c3e5041c092728f920f98dec6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f17cfa3c3e5041c092728f920f98dec6, State: Initialized, Role: FOLLOWER
I20251215 21:02:49.140494 4718 consensus_queue.cc:260] T f2bc9cd03afe4441a099e2054894818a P 0a2363aab21d43f68f26fb68238a29f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a2363aab21d43f68f26fb68238a29f3" member_type: VOTER last_known_addr { host: "127.28.113.3" port: 35071 } } peers { permanent_uuid: "6956a9660a3b4d42867878501794f2d5" member_type: VOTER last_known_addr { host: "127.28.113.1" port: 35255 } } peers { permanent_uuid: "f17cfa3c3e5041c092728f920f98dec6" member_type: VOTER last_known_addr { host: "127.28.113.2" port: 32787 } }
I20251215 21:02:49.140771 4719 consensus_queue.cc:260] T f2bc9cd03afe4441a099e2054894818a P f17cfa3c3e5041c092728f920f98dec6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a2363aab21d43f68f26fb68238a29f3" member_type: VOTER last_known_addr { host: "127.28.113.3" port: 35071 } } peers { permanent_uuid: "6956a9660a3b4d42867878501794f2d5" member_type: VOTER last_known_addr { host: "127.28.113.1" port: 35255 } } peers { permanent_uuid: "f17cfa3c3e5041c092728f920f98dec6" member_type: VOTER last_known_addr { host: "127.28.113.2" port: 32787 } }
I20251215 21:02:49.141363 4718 ts_tablet_manager.cc:1434] T f2bc9cd03afe4441a099e2054894818a P 0a2363aab21d43f68f26fb68238a29f3: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20251215 21:02:49.142524 4685 heartbeater.cc:499] Master 127.28.113.62:41229 was elected leader, sending a full tablet report...
I20251215 21:02:49.143143 4719 ts_tablet_manager.cc:1434] T f2bc9cd03afe4441a099e2054894818a P f17cfa3c3e5041c092728f920f98dec6: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
W20251215 21:02:49.143553 4551 tablet.cc:2378] T f2bc9cd03afe4441a099e2054894818a P f17cfa3c3e5041c092728f920f98dec6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20251215 21:02:49.144531 4549 heartbeater.cc:499] Master 127.28.113.62:41229 was elected leader, sending a full tablet report...
W20251215 21:02:49.204665 4415 tablet.cc:2378] T f2bc9cd03afe4441a099e2054894818a P 6956a9660a3b4d42867878501794f2d5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20251215 21:02:49.327198 4725 raft_consensus.cc:493] T f2bc9cd03afe4441a099e2054894818a P f17cfa3c3e5041c092728f920f98dec6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 21:02:49.327407 4725 raft_consensus.cc:515] T f2bc9cd03afe4441a099e2054894818a P f17cfa3c3e5041c092728f920f98dec6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a2363aab21d43f68f26fb68238a29f3" member_type: VOTER last_known_addr { host: "127.28.113.3" port: 35071 } } peers { permanent_uuid: "6956a9660a3b4d42867878501794f2d5" member_type: VOTER last_known_addr { host: "127.28.113.1" port: 35255 } } peers { permanent_uuid: "f17cfa3c3e5041c092728f920f98dec6" member_type: VOTER last_known_addr { host: "127.28.113.2" port: 32787 } }
W20251215 21:02:49.328323 4687 tablet.cc:2378] T f2bc9cd03afe4441a099e2054894818a P 0a2363aab21d43f68f26fb68238a29f3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20251215 21:02:49.328729 4725 leader_election.cc:290] T f2bc9cd03afe4441a099e2054894818a P f17cfa3c3e5041c092728f920f98dec6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0a2363aab21d43f68f26fb68238a29f3 (127.28.113.3:35071), 6956a9660a3b4d42867878501794f2d5 (127.28.113.1:35255)
I20251215 21:02:49.332178 4554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:02:49.328942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.113.3:35071 (local address 127.28.113.2:59725)
1215 21:02:49.329087 (+ 145us) negotiation.cc:107] Waiting for socket to connect
1215 21:02:49.329103 (+ 16us) client_negotiation.cc:174] Beginning negotiation
1215 21:02:49.329194 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1215 21:02:49.329505 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1215 21:02:49.329508 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1215 21:02:49.329525 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1215 21:02:49.329771 (+ 246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:02:49.329778 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:02:49.330786 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1215 21:02:49.330790 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1215 21:02:49.331758 (+ 968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:02:49.331764 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:02:49.331876 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:02:49.331899 (+ 23us) client_negotiation.cc:769] Sending connection context
1215 21:02:49.331948 (+ 49us) client_negotiation.cc:240] Negotiation successful
1215 21:02:49.332032 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49}
I20251215 21:02:49.332178 4552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:02:49.329203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.113.1:35255 (local address 127.28.113.2:56963)
1215 21:02:49.329307 (+ 104us) negotiation.cc:107] Waiting for socket to connect
1215 21:02:49.329320 (+ 13us) client_negotiation.cc:174] Beginning negotiation
1215 21:02:49.329414 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1215 21:02:49.329688 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1215 21:02:49.329691 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1215 21:02:49.329705 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1215 21:02:49.329905 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:02:49.329911 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:02:49.330812 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1215 21:02:49.330815 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1215 21:02:49.331745 (+ 930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:02:49.331757 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:02:49.331876 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:02:49.331899 (+ 23us) client_negotiation.cc:769] Sending connection context
1215 21:02:49.331948 (+ 49us) client_negotiation.cc:240] Negotiation successful
1215 21:02:49.332032 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20251215 21:02:49.332654 4716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:02:49.329324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.113.2:56963 (local address 127.28.113.1:35255)
1215 21:02:49.329484 (+ 160us) server_negotiation.cc:207] Beginning negotiation
1215 21:02:49.329489 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1215 21:02:49.329504 (+ 15us) server_negotiation.cc:408] Connection header received
1215 21:02:49.329552 (+ 48us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1215 21:02:49.329555 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1215 21:02:49.329600 (+ 45us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1215 21:02:49.329656 (+ 56us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1215 21:02:49.330020 (+ 364us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:02:49.330711 (+ 691us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1215 21:02:49.331890 (+ 1179us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:02:49.332403 (+ 513us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:02:49.332443 (+ 40us) server_negotiation.cc:1036] Waiting for connection context
1215 21:02:49.332492 (+ 49us) server_negotiation.cc:300] Negotiation successful
1215 21:02:49.332543 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20251215 21:02:49.332654 4712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:02:49.329045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.113.2:59725 (local address 127.28.113.3:35071)
1215 21:02:49.329170 (+ 125us) server_negotiation.cc:207] Beginning negotiation
1215 21:02:49.329176 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1215 21:02:49.329193 (+ 17us) server_negotiation.cc:408] Connection header received
1215 21:02:49.329338 (+ 145us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1215 21:02:49.329341 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1215 21:02:49.329396 (+ 55us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1215 21:02:49.329473 (+ 77us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1215 21:02:49.329898 (+ 425us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:02:49.330664 (+ 766us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1215 21:02:49.331890 (+ 1226us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:02:49.332403 (+ 513us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:02:49.332443 (+ 40us) server_negotiation.cc:1036] Waiting for connection context
1215 21:02:49.332492 (+ 49us) server_negotiation.cc:300] Negotiation successful
1215 21:02:49.332543 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20251215 21:02:49.333245 4366 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f2bc9cd03afe4441a099e2054894818a" candidate_uuid: "f17cfa3c3e5041c092728f920f98dec6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6956a9660a3b4d42867878501794f2d5" is_pre_election: true
I20251215 21:02:49.333247 4638 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f2bc9cd03afe4441a099e2054894818a" candidate_uuid: "f17cfa3c3e5041c092728f920f98dec6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a2363aab21d43f68f26fb68238a29f3" is_pre_election: true
I20251215 21:02:49.333491 4366 raft_consensus.cc:2468] T f2bc9cd03afe4441a099e2054894818a P 6956a9660a3b4d42867878501794f2d5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f17cfa3c3e5041c092728f920f98dec6 in term 0.
I20251215 21:02:49.333490 4638 raft_consensus.cc:2468] T f2bc9cd03afe4441a099e2054894818a P 0a2363aab21d43f68f26fb68238a29f3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f17cfa3c3e5041c092728f920f98dec6 in term 0.
I20251215 21:02:49.333930 4436 leader_election.cc:304] T f2bc9cd03afe4441a099e2054894818a P f17cfa3c3e5041c092728f920f98dec6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a2363aab21d43f68f26fb68238a29f3, f17cfa3c3e5041c092728f920f98dec6; no voters:
I20251215 21:02:49.334175 4725 raft_consensus.cc:2804] T f2bc9cd03afe4441a099e2054894818a P f17cfa3c3e5041c092728f920f98dec6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251215 21:02:49.334249 4725 raft_consensus.cc:493] T f2bc9cd03afe4441a099e2054894818a P f17cfa3c3e5041c092728f920f98dec6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 21:02:49.334295 4725 raft_consensus.cc:3060] T f2bc9cd03afe4441a099e2054894818a P f17cfa3c3e5041c092728f920f98dec6 [term 0 FOLLOWER]: Advancing to term 1
I20251215 21:02:49.335302 4725 raft_consensus.cc:515] T f2bc9cd03afe4441a099e2054894818a P f17cfa3c3e5041c092728f920f98dec6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a2363aab21d43f68f26fb68238a29f3" member_type: VOTER last_known_addr { host: "127.28.113.3" port: 35071 } } peers { permanent_uuid: "6956a9660a3b4d42867878501794f2d5" member_type: VOTER last_known_addr { host: "127.28.113.1" port: 35255 } } peers { permanent_uuid: "f17cfa3c3e5041c092728f920f98dec6" member_type: VOTER last_known_addr { host: "127.28.113.2" port: 32787 } }
I20251215 21:02:49.335636 4725 leader_election.cc:290] T f2bc9cd03afe4441a099e2054894818a P f17cfa3c3e5041c092728f920f98dec6 [CANDIDATE]: Term 1 election: Requested vote from peers 0a2363aab21d43f68f26fb68238a29f3 (127.28.113.3:35071), 6956a9660a3b4d42867878501794f2d5 (127.28.113.1:35255)
I20251215 21:02:49.336025 4638 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f2bc9cd03afe4441a099e2054894818a" candidate_uuid: "f17cfa3c3e5041c092728f920f98dec6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a2363aab21d43f68f26fb68238a29f3"
I20251215 21:02:49.336093 4366 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f2bc9cd03afe4441a099e2054894818a" candidate_uuid: "f17cfa3c3e5041c092728f920f98dec6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6956a9660a3b4d42867878501794f2d5"
I20251215 21:02:49.336139 4638 raft_consensus.cc:3060] T f2bc9cd03afe4441a099e2054894818a P 0a2363aab21d43f68f26fb68238a29f3 [term 0 FOLLOWER]: Advancing to term 1
I20251215 21:02:49.336221 4366 raft_consensus.cc:3060] T f2bc9cd03afe4441a099e2054894818a P 6956a9660a3b4d42867878501794f2d5 [term 0 FOLLOWER]: Advancing to term 1
I20251215 21:02:49.337066 4638 raft_consensus.cc:2468] T f2bc9cd03afe4441a099e2054894818a P 0a2363aab21d43f68f26fb68238a29f3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f17cfa3c3e5041c092728f920f98dec6 in term 1.
I20251215 21:02:49.337122 4366 raft_consensus.cc:2468] T f2bc9cd03afe4441a099e2054894818a P 6956a9660a3b4d42867878501794f2d5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f17cfa3c3e5041c092728f920f98dec6 in term 1.
I20251215 21:02:49.337371 4436 leader_election.cc:304] T f2bc9cd03afe4441a099e2054894818a P f17cfa3c3e5041c092728f920f98dec6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a2363aab21d43f68f26fb68238a29f3, f17cfa3c3e5041c092728f920f98dec6; no voters:
I20251215 21:02:49.337528 4725 raft_consensus.cc:2804] T f2bc9cd03afe4441a099e2054894818a P f17cfa3c3e5041c092728f920f98dec6 [term 1 FOLLOWER]: Leader election won for term 1
I20251215 21:02:49.337742 4725 raft_consensus.cc:697] T f2bc9cd03afe4441a099e2054894818a P f17cfa3c3e5041c092728f920f98dec6 [term 1 LEADER]: Becoming Leader. State: Replica: f17cfa3c3e5041c092728f920f98dec6, State: Running, Role: LEADER
I20251215 21:02:49.338081 4725 consensus_queue.cc:237] T f2bc9cd03afe4441a099e2054894818a P f17cfa3c3e5041c092728f920f98dec6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a2363aab21d43f68f26fb68238a29f3" member_type: VOTER last_known_addr { host: "127.28.113.3" port: 35071 } } peers { permanent_uuid: "6956a9660a3b4d42867878501794f2d5" member_type: VOTER last_known_addr { host: "127.28.113.1" port: 35255 } } peers { permanent_uuid: "f17cfa3c3e5041c092728f920f98dec6" member_type: VOTER last_known_addr { host: "127.28.113.2" port: 32787 } }
I20251215 21:02:49.342022 3414 catalog_manager.cc:5654] T f2bc9cd03afe4441a099e2054894818a P f17cfa3c3e5041c092728f920f98dec6 reported cstate change: term changed from 0 to 1, leader changed from <none> to f17cfa3c3e5041c092728f920f98dec6 (127.28.113.2). New cstate: current_term: 1 leader_uuid: "f17cfa3c3e5041c092728f920f98dec6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a2363aab21d43f68f26fb68238a29f3" member_type: VOTER last_known_addr { host: "127.28.113.3" port: 35071 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6956a9660a3b4d42867878501794f2d5" member_type: VOTER last_known_addr { host: "127.28.113.1" port: 35255 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f17cfa3c3e5041c092728f920f98dec6" member_type: VOTER last_known_addr { host: "127.28.113.2" port: 32787 } health_report { overall_health: HEALTHY } } }
I20251215 21:02:49.358745 4715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:02:49.355910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34300 (local address 127.28.113.2:32787)
1215 21:02:49.356066 (+ 156us) server_negotiation.cc:207] Beginning negotiation
1215 21:02:49.356070 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1215 21:02:49.356085 (+ 15us) server_negotiation.cc:408] Connection header received
1215 21:02:49.356167 (+ 82us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1215 21:02:49.356169 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1215 21:02:49.356221 (+ 52us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1215 21:02:49.356302 (+ 81us) server_negotiation.cc:227] Negotiated authn=TOKEN
1215 21:02:49.356629 (+ 327us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:02:49.357108 (+ 479us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1215 21:02:49.357707 (+ 599us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:02:49.357883 (+ 176us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:02:49.357971 (+ 88us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1215 21:02:49.358379 (+ 408us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1215 21:02:49.358483 (+ 104us) server_negotiation.cc:1036] Waiting for connection context
1215 21:02:49.358570 (+ 87us) server_negotiation.cc:300] Negotiation successful
1215 21:02:49.358618 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20251215 21:02:49.363915 4366 raft_consensus.cc:1275] T f2bc9cd03afe4441a099e2054894818a P 6956a9660a3b4d42867878501794f2d5 [term 1 FOLLOWER]: Refusing update from remote peer f17cfa3c3e5041c092728f920f98dec6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251215 21:02:49.363915 4638 raft_consensus.cc:1275] T f2bc9cd03afe4441a099e2054894818a P 0a2363aab21d43f68f26fb68238a29f3 [term 1 FOLLOWER]: Refusing update from remote peer f17cfa3c3e5041c092728f920f98dec6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251215 21:02:49.364570 4725 consensus_queue.cc:1048] T f2bc9cd03afe4441a099e2054894818a P f17cfa3c3e5041c092728f920f98dec6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a2363aab21d43f68f26fb68238a29f3" member_type: VOTER last_known_addr { host: "127.28.113.3" port: 35071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251215 21:02:49.364748 4726 consensus_queue.cc:1048] T f2bc9cd03afe4441a099e2054894818a P f17cfa3c3e5041c092728f920f98dec6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6956a9660a3b4d42867878501794f2d5" member_type: VOTER last_known_addr { host: "127.28.113.1" port: 35255 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251215 21:02:49.373647 4732 mvcc.cc:204] Tried to move back new op lower bound from 7232850204107993088 to 7232850204011266048. Current Snapshot: MvccSnapshot[applied={T|T < 7232850204107993088}]
I20251215 21:02:49.376200 4731 mvcc.cc:204] Tried to move back new op lower bound from 7232850204107993088 to 7232850204011266048. Current Snapshot: MvccSnapshot[applied={T|T < 7232850204107993088 or (T in {7232850204107993088})}]
I20251215 21:02:49.382141 3414 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36730:
table { table_name: "test-table" } modify_external_catalogs: true
I20251215 21:02:49.382272 3414 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36730:
table { table_name: "test-table" } modify_external_catalogs: true
I20251215 21:02:49.386300 3414 catalog_manager.cc:5941] T 00000000000000000000000000000000 P 8c9267b46ebc4bae9a13928cf23fa8d3: Sending DeleteTablet for 3 replicas of tablet f2bc9cd03afe4441a099e2054894818a
I20251215 21:02:49.387039 4618 tablet_service.cc:1552] Processing DeleteTablet for tablet f2bc9cd03afe4441a099e2054894818a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-15 21:02:49 UTC) from {username='oryx', principal='oryx/127.28.113.62@KRBTEST.COM'} at 127.0.0.1:51486
I20251215 21:02:49.387109 4346 tablet_service.cc:1552] Processing DeleteTablet for tablet f2bc9cd03afe4441a099e2054894818a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-15 21:02:49 UTC) from {username='oryx', principal='oryx/127.28.113.62@KRBTEST.COM'} at 127.0.0.1:45060
I20251215 21:02:49.387529 4482 tablet_service.cc:1552] Processing DeleteTablet for tablet f2bc9cd03afe4441a099e2054894818a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-15 21:02:49 UTC) from {username='oryx', principal='oryx/127.28.113.62@KRBTEST.COM'} at 127.0.0.1:34290
I20251215 21:02:49.387879 4739 tablet_replica.cc:333] T f2bc9cd03afe4441a099e2054894818a P 6956a9660a3b4d42867878501794f2d5: stopping tablet replica
I20251215 21:02:49.388231 4739 raft_consensus.cc:2243] T f2bc9cd03afe4441a099e2054894818a P 6956a9660a3b4d42867878501794f2d5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251215 21:02:49.388638 4739 raft_consensus.cc:2272] T f2bc9cd03afe4441a099e2054894818a P 6956a9660a3b4d42867878501794f2d5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251215 21:02:49.388738 4740 tablet_replica.cc:333] T f2bc9cd03afe4441a099e2054894818a P 0a2363aab21d43f68f26fb68238a29f3: stopping tablet replica
I20251215 21:02:49.389014 4740 raft_consensus.cc:2243] T f2bc9cd03afe4441a099e2054894818a P 0a2363aab21d43f68f26fb68238a29f3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251215 21:02:49.389232 29124 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu with pid 3382
I20251215 21:02:49.390767 4739 ts_tablet_manager.cc:1916] T f2bc9cd03afe4441a099e2054894818a P 6956a9660a3b4d42867878501794f2d5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251215 21:02:49.389309 4740 raft_consensus.cc:2272] T f2bc9cd03afe4441a099e2054894818a P 0a2363aab21d43f68f26fb68238a29f3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251215 21:02:49.393196 4739 ts_tablet_manager.cc:1929] T f2bc9cd03afe4441a099e2054894818a P 6956a9660a3b4d42867878501794f2d5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251215 21:02:49.393296 4739 log.cc:1199] T f2bc9cd03afe4441a099e2054894818a P 6956a9660a3b4d42867878501794f2d5: Deleting WAL directory at /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765832506145157-29124-0/minicluster-data/ts-0/wal/wals/f2bc9cd03afe4441a099e2054894818a
I20251215 21:02:49.393599 4739 ts_tablet_manager.cc:1950] T f2bc9cd03afe4441a099e2054894818a P 6956a9660a3b4d42867878501794f2d5: Deleting consensus metadata
I20251215 21:02:49.394162 4740 ts_tablet_manager.cc:1916] T f2bc9cd03afe4441a099e2054894818a P 0a2363aab21d43f68f26fb68238a29f3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251215 21:02:49.395043 4741 tablet_replica.cc:333] T f2bc9cd03afe4441a099e2054894818a P f17cfa3c3e5041c092728f920f98dec6: stopping tablet replica
I20251215 21:02:49.395306 4741 raft_consensus.cc:2243] T f2bc9cd03afe4441a099e2054894818a P f17cfa3c3e5041c092728f920f98dec6 [term 1 LEADER]: Raft consensus shutting down.
I20251215 21:02:49.395741 4741 raft_consensus.cc:2272] T f2bc9cd03afe4441a099e2054894818a P f17cfa3c3e5041c092728f920f98dec6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251215 21:02:49.396770 4740 ts_tablet_manager.cc:1929] T f2bc9cd03afe4441a099e2054894818a P 0a2363aab21d43f68f26fb68238a29f3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251215 21:02:49.396907 4740 log.cc:1199] T f2bc9cd03afe4441a099e2054894818a P 0a2363aab21d43f68f26fb68238a29f3: Deleting WAL directory at /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765832506145157-29124-0/minicluster-data/ts-2/wal/wals/f2bc9cd03afe4441a099e2054894818a
I20251215 21:02:49.397044 4741 ts_tablet_manager.cc:1916] T f2bc9cd03afe4441a099e2054894818a P f17cfa3c3e5041c092728f920f98dec6: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251215 21:02:49.397261 4301 connection.cc:537] server connection from 127.0.0.1:45060 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20251215 21:02:49.397641 4740 ts_tablet_manager.cc:1950] T f2bc9cd03afe4441a099e2054894818a P 0a2363aab21d43f68f26fb68238a29f3: Deleting consensus metadata
W20251215 21:02:49.399065 4571 connection.cc:441] server connection from 127.0.0.1:51486 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51486 (request call id 1) could send its response
I20251215 21:02:49.399384 4741 ts_tablet_manager.cc:1929] T f2bc9cd03afe4441a099e2054894818a P f17cfa3c3e5041c092728f920f98dec6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251215 21:02:49.399461 4741 log.cc:1199] T f2bc9cd03afe4441a099e2054894818a P f17cfa3c3e5041c092728f920f98dec6: Deleting WAL directory at /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765832506145157-29124-0/minicluster-data/ts-1/wal/wals/f2bc9cd03afe4441a099e2054894818a
I20251215 21:02:49.399708 4741 ts_tablet_manager.cc:1950] T f2bc9cd03afe4441a099e2054894818a P f17cfa3c3e5041c092728f920f98dec6: Deleting consensus metadata
W20251215 21:02:49.400782 4437 connection.cc:441] server connection from 127.0.0.1:34290 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34290 (request call id 1) could send its response
I20251215 21:02:49.409660 4698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:02:49.402166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56104 (local address 127.28.113.61:39207)
1215 21:02:49.402302 (+ 136us) server_negotiation.cc:207] Beginning negotiation
1215 21:02:49.402307 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1215 21:02:49.402320 (+ 13us) server_negotiation.cc:408] Connection header received
1215 21:02:49.402360 (+ 40us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1215 21:02:49.402363 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1215 21:02:49.402406 (+ 43us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1215 21:02:49.402474 (+ 68us) server_negotiation.cc:227] Negotiated authn=SASL
1215 21:02:49.403434 (+ 960us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:02:49.403937 (+ 503us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1215 21:02:49.404757 (+ 820us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:02:49.404959 (+ 202us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:02:49.406066 (+ 1107us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1215 21:02:49.406086 (+ 20us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1215 21:02:49.406089 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1215 21:02:49.406116 (+ 27us) server_negotiation.cc:923] calling sasl_server_start()
1215 21:02:49.408137 (+ 2021us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1215 21:02:49.408537 (+ 400us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1215 21:02:49.408541 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1215 21:02:49.408544 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
1215 21:02:49.408596 (+ 52us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1215 21:02:49.408829 (+ 233us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1215 21:02:49.408832 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1215 21:02:49.408834 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1215 21:02:49.409015 (+ 181us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1215 21:02:49.409146 (+ 131us) server_negotiation.cc:1036] Waiting for connection context
1215 21:02:49.409372 (+ 226us) server_negotiation.cc:300] Negotiation successful
1215 21:02:49.409484 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20251215 21:02:49.410836 4700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:02:49.402121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46414 (local address 127.28.113.60:34031)
1215 21:02:49.402241 (+ 120us) server_negotiation.cc:207] Beginning negotiation
1215 21:02:49.402246 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1215 21:02:49.402464 (+ 218us) server_negotiation.cc:408] Connection header received
1215 21:02:49.402573 (+ 109us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1215 21:02:49.402576 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1215 21:02:49.402648 (+ 72us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1215 21:02:49.402731 (+ 83us) server_negotiation.cc:227] Negotiated authn=SASL
1215 21:02:49.404194 (+ 1463us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:02:49.404873 (+ 679us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1215 21:02:49.406022 (+ 1149us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:02:49.406857 (+ 835us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:02:49.407439 (+ 582us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1215 21:02:49.407460 (+ 21us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1215 21:02:49.407464 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1215 21:02:49.407490 (+ 26us) server_negotiation.cc:923] calling sasl_server_start()
1215 21:02:49.409242 (+ 1752us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1215 21:02:49.409702 (+ 460us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1215 21:02:49.409706 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1215 21:02:49.409708 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1215 21:02:49.409756 (+ 48us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1215 21:02:49.410012 (+ 256us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1215 21:02:49.410015 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1215 21:02:49.410017 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1215 21:02:49.410171 (+ 154us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1215 21:02:49.410261 (+ 90us) server_negotiation.cc:1036] Waiting for connection context
1215 21:02:49.410565 (+ 304us) server_negotiation.cc:300] Negotiation successful
1215 21:02:49.410676 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20251215 21:02:50.377743 4748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:02:50.377231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.113.62:41229 (local address 127.28.113.3:47781)
1215 21:02:50.377526 (+ 295us) negotiation.cc:107] Waiting for socket to connect
1215 21:02:50.377609 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.113.62:41229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":78,"threads_started":1}
I20251215 21:02:50.377861 4749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:02:50.377235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.113.62:41229 (local address 127.28.113.2:35613)
1215 21:02:50.377676 (+ 441us) negotiation.cc:107] Waiting for socket to connect
1215 21:02:50.377748 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.113.62:41229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":369,"thread_start_us":82,"threads_started":1}
W20251215 21:02:50.378024 4685 heartbeater.cc:646] Failed to heartbeat to 127.28.113.62:41229 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.28.113.62:41229: connect: Connection refused (error 111)
W20251215 21:02:50.378111 4549 heartbeater.cc:646] Failed to heartbeat to 127.28.113.62:41229 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.28.113.62:41229: connect: Connection refused (error 111)
I20251215 21:02:50.378643 4750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:02:50.378276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.113.62:41229 (local address 127.28.113.1:40239)
1215 21:02:50.378490 (+ 214us) negotiation.cc:107] Waiting for socket to connect
1215 21:02:50.378552 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.113.62:41229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":59,"threads_started":1}
W20251215 21:02:50.378849 4413 heartbeater.cc:646] Failed to heartbeat to 127.28.113.62:41229 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.28.113.62:41229: connect: Connection refused (error 111)
I20251215 21:02:50.886559 4751 raft_consensus.cc:493] T 00000000000000000000000000000000 P de9d22b45210451f997093f481e4dfc6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8c9267b46ebc4bae9a13928cf23fa8d3)
I20251215 21:02:50.886652 4752 raft_consensus.cc:493] T 00000000000000000000000000000000 P 297c0b4eec8b49459719bf6b7429a987 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8c9267b46ebc4bae9a13928cf23fa8d3)
I20251215 21:02:50.886754 4751 raft_consensus.cc:515] T 00000000000000000000000000000000 P de9d22b45210451f997093f481e4dfc6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "8c9267b46ebc4bae9a13928cf23fa8d3" member_type: VOTER last_known_addr { host: "127.28.113.62" port: 41229 } } peers { permanent_uuid: "de9d22b45210451f997093f481e4dfc6" member_type: VOTER last_known_addr { host: "127.28.113.61" port: 39207 } attrs { promote: false } } peers { permanent_uuid: "297c0b4eec8b49459719bf6b7429a987" member_type: VOTER last_known_addr { host: "127.28.113.60" port: 34031 } attrs { promote: false } }
I20251215 21:02:50.886813 4752 raft_consensus.cc:515] T 00000000000000000000000000000000 P 297c0b4eec8b49459719bf6b7429a987 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "8c9267b46ebc4bae9a13928cf23fa8d3" member_type: VOTER last_known_addr { host: "127.28.113.62" port: 41229 } } peers { permanent_uuid: "de9d22b45210451f997093f481e4dfc6" member_type: VOTER last_known_addr { host: "127.28.113.61" port: 39207 } attrs { promote: false } } peers { permanent_uuid: "297c0b4eec8b49459719bf6b7429a987" member_type: VOTER last_known_addr { host: "127.28.113.60" port: 34031 } attrs { promote: false } }
I20251215 21:02:50.887691 4751 leader_election.cc:290] T 00000000000000000000000000000000 P de9d22b45210451f997093f481e4dfc6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8c9267b46ebc4bae9a13928cf23fa8d3 (127.28.113.62:41229), 297c0b4eec8b49459719bf6b7429a987 (127.28.113.60:34031)
I20251215 21:02:50.887687 4752 leader_election.cc:290] T 00000000000000000000000000000000 P 297c0b4eec8b49459719bf6b7429a987 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8c9267b46ebc4bae9a13928cf23fa8d3 (127.28.113.62:41229), de9d22b45210451f997093f481e4dfc6 (127.28.113.61:39207)
I20251215 21:02:50.888353 4753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:02:50.887859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.113.62:41229 (local address 127.0.0.1:37222)
1215 21:02:50.888177 (+ 318us) negotiation.cc:107] Waiting for socket to connect
1215 21:02:50.888253 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.113.62:41229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":54,"threads_started":1}
I20251215 21:02:50.888409 4042 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "297c0b4eec8b49459719bf6b7429a987" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "de9d22b45210451f997093f481e4dfc6" is_pre_election: true
W20251215 21:02:50.888477 4016 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.28.113.62:41229: connect: Connection refused (error 111)
I20251215 21:02:50.888545 4754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:02:50.888025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.113.62:41229 (local address 127.0.0.1:37234)
1215 21:02:50.888400 (+ 375us) negotiation.cc:107] Waiting for socket to connect
1215 21:02:50.888459 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.113.62:41229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":300,"thread_start_us":53,"threads_started":1}
I20251215 21:02:50.888554 4042 raft_consensus.cc:2468] T 00000000000000000000000000000000 P de9d22b45210451f997093f481e4dfc6 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 297c0b4eec8b49459719bf6b7429a987 in term 1.
I20251215 21:02:50.888927 4201 leader_election.cc:304] T 00000000000000000000000000000000 P 297c0b4eec8b49459719bf6b7429a987 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 297c0b4eec8b49459719bf6b7429a987, de9d22b45210451f997093f481e4dfc6; no voters:
W20251215 21:02:50.889001 4200 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.28.113.62:41229: connect: Connection refused (error 111)
I20251215 21:02:50.889041 4226 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "de9d22b45210451f997093f481e4dfc6" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "297c0b4eec8b49459719bf6b7429a987" is_pre_election: true
I20251215 21:02:50.889137 4752 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 297c0b4eec8b49459719bf6b7429a987 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251215 21:02:50.889204 4226 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 297c0b4eec8b49459719bf6b7429a987 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate de9d22b45210451f997093f481e4dfc6 in term 1.
I20251215 21:02:50.889372 4752 raft_consensus.cc:493] T 00000000000000000000000000000000 P 297c0b4eec8b49459719bf6b7429a987 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 8c9267b46ebc4bae9a13928cf23fa8d3)
I20251215 21:02:50.889420 4752 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 297c0b4eec8b49459719bf6b7429a987 [term 1 FOLLOWER]: Advancing to term 2
I20251215 21:02:50.889703 4018 leader_election.cc:304] T 00000000000000000000000000000000 P de9d22b45210451f997093f481e4dfc6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 297c0b4eec8b49459719bf6b7429a987, de9d22b45210451f997093f481e4dfc6; no voters:
I20251215 21:02:50.889981 4754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:02:50.889731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.113.62:41229 (local address 127.0.0.1:37252)
1215 21:02:50.889875 (+ 144us) negotiation.cc:107] Waiting for socket to connect
1215 21:02:50.889919 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.113.62:41229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
W20251215 21:02:50.890102 4200 leader_election.cc:336] T 00000000000000000000000000000000 P 297c0b4eec8b49459719bf6b7429a987 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 8c9267b46ebc4bae9a13928cf23fa8d3 (127.28.113.62:41229): Network error: Client connection negotiation failed: client connection to 127.28.113.62:41229: connect: Connection refused (error 111)
I20251215 21:02:50.889994 4751 raft_consensus.cc:2804] T 00000000000000000000000000000000 P de9d22b45210451f997093f481e4dfc6 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251215 21:02:50.890280 4751 raft_consensus.cc:493] T 00000000000000000000000000000000 P de9d22b45210451f997093f481e4dfc6 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 8c9267b46ebc4bae9a13928cf23fa8d3)
I20251215 21:02:50.890347 4751 raft_consensus.cc:3060] T 00000000000000000000000000000000 P de9d22b45210451f997093f481e4dfc6 [term 1 FOLLOWER]: Advancing to term 2
I20251215 21:02:50.890691 4752 raft_consensus.cc:515] T 00000000000000000000000000000000 P 297c0b4eec8b49459719bf6b7429a987 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "8c9267b46ebc4bae9a13928cf23fa8d3" member_type: VOTER last_known_addr { host: "127.28.113.62" port: 41229 } } peers { permanent_uuid: "de9d22b45210451f997093f481e4dfc6" member_type: VOTER last_known_addr { host: "127.28.113.61" port: 39207 } attrs { promote: false } } peers { permanent_uuid: "297c0b4eec8b49459719bf6b7429a987" member_type: VOTER last_known_addr { host: "127.28.113.60" port: 34031 } attrs { promote: false } }
I20251215 21:02:50.890937 4753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:02:50.889390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.113.62:41229 (local address 127.0.0.1:37244)
1215 21:02:50.890140 (+ 750us) negotiation.cc:107] Waiting for socket to connect
1215 21:02:50.890881 (+ 741us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.113.62:41229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":681}
W20251215 21:02:50.891028 4016 leader_election.cc:336] T 00000000000000000000000000000000 P de9d22b45210451f997093f481e4dfc6 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 8c9267b46ebc4bae9a13928cf23fa8d3 (127.28.113.62:41229): Network error: Client connection negotiation failed: client connection to 127.28.113.62:41229: connect: Connection refused (error 111)
I20251215 21:02:50.891038 4752 leader_election.cc:290] T 00000000000000000000000000000000 P 297c0b4eec8b49459719bf6b7429a987 [CANDIDATE]: Term 2 election: Requested vote from peers 8c9267b46ebc4bae9a13928cf23fa8d3 (127.28.113.62:41229), de9d22b45210451f997093f481e4dfc6 (127.28.113.61:39207)
I20251215 21:02:50.891373 4754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:02:50.891129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.113.62:41229 (local address 127.0.0.1:37262)
1215 21:02:50.891270 (+ 141us) negotiation.cc:107] Waiting for socket to connect
1215 21:02:50.891313 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.113.62:41229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20251215 21:02:50.891435 4042 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "297c0b4eec8b49459719bf6b7429a987" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "de9d22b45210451f997093f481e4dfc6"
I20251215 21:02:50.891438 4751 raft_consensus.cc:515] T 00000000000000000000000000000000 P de9d22b45210451f997093f481e4dfc6 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "8c9267b46ebc4bae9a13928cf23fa8d3" member_type: VOTER last_known_addr { host: "127.28.113.62" port: 41229 } } peers { permanent_uuid: "de9d22b45210451f997093f481e4dfc6" member_type: VOTER last_known_addr { host: "127.28.113.61" port: 39207 } attrs { promote: false } } peers { permanent_uuid: "297c0b4eec8b49459719bf6b7429a987" member_type: VOTER last_known_addr { host: "127.28.113.60" port: 34031 } attrs { promote: false } }
I20251215 21:02:50.891860 4042 raft_consensus.cc:2393] T 00000000000000000000000000000000 P de9d22b45210451f997093f481e4dfc6 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 297c0b4eec8b49459719bf6b7429a987 in current term 2: Already voted for candidate de9d22b45210451f997093f481e4dfc6 in this term.
I20251215 21:02:50.892043 4754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:02:50.891802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.113.62:41229 (local address 127.0.0.1:37270)
1215 21:02:50.891912 (+ 110us) negotiation.cc:107] Waiting for socket to connect
1215 21:02:50.891985 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.113.62:41229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20251215 21:02:50.892201 4200 leader_election.cc:336] T 00000000000000000000000000000000 P 297c0b4eec8b49459719bf6b7429a987 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 8c9267b46ebc4bae9a13928cf23fa8d3 (127.28.113.62:41229): Network error: Client connection negotiation failed: client connection to 127.28.113.62:41229: connect: Connection refused (error 111)
I20251215 21:02:50.892251 4751 leader_election.cc:290] T 00000000000000000000000000000000 P de9d22b45210451f997093f481e4dfc6 [CANDIDATE]: Term 2 election: Requested vote from peers 8c9267b46ebc4bae9a13928cf23fa8d3 (127.28.113.62:41229), 297c0b4eec8b49459719bf6b7429a987 (127.28.113.60:34031)
I20251215 21:02:50.892279 4201 leader_election.cc:304] T 00000000000000000000000000000000 P 297c0b4eec8b49459719bf6b7429a987 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 297c0b4eec8b49459719bf6b7429a987; no voters: 8c9267b46ebc4bae9a13928cf23fa8d3, de9d22b45210451f997093f481e4dfc6
I20251215 21:02:50.892304 4226 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "de9d22b45210451f997093f481e4dfc6" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "297c0b4eec8b49459719bf6b7429a987"
I20251215 21:02:50.892453 4226 raft_consensus.cc:2393] T 00000000000000000000000000000000 P 297c0b4eec8b49459719bf6b7429a987 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate de9d22b45210451f997093f481e4dfc6 in current term 2: Already voted for candidate 297c0b4eec8b49459719bf6b7429a987 in this term.
I20251215 21:02:50.892511 4752 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 297c0b4eec8b49459719bf6b7429a987 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20251215 21:02:50.892561 4753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:02:50.892091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.113.62:41229 (local address 127.0.0.1:37274)
1215 21:02:50.892456 (+ 365us) negotiation.cc:107] Waiting for socket to connect
1215 21:02:50.892501 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.113.62:41229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":280}
I20251215 21:02:50.893071 4753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:02:50.892885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.113.62:41229 (local address 127.0.0.1:37288)
1215 21:02:50.892977 (+ 92us) negotiation.cc:107] Waiting for socket to connect
1215 21:02:50.893017 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.113.62:41229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20251215 21:02:50.893164 4016 leader_election.cc:336] T 00000000000000000000000000000000 P de9d22b45210451f997093f481e4dfc6 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 8c9267b46ebc4bae9a13928cf23fa8d3 (127.28.113.62:41229): Network error: Client connection negotiation failed: client connection to 127.28.113.62:41229: connect: Connection refused (error 111)
I20251215 21:02:50.893221 4016 leader_election.cc:304] T 00000000000000000000000000000000 P de9d22b45210451f997093f481e4dfc6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: de9d22b45210451f997093f481e4dfc6; no voters: 297c0b4eec8b49459719bf6b7429a987, 8c9267b46ebc4bae9a13928cf23fa8d3
I20251215 21:02:50.893353 4751 raft_consensus.cc:2749] T 00000000000000000000000000000000 P de9d22b45210451f997093f481e4dfc6 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20251215 21:02:51.379069 4758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:02:51.378624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.113.62:41229 (local address 127.28.113.2:34405)
1215 21:02:51.378909 (+ 285us) negotiation.cc:107] Waiting for socket to connect
1215 21:02:51.378967 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.113.62:41229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":77,"threads_started":1}
I20251215 21:02:51.379207 4757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:02:51.378511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.113.62:41229 (local address 127.28.113.3:41661)
1215 21:02:51.378810 (+ 299us) negotiation.cc:107] Waiting for socket to connect
1215 21:02:51.378871 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.113.62:41229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":76,"threads_started":1}
I20251215 21:02:51.379791 4759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:02:51.379276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.113.62:41229 (local address 127.28.113.1:49935)
1215 21:02:51.379652 (+ 376us) negotiation.cc:107] Waiting for socket to connect
1215 21:02:51.379704 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.113.62:41229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":307,"thread_start_us":65,"threads_started":1}
I20251215 21:02:52.380185 4761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:02:52.379735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.113.62:41229 (local address 127.28.113.3:37523)
1215 21:02:52.380007 (+ 272us) negotiation.cc:107] Waiting for socket to connect
1215 21:02:52.380075 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.113.62:41229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":67,"threads_started":1}
I20251215 21:02:52.380223 4760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:02:52.379673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.113.62:41229 (local address 127.28.113.2:43841)
1215 21:02:52.380054 (+ 381us) negotiation.cc:107] Waiting for socket to connect
1215 21:02:52.380122 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.113.62:41229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":77,"threads_started":1}
I20251215 21:02:52.380720 4762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:02:52.380384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.113.62:41229 (local address 127.28.113.1:39695)
1215 21:02:52.380583 (+ 199us) negotiation.cc:107] Waiting for socket to connect
1215 21:02:52.380635 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.113.62:41229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":65,"threads_started":1}
I20251215 21:02:52.655947 4763 raft_consensus.cc:493] T 00000000000000000000000000000000 P de9d22b45210451f997093f481e4dfc6 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 21:02:52.656108 4763 raft_consensus.cc:515] T 00000000000000000000000000000000 P de9d22b45210451f997093f481e4dfc6 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "8c9267b46ebc4bae9a13928cf23fa8d3" member_type: VOTER last_known_addr { host: "127.28.113.62" port: 41229 } } peers { permanent_uuid: "de9d22b45210451f997093f481e4dfc6" member_type: VOTER last_known_addr { host: "127.28.113.61" port: 39207 } attrs { promote: false } } peers { permanent_uuid: "297c0b4eec8b49459719bf6b7429a987" member_type: VOTER last_known_addr { host: "127.28.113.60" port: 34031 } attrs { promote: false } }
I20251215 21:02:52.656497 4763 leader_election.cc:290] T 00000000000000000000000000000000 P de9d22b45210451f997093f481e4dfc6 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8c9267b46ebc4bae9a13928cf23fa8d3 (127.28.113.62:41229), 297c0b4eec8b49459719bf6b7429a987 (127.28.113.60:34031)
I20251215 21:02:52.657025 4226 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "de9d22b45210451f997093f481e4dfc6" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "297c0b4eec8b49459719bf6b7429a987" is_pre_election: true
I20251215 21:02:52.657097 4764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:02:52.656681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.113.62:41229 (local address 127.0.0.1:37500)
1215 21:02:52.656964 (+ 283us) negotiation.cc:107] Waiting for socket to connect
1215 21:02:52.657013 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.113.62:41229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":68,"threads_started":1}
I20251215 21:02:52.657155 4226 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 297c0b4eec8b49459719bf6b7429a987 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate de9d22b45210451f997093f481e4dfc6 in term 2.
I20251215 21:02:52.657486 4018 leader_election.cc:304] T 00000000000000000000000000000000 P de9d22b45210451f997093f481e4dfc6 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 297c0b4eec8b49459719bf6b7429a987, de9d22b45210451f997093f481e4dfc6; no voters:
I20251215 21:02:52.657668 4763 raft_consensus.cc:2804] T 00000000000000000000000000000000 P de9d22b45210451f997093f481e4dfc6 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251215 21:02:52.657727 4763 raft_consensus.cc:493] T 00000000000000000000000000000000 P de9d22b45210451f997093f481e4dfc6 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 21:02:52.657763 4763 raft_consensus.cc:3060] T 00000000000000000000000000000000 P de9d22b45210451f997093f481e4dfc6 [term 2 FOLLOWER]: Advancing to term 3
I20251215 21:02:52.657876 4764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:02:52.657640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.113.62:41229 (local address 127.0.0.1:37514)
1215 21:02:52.657759 (+ 119us) negotiation.cc:107] Waiting for socket to connect
1215 21:02:52.657815 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.113.62:41229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20251215 21:02:52.657984 4016 leader_election.cc:336] T 00000000000000000000000000000000 P de9d22b45210451f997093f481e4dfc6 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 8c9267b46ebc4bae9a13928cf23fa8d3 (127.28.113.62:41229): Network error: Client connection negotiation failed: client connection to 127.28.113.62:41229: connect: Connection refused (error 111)
I20251215 21:02:52.658716 4763 raft_consensus.cc:515] T 00000000000000000000000000000000 P de9d22b45210451f997093f481e4dfc6 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "8c9267b46ebc4bae9a13928cf23fa8d3" member_type: VOTER last_known_addr { host: "127.28.113.62" port: 41229 } } peers { permanent_uuid: "de9d22b45210451f997093f481e4dfc6" member_type: VOTER last_known_addr { host: "127.28.113.61" port: 39207 } attrs { promote: false } } peers { permanent_uuid: "297c0b4eec8b49459719bf6b7429a987" member_type: VOTER last_known_addr { host: "127.28.113.60" port: 34031 } attrs { promote: false } }
I20251215 21:02:52.659044 4763 leader_election.cc:290] T 00000000000000000000000000000000 P de9d22b45210451f997093f481e4dfc6 [CANDIDATE]: Term 3 election: Requested vote from peers 8c9267b46ebc4bae9a13928cf23fa8d3 (127.28.113.62:41229), 297c0b4eec8b49459719bf6b7429a987 (127.28.113.60:34031)
I20251215 21:02:52.659349 4764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:02:52.659132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.113.62:41229 (local address 127.0.0.1:37520)
1215 21:02:52.659247 (+ 115us) negotiation.cc:107] Waiting for socket to connect
1215 21:02:52.659293 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.113.62:41229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20251215 21:02:52.659364 4226 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "de9d22b45210451f997093f481e4dfc6" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "297c0b4eec8b49459719bf6b7429a987"
I20251215 21:02:52.659476 4226 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 297c0b4eec8b49459719bf6b7429a987 [term 2 FOLLOWER]: Advancing to term 3
I20251215 21:02:52.659931 4764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:02:52.659744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.113.62:41229 (local address 127.0.0.1:37526)
1215 21:02:52.659841 (+ 97us) negotiation.cc:107] Waiting for socket to connect
1215 21:02:52.659885 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.113.62:41229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20251215 21:02:52.660029 4016 leader_election.cc:336] T 00000000000000000000000000000000 P de9d22b45210451f997093f481e4dfc6 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 8c9267b46ebc4bae9a13928cf23fa8d3 (127.28.113.62:41229): Network error: Client connection negotiation failed: client connection to 127.28.113.62:41229: connect: Connection refused (error 111)
I20251215 21:02:52.660446 4226 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 297c0b4eec8b49459719bf6b7429a987 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate de9d22b45210451f997093f481e4dfc6 in term 3.
I20251215 21:02:52.660715 4018 leader_election.cc:304] T 00000000000000000000000000000000 P de9d22b45210451f997093f481e4dfc6 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 297c0b4eec8b49459719bf6b7429a987, de9d22b45210451f997093f481e4dfc6; no voters: 8c9267b46ebc4bae9a13928cf23fa8d3
I20251215 21:02:52.660876 4763 raft_consensus.cc:2804] T 00000000000000000000000000000000 P de9d22b45210451f997093f481e4dfc6 [term 3 FOLLOWER]: Leader election won for term 3
I20251215 21:02:52.661028 4763 raft_consensus.cc:697] T 00000000000000000000000000000000 P de9d22b45210451f997093f481e4dfc6 [term 3 LEADER]: Becoming Leader. State: Replica: de9d22b45210451f997093f481e4dfc6, State: Running, Role: LEADER
I20251215 21:02:52.661262 4763 consensus_queue.cc:237] T 00000000000000000000000000000000 P de9d22b45210451f997093f481e4dfc6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "8c9267b46ebc4bae9a13928cf23fa8d3" member_type: VOTER last_known_addr { host: "127.28.113.62" port: 41229 } } peers { permanent_uuid: "de9d22b45210451f997093f481e4dfc6" member_type: VOTER last_known_addr { host: "127.28.113.61" port: 39207 } attrs { promote: false } } peers { permanent_uuid: "297c0b4eec8b49459719bf6b7429a987" member_type: VOTER last_known_addr { host: "127.28.113.60" port: 34031 } attrs { promote: false } }
I20251215 21:02:52.662268 4766 sys_catalog.cc:455] T 00000000000000000000000000000000 P de9d22b45210451f997093f481e4dfc6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader de9d22b45210451f997093f481e4dfc6. Latest consensus state: current_term: 3 leader_uuid: "de9d22b45210451f997093f481e4dfc6" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "8c9267b46ebc4bae9a13928cf23fa8d3" member_type: VOTER last_known_addr { host: "127.28.113.62" port: 41229 } } peers { permanent_uuid: "de9d22b45210451f997093f481e4dfc6" member_type: VOTER last_known_addr { host: "127.28.113.61" port: 39207 } attrs { promote: false } } peers { permanent_uuid: "297c0b4eec8b49459719bf6b7429a987" member_type: VOTER last_known_addr { host: "127.28.113.60" port: 34031 } attrs { promote: false } } }
I20251215 21:02:52.662375 4766 sys_catalog.cc:458] T 00000000000000000000000000000000 P de9d22b45210451f997093f481e4dfc6 [sys.catalog]: This master's current role is: LEADER
I20251215 21:02:52.662657 4768 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251215 21:02:52.664672 4768 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251215 21:02:52.665042 4768 catalog_manager.cc:1269] Loaded cluster ID: 587ee523bb434fd8a7a37f28e6cc4cb1
I20251215 21:02:52.665086 4768 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251215 21:02:52.665592 4768 catalog_manager.cc:1514] Loading token signing keys...
I20251215 21:02:52.665962 4768 catalog_manager.cc:6038] T 00000000000000000000000000000000 P de9d22b45210451f997093f481e4dfc6: Loaded TSK: 0
I20251215 21:02:52.666603 4768 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251215 21:02:52.680001 4032 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56104:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251215 21:02:52.682287 4032 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251215 21:02:52.684909 4764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:02:52.684681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.113.62:41229 (local address 127.0.0.1:37544)
1215 21:02:52.684797 (+ 116us) negotiation.cc:107] Waiting for socket to connect
1215 21:02:52.684843 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.113.62:41229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20251215 21:02:52.684994 4226 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 297c0b4eec8b49459719bf6b7429a987 [term 3 FOLLOWER]: Refusing update from remote peer de9d22b45210451f997093f481e4dfc6: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20251215 21:02:52.685427 4766 consensus_queue.cc:1048] T 00000000000000000000000000000000 P de9d22b45210451f997093f481e4dfc6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "297c0b4eec8b49459719bf6b7429a987" member_type: VOTER last_known_addr { host: "127.28.113.60" port: 34031 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20251215 21:02:52.685585 4764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:02:52.685376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.113.62:41229 (local address 127.0.0.1:37556)
1215 21:02:52.685479 (+ 103us) negotiation.cc:107] Waiting for socket to connect
1215 21:02:52.685518 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.113.62:41229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20251215 21:02:52.685695 4016 consensus_peers.cc:597] T 00000000000000000000000000000000 P de9d22b45210451f997093f481e4dfc6 -> Peer 8c9267b46ebc4bae9a13928cf23fa8d3 (127.28.113.62:41229): Couldn't send request to peer 8c9267b46ebc4bae9a13928cf23fa8d3. Status: Network error: Client connection negotiation failed: client connection to 127.28.113.62:41229: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20251215 21:02:52.686681 4770 sys_catalog.cc:455] T 00000000000000000000000000000000 P 297c0b4eec8b49459719bf6b7429a987 [sys.catalog]: SysCatalogTable state changed. Reason: New leader de9d22b45210451f997093f481e4dfc6. Latest consensus state: current_term: 3 leader_uuid: "de9d22b45210451f997093f481e4dfc6" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "8c9267b46ebc4bae9a13928cf23fa8d3" member_type: VOTER last_known_addr { host: "127.28.113.62" port: 41229 } } peers { permanent_uuid: "de9d22b45210451f997093f481e4dfc6" member_type: VOTER last_known_addr { host: "127.28.113.61" port: 39207 } attrs { promote: false } } peers { permanent_uuid: "297c0b4eec8b49459719bf6b7429a987" member_type: VOTER last_known_addr { host: "127.28.113.60" port: 34031 } attrs { promote: false } } }
I20251215 21:02:52.686776 4770 sys_catalog.cc:458] T 00000000000000000000000000000000 P 297c0b4eec8b49459719bf6b7429a987 [sys.catalog]: This master's current role is: FOLLOWER
I20251215 21:02:52.687776 4763 sys_catalog.cc:455] T 00000000000000000000000000000000 P de9d22b45210451f997093f481e4dfc6 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "de9d22b45210451f997093f481e4dfc6" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "8c9267b46ebc4bae9a13928cf23fa8d3" member_type: VOTER last_known_addr { host: "127.28.113.62" port: 41229 } } peers { permanent_uuid: "de9d22b45210451f997093f481e4dfc6" member_type: VOTER last_known_addr { host: "127.28.113.61" port: 39207 } attrs { promote: false } } peers { permanent_uuid: "297c0b4eec8b49459719bf6b7429a987" member_type: VOTER last_known_addr { host: "127.28.113.60" port: 34031 } attrs { promote: false } } }
I20251215 21:02:52.687870 4763 sys_catalog.cc:458] T 00000000000000000000000000000000 P de9d22b45210451f997093f481e4dfc6 [sys.catalog]: This master's current role is: LEADER
I20251215 21:02:52.689965 4770 sys_catalog.cc:455] T 00000000000000000000000000000000 P 297c0b4eec8b49459719bf6b7429a987 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "de9d22b45210451f997093f481e4dfc6" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "8c9267b46ebc4bae9a13928cf23fa8d3" member_type: VOTER last_known_addr { host: "127.28.113.62" port: 41229 } } peers { permanent_uuid: "de9d22b45210451f997093f481e4dfc6" member_type: VOTER last_known_addr { host: "127.28.113.61" port: 39207 } attrs { promote: false } } peers { permanent_uuid: "297c0b4eec8b49459719bf6b7429a987" member_type: VOTER last_known_addr { host: "127.28.113.60" port: 34031 } attrs { promote: false } } }
I20251215 21:02:52.690058 4770 sys_catalog.cc:458] T 00000000000000000000000000000000 P 297c0b4eec8b49459719bf6b7429a987 [sys.catalog]: This master's current role is: FOLLOWER
I20251215 21:02:52.690138 4771 mvcc.cc:204] Tried to move back new op lower bound from 7232850217713684480 to 7232850217622593536. Current Snapshot: MvccSnapshot[applied={T|T < 7232850217713684480 or (T in {7232850217713684480})}]
I20251215 21:02:52.696566 4777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:02:52.693570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.113.1:35255 (local address 127.0.0.1:45076)
1215 21:02:52.693789 (+ 219us) negotiation.cc:107] Waiting for socket to connect
1215 21:02:52.693802 (+ 13us) client_negotiation.cc:174] Beginning negotiation
1215 21:02:52.693887 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1215 21:02:52.694270 (+ 383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1215 21:02:52.694273 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1215 21:02:52.694288 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1215 21:02:52.694536 (+ 248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:02:52.694544 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:02:52.695456 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1215 21:02:52.695460 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1215 21:02:52.696203 (+ 743us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:02:52.696211 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:02:52.696317 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:02:52.696338 (+ 21us) client_negotiation.cc:769] Sending connection context
1215 21:02:52.696386 (+ 48us) client_negotiation.cc:240] Negotiation successful
1215 21:02:52.696436 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":84,"threads_started":1}
I20251215 21:02:52.697160 4778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:02:52.693678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45076 (local address 127.28.113.1:35255)
1215 21:02:52.693931 (+ 253us) server_negotiation.cc:207] Beginning negotiation
1215 21:02:52.693935 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1215 21:02:52.693951 (+ 16us) server_negotiation.cc:408] Connection header received
1215 21:02:52.694009 (+ 58us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1215 21:02:52.694013 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1215 21:02:52.694075 (+ 62us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1215 21:02:52.694152 (+ 77us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1215 21:02:52.694661 (+ 509us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:02:52.695345 (+ 684us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1215 21:02:52.696306 (+ 961us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:02:52.696898 (+ 592us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:02:52.696934 (+ 36us) server_negotiation.cc:1036] Waiting for connection context
1215 21:02:52.696981 (+ 47us) server_negotiation.cc:300] Negotiation successful
1215 21:02:52.697033 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":171,"thread_start_us":75,"threads_started":1}
I20251215 21:02:52.697443 4764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:02:52.692959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.113.3:35071 (local address 127.0.0.1:51494)
1215 21:02:52.693118 (+ 159us) negotiation.cc:107] Waiting for socket to connect
1215 21:02:52.693132 (+ 14us) client_negotiation.cc:174] Beginning negotiation
1215 21:02:52.693243 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1215 21:02:52.695123 (+ 1880us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1215 21:02:52.695126 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1215 21:02:52.695144 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1215 21:02:52.695344 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:02:52.695352 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:02:52.696271 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1215 21:02:52.696274 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1215 21:02:52.697078 (+ 804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:02:52.697087 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:02:52.697188 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:02:52.697204 (+ 16us) client_negotiation.cc:769] Sending connection context
1215 21:02:52.697254 (+ 50us) client_negotiation.cc:240] Negotiation successful
1215 21:02:52.697317 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":95}
I20251215 21:02:52.697691 4346 tablet_service.cc:1505] Processing CreateTablet for tablet c35fb133c71e4054a03b62f133fcaeb2 (DEFAULT_TABLE table=test-table [id=c830df2bceaa436e99a7ed58a423e1e5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251215 21:02:52.697751 4780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:02:52.693974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34304 (local address 127.28.113.2:32787)
1215 21:02:52.694487 (+ 513us) server_negotiation.cc:207] Beginning negotiation
1215 21:02:52.694492 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1215 21:02:52.694503 (+ 11us) server_negotiation.cc:408] Connection header received
1215 21:02:52.694549 (+ 46us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1215 21:02:52.694552 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1215 21:02:52.694600 (+ 48us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1215 21:02:52.694682 (+ 82us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1215 21:02:52.695012 (+ 330us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:02:52.695790 (+ 778us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1215 21:02:52.697023 (+ 1233us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:02:52.697481 (+ 458us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:02:52.697517 (+ 36us) server_negotiation.cc:1036] Waiting for connection context
1215 21:02:52.697573 (+ 56us) server_negotiation.cc:300] Negotiation successful
1215 21:02:52.697633 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":434,"thread_start_us":69,"threads_started":1}
I20251215 21:02:52.697346 4779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:02:52.693849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.113.2:32787 (local address 127.0.0.1:34304)
1215 21:02:52.694223 (+ 374us) negotiation.cc:107] Waiting for socket to connect
1215 21:02:52.694236 (+ 13us) client_negotiation.cc:174] Beginning negotiation
1215 21:02:52.694298 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1215 21:02:52.694700 (+ 402us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1215 21:02:52.694703 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1215 21:02:52.694719 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1215 21:02:52.694896 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:02:52.694900 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:02:52.695904 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1215 21:02:52.695908 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1215 21:02:52.696904 (+ 996us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:02:52.696913 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:02:52.697008 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:02:52.697033 (+ 25us) client_negotiation.cc:769] Sending connection context
1215 21:02:52.697080 (+ 47us) client_negotiation.cc:240] Negotiation successful
1215 21:02:52.697132 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":314,"thread_start_us":76,"threads_started":1}
I20251215 21:02:52.697979 4346 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c35fb133c71e4054a03b62f133fcaeb2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 21:02:52.698406 4482 tablet_service.cc:1505] Processing CreateTablet for tablet c35fb133c71e4054a03b62f133fcaeb2 (DEFAULT_TABLE table=test-table [id=c830df2bceaa436e99a7ed58a423e1e5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251215 21:02:52.698505 4776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:02:52.693057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51494 (local address 127.28.113.3:35071)
1215 21:02:52.694899 (+ 1842us) server_negotiation.cc:207] Beginning negotiation
1215 21:02:52.694903 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1215 21:02:52.694917 (+ 14us) server_negotiation.cc:408] Connection header received
1215 21:02:52.694962 (+ 45us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1215 21:02:52.694965 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1215 21:02:52.695011 (+ 46us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1215 21:02:52.695096 (+ 85us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1215 21:02:52.695475 (+ 379us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:02:52.696166 (+ 691us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1215 21:02:52.697591 (+ 1425us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:02:52.698251 (+ 660us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:02:52.698285 (+ 34us) server_negotiation.cc:1036] Waiting for connection context
1215 21:02:52.698334 (+ 49us) server_negotiation.cc:300] Negotiation successful
1215 21:02:52.698390 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1763,"thread_start_us":65,"threads_started":1}
I20251215 21:02:52.698619 4482 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c35fb133c71e4054a03b62f133fcaeb2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 21:02:52.699030 4618 tablet_service.cc:1505] Processing CreateTablet for tablet c35fb133c71e4054a03b62f133fcaeb2 (DEFAULT_TABLE table=test-table [id=c830df2bceaa436e99a7ed58a423e1e5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251215 21:02:52.699255 4618 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c35fb133c71e4054a03b62f133fcaeb2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 21:02:52.700609 4781 tablet_bootstrap.cc:492] T c35fb133c71e4054a03b62f133fcaeb2 P 6956a9660a3b4d42867878501794f2d5: Bootstrap starting.
I20251215 21:02:52.701416 4783 tablet_bootstrap.cc:492] T c35fb133c71e4054a03b62f133fcaeb2 P 0a2363aab21d43f68f26fb68238a29f3: Bootstrap starting.
I20251215 21:02:52.701453 4781 tablet_bootstrap.cc:654] T c35fb133c71e4054a03b62f133fcaeb2 P 6956a9660a3b4d42867878501794f2d5: Neither blocks nor log segments found. Creating new log.
I20251215 21:02:52.702111 4782 tablet_bootstrap.cc:492] T c35fb133c71e4054a03b62f133fcaeb2 P f17cfa3c3e5041c092728f920f98dec6: Bootstrap starting.
I20251215 21:02:52.702311 4783 tablet_bootstrap.cc:654] T c35fb133c71e4054a03b62f133fcaeb2 P 0a2363aab21d43f68f26fb68238a29f3: Neither blocks nor log segments found. Creating new log.
I20251215 21:02:52.703032 4782 tablet_bootstrap.cc:654] T c35fb133c71e4054a03b62f133fcaeb2 P f17cfa3c3e5041c092728f920f98dec6: Neither blocks nor log segments found. Creating new log.
I20251215 21:02:52.703905 4781 tablet_bootstrap.cc:492] T c35fb133c71e4054a03b62f133fcaeb2 P 6956a9660a3b4d42867878501794f2d5: No bootstrap required, opened a new log
I20251215 21:02:52.704037 4781 ts_tablet_manager.cc:1403] T c35fb133c71e4054a03b62f133fcaeb2 P 6956a9660a3b4d42867878501794f2d5: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20251215 21:02:52.704226 4782 tablet_bootstrap.cc:492] T c35fb133c71e4054a03b62f133fcaeb2 P f17cfa3c3e5041c092728f920f98dec6: No bootstrap required, opened a new log
I20251215 21:02:52.704308 4782 ts_tablet_manager.cc:1403] T c35fb133c71e4054a03b62f133fcaeb2 P f17cfa3c3e5041c092728f920f98dec6: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20251215 21:02:52.704499 4781 raft_consensus.cc:359] T c35fb133c71e4054a03b62f133fcaeb2 P 6956a9660a3b4d42867878501794f2d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a2363aab21d43f68f26fb68238a29f3" member_type: VOTER last_known_addr { host: "127.28.113.3" port: 35071 } } peers { permanent_uuid: "6956a9660a3b4d42867878501794f2d5" member_type: VOTER last_known_addr { host: "127.28.113.1" port: 35255 } } peers { permanent_uuid: "f17cfa3c3e5041c092728f920f98dec6" member_type: VOTER last_known_addr { host: "127.28.113.2" port: 32787 } }
I20251215 21:02:52.704610 4781 raft_consensus.cc:385] T c35fb133c71e4054a03b62f133fcaeb2 P 6956a9660a3b4d42867878501794f2d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251215 21:02:52.704651 4781 raft_consensus.cc:740] T c35fb133c71e4054a03b62f133fcaeb2 P 6956a9660a3b4d42867878501794f2d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6956a9660a3b4d42867878501794f2d5, State: Initialized, Role: FOLLOWER
I20251215 21:02:52.704716 4782 raft_consensus.cc:359] T c35fb133c71e4054a03b62f133fcaeb2 P f17cfa3c3e5041c092728f920f98dec6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a2363aab21d43f68f26fb68238a29f3" member_type: VOTER last_known_addr { host: "127.28.113.3" port: 35071 } } peers { permanent_uuid: "6956a9660a3b4d42867878501794f2d5" member_type: VOTER last_known_addr { host: "127.28.113.1" port: 35255 } } peers { permanent_uuid: "f17cfa3c3e5041c092728f920f98dec6" member_type: VOTER last_known_addr { host: "127.28.113.2" port: 32787 } }
I20251215 21:02:52.704807 4782 raft_consensus.cc:385] T c35fb133c71e4054a03b62f133fcaeb2 P f17cfa3c3e5041c092728f920f98dec6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251215 21:02:52.704844 4782 raft_consensus.cc:740] T c35fb133c71e4054a03b62f133fcaeb2 P f17cfa3c3e5041c092728f920f98dec6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f17cfa3c3e5041c092728f920f98dec6, State: Initialized, Role: FOLLOWER
I20251215 21:02:52.704815 4781 consensus_queue.cc:260] T c35fb133c71e4054a03b62f133fcaeb2 P 6956a9660a3b4d42867878501794f2d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a2363aab21d43f68f26fb68238a29f3" member_type: VOTER last_known_addr { host: "127.28.113.3" port: 35071 } } peers { permanent_uuid: "6956a9660a3b4d42867878501794f2d5" member_type: VOTER last_known_addr { host: "127.28.113.1" port: 35255 } } peers { permanent_uuid: "f17cfa3c3e5041c092728f920f98dec6" member_type: VOTER last_known_addr { host: "127.28.113.2" port: 32787 } }
I20251215 21:02:52.704967 4782 consensus_queue.cc:260] T c35fb133c71e4054a03b62f133fcaeb2 P f17cfa3c3e5041c092728f920f98dec6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a2363aab21d43f68f26fb68238a29f3" member_type: VOTER last_known_addr { host: "127.28.113.3" port: 35071 } } peers { permanent_uuid: "6956a9660a3b4d42867878501794f2d5" member_type: VOTER last_known_addr { host: "127.28.113.1" port: 35255 } } peers { permanent_uuid: "f17cfa3c3e5041c092728f920f98dec6" member_type: VOTER last_known_addr { host: "127.28.113.2" port: 32787 } }
I20251215 21:02:52.705154 4781 ts_tablet_manager.cc:1434] T c35fb133c71e4054a03b62f133fcaeb2 P 6956a9660a3b4d42867878501794f2d5: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20251215 21:02:52.705317 4782 ts_tablet_manager.cc:1434] T c35fb133c71e4054a03b62f133fcaeb2 P f17cfa3c3e5041c092728f920f98dec6: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20251215 21:02:52.706265 4760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:02:52.705947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.113.62:41229 (local address 127.28.113.2:50475)
1215 21:02:52.706142 (+ 195us) negotiation.cc:107] Waiting for socket to connect
1215 21:02:52.706197 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.113.62:41229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115}
I20251215 21:02:52.706349 4762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:02:52.705864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.113.62:41229 (local address 127.28.113.1:33195)
1215 21:02:52.706233 (+ 369us) negotiation.cc:107] Waiting for socket to connect
1215 21:02:52.706284 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.113.62:41229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":292}
I20251215 21:02:52.708509 4414 heartbeater.cc:499] Master 127.28.113.61:39207 was elected leader, sending a full tablet report...
I20251215 21:02:52.708828 4762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:02:52.708562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.113.62:41229 (local address 127.28.113.1:56971)
1215 21:02:52.708709 (+ 147us) negotiation.cc:107] Waiting for socket to connect
1215 21:02:52.708751 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.113.62:41229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20251215 21:02:52.710170 4550 heartbeater.cc:499] Master 127.28.113.61:39207 was elected leader, sending a full tablet report...
I20251215 21:02:52.710402 4783 tablet_bootstrap.cc:492] T c35fb133c71e4054a03b62f133fcaeb2 P 0a2363aab21d43f68f26fb68238a29f3: No bootstrap required, opened a new log
I20251215 21:02:52.710498 4783 ts_tablet_manager.cc:1403] T c35fb133c71e4054a03b62f133fcaeb2 P 0a2363aab21d43f68f26fb68238a29f3: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20251215 21:02:52.710582 4760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:02:52.710320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.113.62:41229 (local address 127.28.113.2:58677)
1215 21:02:52.710470 (+ 150us) negotiation.cc:107] Waiting for socket to connect
1215 21:02:52.710517 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.113.62:41229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20251215 21:02:52.711045 4783 raft_consensus.cc:359] T c35fb133c71e4054a03b62f133fcaeb2 P 0a2363aab21d43f68f26fb68238a29f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a2363aab21d43f68f26fb68238a29f3" member_type: VOTER last_known_addr { host: "127.28.113.3" port: 35071 } } peers { permanent_uuid: "6956a9660a3b4d42867878501794f2d5" member_type: VOTER last_known_addr { host: "127.28.113.1" port: 35255 } } peers { permanent_uuid: "f17cfa3c3e5041c092728f920f98dec6" member_type: VOTER last_known_addr { host: "127.28.113.2" port: 32787 } }
I20251215 21:02:52.711174 4783 raft_consensus.cc:385] T c35fb133c71e4054a03b62f133fcaeb2 P 0a2363aab21d43f68f26fb68238a29f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251215 21:02:52.711213 4783 raft_consensus.cc:740] T c35fb133c71e4054a03b62f133fcaeb2 P 0a2363aab21d43f68f26fb68238a29f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a2363aab21d43f68f26fb68238a29f3, State: Initialized, Role: FOLLOWER
I20251215 21:02:52.711351 4783 consensus_queue.cc:260] T c35fb133c71e4054a03b62f133fcaeb2 P 0a2363aab21d43f68f26fb68238a29f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a2363aab21d43f68f26fb68238a29f3" member_type: VOTER last_known_addr { host: "127.28.113.3" port: 35071 } } peers { permanent_uuid: "6956a9660a3b4d42867878501794f2d5" member_type: VOTER last_known_addr { host: "127.28.113.1" port: 35255 } } peers { permanent_uuid: "f17cfa3c3e5041c092728f920f98dec6" member_type: VOTER last_known_addr { host: "127.28.113.2" port: 32787 } }
I20251215 21:02:52.711714 4783 ts_tablet_manager.cc:1434] T c35fb133c71e4054a03b62f133fcaeb2 P 0a2363aab21d43f68f26fb68238a29f3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20251215 21:02:52.712296 4761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:02:52.712045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.113.62:41229 (local address 127.28.113.3:33227)
1215 21:02:52.712189 (+ 144us) negotiation.cc:107] Waiting for socket to connect
1215 21:02:52.712234 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.113.62:41229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20251215 21:02:52.713157 4686 heartbeater.cc:499] Master 127.28.113.61:39207 was elected leader, sending a full tablet report...
I20251215 21:02:52.713518 4761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:02:52.713029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.113.62:41229 (local address 127.28.113.3:35159)
1215 21:02:52.713410 (+ 381us) negotiation.cc:107] Waiting for socket to connect
1215 21:02:52.713457 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.113.62:41229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":263}
I20251215 21:02:52.770325 4788 raft_consensus.cc:493] T c35fb133c71e4054a03b62f133fcaeb2 P f17cfa3c3e5041c092728f920f98dec6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 21:02:52.770462 4788 raft_consensus.cc:515] T c35fb133c71e4054a03b62f133fcaeb2 P f17cfa3c3e5041c092728f920f98dec6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a2363aab21d43f68f26fb68238a29f3" member_type: VOTER last_known_addr { host: "127.28.113.3" port: 35071 } } peers { permanent_uuid: "6956a9660a3b4d42867878501794f2d5" member_type: VOTER last_known_addr { host: "127.28.113.1" port: 35255 } } peers { permanent_uuid: "f17cfa3c3e5041c092728f920f98dec6" member_type: VOTER last_known_addr { host: "127.28.113.2" port: 32787 } }
I20251215 21:02:52.770947 4788 leader_election.cc:290] T c35fb133c71e4054a03b62f133fcaeb2 P f17cfa3c3e5041c092728f920f98dec6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0a2363aab21d43f68f26fb68238a29f3 (127.28.113.3:35071), 6956a9660a3b4d42867878501794f2d5 (127.28.113.1:35255)
I20251215 21:02:52.771389 4638 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c35fb133c71e4054a03b62f133fcaeb2" candidate_uuid: "f17cfa3c3e5041c092728f920f98dec6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a2363aab21d43f68f26fb68238a29f3" is_pre_election: true
I20251215 21:02:52.771423 4366 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c35fb133c71e4054a03b62f133fcaeb2" candidate_uuid: "f17cfa3c3e5041c092728f920f98dec6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6956a9660a3b4d42867878501794f2d5" is_pre_election: true
I20251215 21:02:52.771543 4366 raft_consensus.cc:2468] T c35fb133c71e4054a03b62f133fcaeb2 P 6956a9660a3b4d42867878501794f2d5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f17cfa3c3e5041c092728f920f98dec6 in term 0.
I20251215 21:02:52.771543 4638 raft_consensus.cc:2468] T c35fb133c71e4054a03b62f133fcaeb2 P 0a2363aab21d43f68f26fb68238a29f3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f17cfa3c3e5041c092728f920f98dec6 in term 0.
I20251215 21:02:52.771862 4436 leader_election.cc:304] T c35fb133c71e4054a03b62f133fcaeb2 P f17cfa3c3e5041c092728f920f98dec6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a2363aab21d43f68f26fb68238a29f3, f17cfa3c3e5041c092728f920f98dec6; no voters:
I20251215 21:02:52.772029 4788 raft_consensus.cc:2804] T c35fb133c71e4054a03b62f133fcaeb2 P f17cfa3c3e5041c092728f920f98dec6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251215 21:02:52.772101 4788 raft_consensus.cc:493] T c35fb133c71e4054a03b62f133fcaeb2 P f17cfa3c3e5041c092728f920f98dec6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 21:02:52.772141 4788 raft_consensus.cc:3060] T c35fb133c71e4054a03b62f133fcaeb2 P f17cfa3c3e5041c092728f920f98dec6 [term 0 FOLLOWER]: Advancing to term 1
I20251215 21:02:52.773185 4788 raft_consensus.cc:515] T c35fb133c71e4054a03b62f133fcaeb2 P f17cfa3c3e5041c092728f920f98dec6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a2363aab21d43f68f26fb68238a29f3" member_type: VOTER last_known_addr { host: "127.28.113.3" port: 35071 } } peers { permanent_uuid: "6956a9660a3b4d42867878501794f2d5" member_type: VOTER last_known_addr { host: "127.28.113.1" port: 35255 } } peers { permanent_uuid: "f17cfa3c3e5041c092728f920f98dec6" member_type: VOTER last_known_addr { host: "127.28.113.2" port: 32787 } }
I20251215 21:02:52.773486 4788 leader_election.cc:290] T c35fb133c71e4054a03b62f133fcaeb2 P f17cfa3c3e5041c092728f920f98dec6 [CANDIDATE]: Term 1 election: Requested vote from peers 0a2363aab21d43f68f26fb68238a29f3 (127.28.113.3:35071), 6956a9660a3b4d42867878501794f2d5 (127.28.113.1:35255)
I20251215 21:02:52.773836 4638 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c35fb133c71e4054a03b62f133fcaeb2" candidate_uuid: "f17cfa3c3e5041c092728f920f98dec6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a2363aab21d43f68f26fb68238a29f3"
I20251215 21:02:52.773855 4366 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c35fb133c71e4054a03b62f133fcaeb2" candidate_uuid: "f17cfa3c3e5041c092728f920f98dec6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6956a9660a3b4d42867878501794f2d5"
I20251215 21:02:52.773952 4366 raft_consensus.cc:3060] T c35fb133c71e4054a03b62f133fcaeb2 P 6956a9660a3b4d42867878501794f2d5 [term 0 FOLLOWER]: Advancing to term 1
I20251215 21:02:52.773952 4638 raft_consensus.cc:3060] T c35fb133c71e4054a03b62f133fcaeb2 P 0a2363aab21d43f68f26fb68238a29f3 [term 0 FOLLOWER]: Advancing to term 1
I20251215 21:02:52.774899 4638 raft_consensus.cc:2468] T c35fb133c71e4054a03b62f133fcaeb2 P 0a2363aab21d43f68f26fb68238a29f3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f17cfa3c3e5041c092728f920f98dec6 in term 1.
I20251215 21:02:52.774899 4366 raft_consensus.cc:2468] T c35fb133c71e4054a03b62f133fcaeb2 P 6956a9660a3b4d42867878501794f2d5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f17cfa3c3e5041c092728f920f98dec6 in term 1.
I20251215 21:02:52.775175 4436 leader_election.cc:304] T c35fb133c71e4054a03b62f133fcaeb2 P f17cfa3c3e5041c092728f920f98dec6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a2363aab21d43f68f26fb68238a29f3, f17cfa3c3e5041c092728f920f98dec6; no voters:
I20251215 21:02:52.775324 4788 raft_consensus.cc:2804] T c35fb133c71e4054a03b62f133fcaeb2 P f17cfa3c3e5041c092728f920f98dec6 [term 1 FOLLOWER]: Leader election won for term 1
I20251215 21:02:52.775514 4788 raft_consensus.cc:697] T c35fb133c71e4054a03b62f133fcaeb2 P f17cfa3c3e5041c092728f920f98dec6 [term 1 LEADER]: Becoming Leader. State: Replica: f17cfa3c3e5041c092728f920f98dec6, State: Running, Role: LEADER
I20251215 21:02:52.775645 4788 consensus_queue.cc:237] T c35fb133c71e4054a03b62f133fcaeb2 P f17cfa3c3e5041c092728f920f98dec6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a2363aab21d43f68f26fb68238a29f3" member_type: VOTER last_known_addr { host: "127.28.113.3" port: 35071 } } peers { permanent_uuid: "6956a9660a3b4d42867878501794f2d5" member_type: VOTER last_known_addr { host: "127.28.113.1" port: 35255 } } peers { permanent_uuid: "f17cfa3c3e5041c092728f920f98dec6" member_type: VOTER last_known_addr { host: "127.28.113.2" port: 32787 } }
I20251215 21:02:52.776224 4760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:02:52.775985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.113.62:41229 (local address 127.28.113.2:55453)
1215 21:02:52.776087 (+ 102us) negotiation.cc:107] Waiting for socket to connect
1215 21:02:52.776155 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.113.62:41229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
I20251215 21:02:52.777354 4031 catalog_manager.cc:5654] T c35fb133c71e4054a03b62f133fcaeb2 P f17cfa3c3e5041c092728f920f98dec6 reported cstate change: term changed from 0 to 1, leader changed from <none> to f17cfa3c3e5041c092728f920f98dec6 (127.28.113.2). New cstate: current_term: 1 leader_uuid: "f17cfa3c3e5041c092728f920f98dec6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a2363aab21d43f68f26fb68238a29f3" member_type: VOTER last_known_addr { host: "127.28.113.3" port: 35071 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6956a9660a3b4d42867878501794f2d5" member_type: VOTER last_known_addr { host: "127.28.113.1" port: 35255 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f17cfa3c3e5041c092728f920f98dec6" member_type: VOTER last_known_addr { host: "127.28.113.2" port: 32787 } health_report { overall_health: HEALTHY } } }
I20251215 21:02:52.798486 4780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:02:52.796352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34310 (local address 127.28.113.2:32787)
1215 21:02:52.796474 (+ 122us) server_negotiation.cc:207] Beginning negotiation
1215 21:02:52.796478 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1215 21:02:52.796490 (+ 12us) server_negotiation.cc:408] Connection header received
1215 21:02:52.796533 (+ 43us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1215 21:02:52.796535 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1215 21:02:52.796580 (+ 45us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1215 21:02:52.796651 (+ 71us) server_negotiation.cc:227] Negotiated authn=TOKEN
1215 21:02:52.796939 (+ 288us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:02:52.797411 (+ 472us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1215 21:02:52.797969 (+ 558us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:02:52.798096 (+ 127us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:02:52.798132 (+ 36us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1215 21:02:52.798208 (+ 76us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1215 21:02:52.798259 (+ 51us) server_negotiation.cc:1036] Waiting for connection context
1215 21:02:52.798341 (+ 82us) server_negotiation.cc:300] Negotiation successful
1215 21:02:52.798382 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20251215 21:02:52.800493 4638 raft_consensus.cc:1275] T c35fb133c71e4054a03b62f133fcaeb2 P 0a2363aab21d43f68f26fb68238a29f3 [term 1 FOLLOWER]: Refusing update from remote peer f17cfa3c3e5041c092728f920f98dec6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251215 21:02:52.800560 4366 raft_consensus.cc:1275] T c35fb133c71e4054a03b62f133fcaeb2 P 6956a9660a3b4d42867878501794f2d5 [term 1 FOLLOWER]: Refusing update from remote peer f17cfa3c3e5041c092728f920f98dec6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251215 21:02:52.800976 4790 consensus_queue.cc:1048] T c35fb133c71e4054a03b62f133fcaeb2 P f17cfa3c3e5041c092728f920f98dec6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6956a9660a3b4d42867878501794f2d5" member_type: VOTER last_known_addr { host: "127.28.113.1" port: 35255 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251215 21:02:52.801138 4788 consensus_queue.cc:1048] T c35fb133c71e4054a03b62f133fcaeb2 P f17cfa3c3e5041c092728f920f98dec6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a2363aab21d43f68f26fb68238a29f3" member_type: VOTER last_known_addr { host: "127.28.113.3" port: 35071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251215 21:02:52.802814 4761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:02:52.802514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.113.62:41229 (local address 127.28.113.3:43403)
1215 21:02:52.802671 (+ 157us) negotiation.cc:107] Waiting for socket to connect
1215 21:02:52.802734 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.113.62:41229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20251215 21:02:52.803125 4762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:02:52.802743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.113.62:41229 (local address 127.28.113.1:34751)
1215 21:02:52.802998 (+ 255us) negotiation.cc:107] Waiting for socket to connect
1215 21:02:52.803059 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.113.62:41229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181}
I20251215 21:02:52.805662 4760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:02:52.805411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.113.62:41229 (local address 127.28.113.2:54843)
1215 21:02:52.805550 (+ 139us) negotiation.cc:107] Waiting for socket to connect
1215 21:02:52.805596 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.113.62:41229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20251215 21:02:52.806694 4761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:02:52.806071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.113.62:41229 (local address 127.28.113.3:55121)
1215 21:02:52.806581 (+ 510us) negotiation.cc:107] Waiting for socket to connect
1215 21:02:52.806630 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.113.62:41229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":444}
I20251215 21:02:52.807297 4793 mvcc.cc:204] Tried to move back new op lower bound from 7232850218186829824 to 7232850218090643456. Current Snapshot: MvccSnapshot[applied={T|T < 7232850218186829824 or (T in {7232850218186829824})}]
I20251215 21:02:52.808375 4760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:02:52.808164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.113.62:41229 (local address 127.28.113.2:46417)
1215 21:02:52.808290 (+ 126us) negotiation.cc:107] Waiting for socket to connect
1215 21:02:52.808325 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.113.62:41229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20251215 21:02:52.810272 4762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:02:52.810000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.113.62:41229 (local address 127.28.113.1:33949)
1215 21:02:52.810156 (+ 156us) negotiation.cc:107] Waiting for socket to connect
1215 21:02:52.810205 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.113.62:41229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20251215 21:02:52.810808 4031 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56104:
table { table_name: "test-table" } modify_external_catalogs: true
I20251215 21:02:52.810932 4031 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56104:
table { table_name: "test-table" } modify_external_catalogs: true
I20251215 21:02:52.814194 4031 catalog_manager.cc:5941] T 00000000000000000000000000000000 P de9d22b45210451f997093f481e4dfc6: Sending DeleteTablet for 3 replicas of tablet c35fb133c71e4054a03b62f133fcaeb2
I20251215 21:02:52.814806 4618 tablet_service.cc:1552] Processing DeleteTablet for tablet c35fb133c71e4054a03b62f133fcaeb2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-15 21:02:52 UTC) from {username='oryx', principal='oryx/127.28.113.61@KRBTEST.COM'} at 127.0.0.1:51494
I20251215 21:02:52.814868 4346 tablet_service.cc:1552] Processing DeleteTablet for tablet c35fb133c71e4054a03b62f133fcaeb2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-15 21:02:52 UTC) from {username='oryx', principal='oryx/127.28.113.61@KRBTEST.COM'} at 127.0.0.1:45076
I20251215 21:02:52.814910 4482 tablet_service.cc:1552] Processing DeleteTablet for tablet c35fb133c71e4054a03b62f133fcaeb2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-15 21:02:52 UTC) from {username='oryx', principal='oryx/127.28.113.61@KRBTEST.COM'} at 127.0.0.1:34304
I20251215 21:02:52.815145 4801 tablet_replica.cc:333] T c35fb133c71e4054a03b62f133fcaeb2 P 0a2363aab21d43f68f26fb68238a29f3: stopping tablet replica
I20251215 21:02:52.815263 4801 raft_consensus.cc:2243] T c35fb133c71e4054a03b62f133fcaeb2 P 0a2363aab21d43f68f26fb68238a29f3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251215 21:02:52.815328 29124 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu with pid 4283
I20251215 21:02:52.815522 4803 tablet_replica.cc:333] T c35fb133c71e4054a03b62f133fcaeb2 P f17cfa3c3e5041c092728f920f98dec6: stopping tablet replica
I20251215 21:02:52.815609 4803 raft_consensus.cc:2243] T c35fb133c71e4054a03b62f133fcaeb2 P f17cfa3c3e5041c092728f920f98dec6 [term 1 LEADER]: Raft consensus shutting down.
I20251215 21:02:52.815845 4803 raft_consensus.cc:2272] T c35fb133c71e4054a03b62f133fcaeb2 P f17cfa3c3e5041c092728f920f98dec6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251215 21:02:52.815865 4801 raft_consensus.cc:2272] T c35fb133c71e4054a03b62f133fcaeb2 P 0a2363aab21d43f68f26fb68238a29f3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251215 21:02:52.816502 4803 ts_tablet_manager.cc:1916] T c35fb133c71e4054a03b62f133fcaeb2 P f17cfa3c3e5041c092728f920f98dec6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251215 21:02:52.816502 4801 ts_tablet_manager.cc:1916] T c35fb133c71e4054a03b62f133fcaeb2 P 0a2363aab21d43f68f26fb68238a29f3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251215 21:02:52.818655 4803 ts_tablet_manager.cc:1929] T c35fb133c71e4054a03b62f133fcaeb2 P f17cfa3c3e5041c092728f920f98dec6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251215 21:02:52.818655 4801 ts_tablet_manager.cc:1929] T c35fb133c71e4054a03b62f133fcaeb2 P 0a2363aab21d43f68f26fb68238a29f3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251215 21:02:52.818728 4803 log.cc:1199] T c35fb133c71e4054a03b62f133fcaeb2 P f17cfa3c3e5041c092728f920f98dec6: Deleting WAL directory at /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765832506145157-29124-0/minicluster-data/ts-1/wal/wals/c35fb133c71e4054a03b62f133fcaeb2
I20251215 21:02:52.818734 4801 log.cc:1199] T c35fb133c71e4054a03b62f133fcaeb2 P 0a2363aab21d43f68f26fb68238a29f3: Deleting WAL directory at /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765832506145157-29124-0/minicluster-data/ts-2/wal/wals/c35fb133c71e4054a03b62f133fcaeb2
I20251215 21:02:52.818977 4803 ts_tablet_manager.cc:1950] T c35fb133c71e4054a03b62f133fcaeb2 P f17cfa3c3e5041c092728f920f98dec6: Deleting consensus metadata
I20251215 21:02:52.818995 4801 ts_tablet_manager.cc:1950] T c35fb133c71e4054a03b62f133fcaeb2 P 0a2363aab21d43f68f26fb68238a29f3: Deleting consensus metadata
I20251215 21:02:52.819680 4017 catalog_manager.cc:4985] TS 0a2363aab21d43f68f26fb68238a29f3 (127.28.113.3:35071): tablet c35fb133c71e4054a03b62f133fcaeb2 (table test-table [id=c830df2bceaa436e99a7ed58a423e1e5]) successfully deleted
I20251215 21:02:52.819839 4017 catalog_manager.cc:4985] TS f17cfa3c3e5041c092728f920f98dec6 (127.28.113.2:32787): tablet c35fb133c71e4054a03b62f133fcaeb2 (table test-table [id=c830df2bceaa436e99a7ed58a423e1e5]) successfully deleted
I20251215 21:02:52.822999 29124 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu with pid 4419
I20251215 21:02:52.823127 4779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:02:52.822887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.113.1:35255 (local address 127.0.0.1:45080)
1215 21:02:52.823007 (+ 120us) negotiation.cc:107] Waiting for socket to connect
1215 21:02:52.823050 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.113.1:35255: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20251215 21:02:52.824846 4017 catalog_manager.cc:4712] TS 6956a9660a3b4d42867878501794f2d5 (127.28.113.1:35255): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c35fb133c71e4054a03b62f133fcaeb2: Network error: Client connection negotiation failed: client connection to 127.28.113.1:35255: connect: Connection refused (error 111)
I20251215 21:02:52.829703 29124 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu with pid 4555
I20251215 21:02:52.834537 29124 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu with pid 3886
I20251215 21:02:52.839553 29124 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu with pid 4004
2025-12-15T21:02:52Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (9851 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765832506145157-29124-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Dec 15 21:02:52 dist-test-slave-bjsf krb5kdc[4806](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Dec 15 21:02:52 dist-test-slave-bjsf krb5kdc[4806](info): set up 2 sockets
Dec 15 21:02:52 dist-test-slave-bjsf krb5kdc[4806](info): commencing operation
krb5kdc: starting...
W20251215 21:02:54.664436 29124 mini_kdc.cc:121] Time spent starting KDC: real 1.804s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765832506145157-29124-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765832506145157-29124-0/krb5kdc/test-user.keytab.
Dec 15 21:02:54 dist-test-slave-bjsf krb5kdc[4806](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765832574, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-12-15T21:02:54Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-12-15T21:02:54Z Disabled control of system clock
WARNING: no policy specified for oryx/127.28.113.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.28.113.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.28.113.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765832506145157-29124-0/krb5kdc/oryx.keytab.
Entry for principal oryx/127.28.113.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765832506145157-29124-0/krb5kdc/oryx.keytab.
WARNING: no policy specified for HTTP/127.28.113.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.28.113.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.28.113.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765832506145157-29124-0/krb5kdc/oryx.keytab.
Entry for principal HTTP/127.28.113.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765832506145157-29124-0/krb5kdc/oryx.keytab.
I20251215 21:02:54.792337 29124 external_mini_cluster.cc:1366] Running /tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu
/tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765832506145157-29124-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765832506145157-29124-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765832506145157-29124-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765832506145157-29124-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.113.62:38827
--webserver_interface=127.28.113.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765832506145157-29124-0/krb5kdc/oryx.keytab
--principal=oryx/127.28.113.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.28.113.20:35731
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.113.62:38827
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765832506145157-29124-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765832506145157-29124-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765832506145157-29124-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251215 21:02:54.891888 4822 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251215 21:02:54.892148 4822 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251215 21:02:54.892205 4822 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251215 21:02:54.895344 4822 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251215 21:02:54.895401 4822 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20251215 21:02:54.895417 4822 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251215 21:02:54.895437 4822 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251215 21:02:54.895455 4822 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251215 21:02:54.895473 4822 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251215 21:02:54.899590 4822 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.113.20:35731
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765832506145157-29124-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765832506145157-29124-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.113.62:38827
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765832506145157-29124-0/krb5kdc/oryx.keytab
--principal=oryx/127.28.113.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.113.62:38827
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765832506145157-29124-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.113.62
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4822
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765832506145157-29124-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision dc741732bb6a40213e11a13869fad06a49a209a3
build type DEBUG
built by None at 15 Dec 2025 20:43:14 UTC on 24a791456cd2
build id 9530
I20251215 21:02:54.900609 4822 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251215 21:02:54.901399 4822 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251215 21:02:54.906863 4827 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251215 21:02:54.906970 4822 server_base.cc:1047] running on GCE node
W20251215 21:02:54.906858 4828 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251215 21:02:54.906885 4830 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251215 21:02:54.907603 4822 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251215 21:02:54.908502 4822 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251215 21:02:54.909659 4822 hybrid_clock.cc:648] HybridClock initialized: now 1765832574909646 us; error 32 us; skew 500 ppm
Dec 15 21:02:54 dist-test-slave-bjsf krb5kdc[4806](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765832574, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.113.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251215 21:02:54.912067 4822 init.cc:377] Logged in from keytab as oryx/127.28.113.62@KRBTEST.COM (short username oryx)
I20251215 21:02:54.913076 4822 webserver.cc:492] Webserver started at http://127.28.113.62:40023/ using document root <none> and password file <none>
I20251215 21:02:54.913586 4822 fs_manager.cc:362] Metadata directory not provided
I20251215 21:02:54.913637 4822 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251215 21:02:54.913813 4822 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251215 21:02:54.915445 4822 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765832506145157-29124-0/minicluster-data/master-0/data/instance:
uuid: "4fc227cc99184e17b315174b90a73b5c"
format_stamp: "Formatted at 2025-12-15 21:02:54 on dist-test-slave-bjsf"
server_key: "2f64100deb296e8a394fdc23d3413d18"
server_key_iv: "3fbb90bd86aac7430f9b2d9b12f4d1b7"
server_key_version: "encryptionkey@0"
I20251215 21:02:54.915897 4822 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765832506145157-29124-0/minicluster-data/master-0/wal/instance:
uuid: "4fc227cc99184e17b315174b90a73b5c"
format_stamp: "Formatted at 2025-12-15 21:02:54 on dist-test-slave-bjsf"
server_key: "2f64100deb296e8a394fdc23d3413d18"
server_key_iv: "3fbb90bd86aac7430f9b2d9b12f4d1b7"
server_key_version: "encryptionkey@0"
I20251215 21:02:54.919050 4822 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20251215 21:02:54.921159 4837 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251215 21:02:54.922101 4822 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20251215 21:02:54.922209 4822 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765832506145157-29124-0/minicluster-data/master-0/data,/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765832506145157-29124-0/minicluster-data/master-0/wal
uuid: "4fc227cc99184e17b315174b90a73b5c"
format_stamp: "Formatted at 2025-12-15 21:02:54 on dist-test-slave-bjsf"
server_key: "2f64100deb296e8a394fdc23d3413d18"
server_key_iv: "3fbb90bd86aac7430f9b2d9b12f4d1b7"
server_key_version: "encryptionkey@0"
I20251215 21:02:54.922297 4822 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765832506145157-29124-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765832506145157-29124-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765832506145157-29124-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251215 21:02:54.946698 4822 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251215 21:02:54.949553 4822 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251215 21:02:54.949749 4822 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251215 21:02:54.957234 4822 rpc_server.cc:307] RPC server started. Bound to: 127.28.113.62:38827
I20251215 21:02:54.957433 4899 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.28.113.62:38827 every 8 connection(s)
I20251215 21:02:54.958201 4822 server_base.cc:1179] Dumped server information to /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765832506145157-29124-0/minicluster-data/master-0/data/info.pb
I20251215 21:02:54.960791 4900 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 21:02:54.965543 4900 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4fc227cc99184e17b315174b90a73b5c: Bootstrap starting.
I20251215 21:02:54.967135 29124 external_mini_cluster.cc:1428] Started /tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu as pid 4822
I20251215 21:02:54.967247 29124 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765832506145157-29124-0/minicluster-data/master-0/wal/instance
I20251215 21:02:54.967473 29124 external_mini_cluster.cc:1468] Setting key 054e3a27c10344a01365f609f96b1732
I20251215 21:02:54.967844 4900 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4fc227cc99184e17b315174b90a73b5c: Neither blocks nor log segments found. Creating new log.
I20251215 21:02:54.968672 4900 log.cc:826] T 00000000000000000000000000000000 P 4fc227cc99184e17b315174b90a73b5c: Log is configured to *not* fsync() on all Append() calls
I20251215 21:02:54.970501 4900 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4fc227cc99184e17b315174b90a73b5c: No bootstrap required, opened a new log
I20251215 21:02:54.973280 4900 raft_consensus.cc:359] T 00000000000000000000000000000000 P 4fc227cc99184e17b315174b90a73b5c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fc227cc99184e17b315174b90a73b5c" member_type: VOTER last_known_addr { host: "127.28.113.62" port: 38827 } }
I20251215 21:02:54.973497 4900 raft_consensus.cc:385] T 00000000000000000000000000000000 P 4fc227cc99184e17b315174b90a73b5c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251215 21:02:54.973556 4900 raft_consensus.cc:740] T 00000000000000000000000000000000 P 4fc227cc99184e17b315174b90a73b5c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4fc227cc99184e17b315174b90a73b5c, State: Initialized, Role: FOLLOWER
I20251215 21:02:54.974033 4900 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4fc227cc99184e17b315174b90a73b5c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fc227cc99184e17b315174b90a73b5c" member_type: VOTER last_known_addr { host: "127.28.113.62" port: 38827 } }
I20251215 21:02:54.974174 4900 raft_consensus.cc:399] T 00000000000000000000000000000000 P 4fc227cc99184e17b315174b90a73b5c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251215 21:02:54.974237 4900 raft_consensus.cc:493] T 00000000000000000000000000000000 P 4fc227cc99184e17b315174b90a73b5c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251215 21:02:54.974367 4900 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 4fc227cc99184e17b315174b90a73b5c [term 0 FOLLOWER]: Advancing to term 1
Dec 15 21:02:54 dist-test-slave-bjsf krb5kdc[4806](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765832574, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.28.113.62@KRBTEST.COM
I20251215 21:02:54.975178 4900 raft_consensus.cc:515] T 00000000000000000000000000000000 P 4fc227cc99184e17b315174b90a73b5c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fc227cc99184e17b315174b90a73b5c" member_type: VOTER last_known_addr { host: "127.28.113.62" port: 38827 } }
I20251215 21:02:54.975489 4900 leader_election.cc:304] T 00000000000000000000000000000000 P 4fc227cc99184e17b315174b90a73b5c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4fc227cc99184e17b315174b90a73b5c; no voters:
I20251215 21:02:54.975733 4900 leader_election.cc:290] T 00000000000000000000000000000000 P 4fc227cc99184e17b315174b90a73b5c [CANDIDATE]: Term 1 election: Requested vote from peers
I20251215 21:02:54.976123 4905 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 4fc227cc99184e17b315174b90a73b5c [term 1 FOLLOWER]: Leader election won for term 1
I20251215 21:02:54.976611 4905 raft_consensus.cc:697] T 00000000000000000000000000000000 P 4fc227cc99184e17b315174b90a73b5c [term 1 LEADER]: Becoming Leader. State: Replica: 4fc227cc99184e17b315174b90a73b5c, State: Running, Role: LEADER
I20251215 21:02:54.976883 4900 sys_catalog.cc:565] T 00000000000000000000000000000000 P 4fc227cc99184e17b315174b90a73b5c [sys.catalog]: configured and running, proceeding with master startup.
I20251215 21:02:54.976953 4905 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4fc227cc99184e17b315174b90a73b5c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fc227cc99184e17b315174b90a73b5c" member_type: VOTER last_known_addr { host: "127.28.113.62" port: 38827 } }
I20251215 21:02:54.978375 4907 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4fc227cc99184e17b315174b90a73b5c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4fc227cc99184e17b315174b90a73b5c. Latest consensus state: current_term: 1 leader_uuid: "4fc227cc99184e17b315174b90a73b5c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fc227cc99184e17b315174b90a73b5c" member_type: VOTER last_known_addr { host: "127.28.113.62" port: 38827 } } }
I20251215 21:02:54.978475 4907 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4fc227cc99184e17b315174b90a73b5c [sys.catalog]: This master's current role is: LEADER
I20251215 21:02:54.978754 4906 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4fc227cc99184e17b315174b90a73b5c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4fc227cc99184e17b315174b90a73b5c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fc227cc99184e17b315174b90a73b5c" member_type: VOTER last_known_addr { host: "127.28.113.62" port: 38827 } } }
I20251215 21:02:54.978821 4906 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4fc227cc99184e17b315174b90a73b5c [sys.catalog]: This master's current role is: LEADER
I20251215 21:02:54.979202 4917 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251215 21:02:54.980849 4903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:02:54.968519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34140 (local address 127.28.113.62:38827)
1215 21:02:54.968901 (+ 382us) server_negotiation.cc:207] Beginning negotiation
1215 21:02:54.968911 (+ 10us) server_negotiation.cc:400] Waiting for connection header
1215 21:02:54.968944 (+ 33us) server_negotiation.cc:408] Connection header received
1215 21:02:54.969575 (+ 631us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1215 21:02:54.969596 (+ 21us) server_negotiation.cc:462] Received NEGOTIATE request from client
1215 21:02:54.969878 (+ 282us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1215 21:02:54.970153 (+ 275us) server_negotiation.cc:227] Negotiated authn=SASL
1215 21:02:54.971149 (+ 996us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:02:54.971873 (+ 724us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1215 21:02:54.972761 (+ 888us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:02:54.972988 (+ 227us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:02:54.975312 (+ 2324us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1215 21:02:54.975335 (+ 23us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1215 21:02:54.975349 (+ 14us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1215 21:02:54.975385 (+ 36us) server_negotiation.cc:923] calling sasl_server_start()
1215 21:02:54.977597 (+ 2212us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1215 21:02:54.978103 (+ 506us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1215 21:02:54.978109 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1215 21:02:54.978126 (+ 17us) server_negotiation.cc:968] calling sasl_server_step()
1215 21:02:54.978214 (+ 88us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1215 21:02:54.978484 (+ 270us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1215 21:02:54.978488 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1215 21:02:54.978490 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1215 21:02:54.978911 (+ 421us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1215 21:02:54.979723 (+ 812us) server_negotiation.cc:1036] Waiting for connection context
1215 21:02:54.979882 (+ 159us) server_negotiation.cc:300] Negotiation successful
1215 21:02:54.980163 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":256,"thread_start_us":117,"threads_started":1}
I20251215 21:02:54.981899 4917 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251215 21:02:54.986755 4917 catalog_manager.cc:1357] Generated new cluster ID: 462fd5cb6694459985377a8a67c16bf9
I20251215 21:02:54.986824 4917 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251215 21:02:54.999979 4917 catalog_manager.cc:1380] Generated new certificate authority record
I20251215 21:02:55.000936 4917 catalog_manager.cc:1514] Loading token signing keys...
I20251215 21:02:55.004997 4917 catalog_manager.cc:6027] T 00000000000000000000000000000000 P 4fc227cc99184e17b315174b90a73b5c: Generated new TSK 0
I20251215 21:02:55.005596 4917 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.28.113.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.28.113.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.28.113.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765832506145157-29124-0/krb5kdc/oryx.keytab.
Entry for principal oryx/127.28.113.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765832506145157-29124-0/krb5kdc/oryx.keytab.
WARNING: no policy specified for HTTP/127.28.113.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.28.113.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.28.113.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765832506145157-29124-0/krb5kdc/oryx.keytab.
Entry for principal HTTP/127.28.113.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765832506145157-29124-0/krb5kdc/oryx.keytab.
I20251215 21:02:55.052047 29124 external_mini_cluster.cc:1366] Running /tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu
/tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765832506145157-29124-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765832506145157-29124-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765832506145157-29124-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765832506145157-29124-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.113.1:0
--local_ip_for_outbound_sockets=127.28.113.1
--webserver_interface=127.28.113.1
--webserver_port=0
--tserver_master_addrs=127.28.113.62:38827
--keytab_file=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765832506145157-29124-0/krb5kdc/oryx.keytab
--principal=oryx/127.28.113.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.28.113.20:35731
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765832506145157-29124-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765832506145157-29124-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765832506145157-29124-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251215 21:02:55.153419 4928 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251215 21:02:55.153609 4928 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251215 21:02:55.153669 4928 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251215 21:02:55.156800 4928 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251215 21:02:55.156849 4928 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251215 21:02:55.156910 4928 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20251215 21:02:55.156971 4928 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.113.1
I20251215 21:02:55.160893 4928 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.113.20:35731
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765832506145157-29124-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765832506145157-29124-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765832506145157-29124-0/krb5kdc/oryx.keytab
--principal=oryx/127.28.113.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.113.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765832506145157-29124-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.113.1
--webserver_port=0
--webserver_require_spnego=true
--enable_txn_system_client_init=true
--tserver_master_addrs=127.28.113.62:38827
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4928
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.113.1
--log_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765832506145157-29124-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision dc741732bb6a40213e11a13869fad06a49a209a3
build type DEBUG
built by None at 15 Dec 2025 20:43:14 UTC on 24a791456cd2
build id 9530
I20251215 21:02:55.161800 4928 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251215 21:02:55.162561 4928 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251215 21:02:55.168675 4934 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251215 21:02:55.168674 4933 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251215 21:02:55.168685 4936 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251215 21:02:55.168996 4928 server_base.cc:1047] running on GCE node
I20251215 21:02:55.169315 4928 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251215 21:02:55.169828 4928 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251215 21:02:55.170979 4928 hybrid_clock.cc:648] HybridClock initialized: now 1765832575170964 us; error 30 us; skew 500 ppm
Dec 15 21:02:55 dist-test-slave-bjsf krb5kdc[4806](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765832575, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.113.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251215 21:02:55.173408 4928 init.cc:377] Logged in from keytab as oryx/127.28.113.1@KRBTEST.COM (short username oryx)
I20251215 21:02:55.174338 4928 webserver.cc:492] Webserver started at http://127.28.113.1:37777/ using document root <none> and password file <none>
I20251215 21:02:55.174820 4928 fs_manager.cc:362] Metadata directory not provided
I20251215 21:02:55.174862 4928 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251215 21:02:55.175004 4928 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251215 21:02:55.176630 4928 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765832506145157-29124-0/minicluster-data/ts-0/data/instance:
uuid: "38e72b9cde82428ea8acd83a5194f75d"
format_stamp: "Formatted at 2025-12-15 21:02:55 on dist-test-slave-bjsf"
server_key: "576c1554a8ee6e83a0681aea96b5b0c1"
server_key_iv: "6d1612537f1d00c03f71c238eabb331d"
server_key_version: "encryptionkey@0"
I20251215 21:02:55.177040 4928 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765832506145157-29124-0/minicluster-data/ts-0/wal/instance:
uuid: "38e72b9cde82428ea8acd83a5194f75d"
format_stamp: "Formatted at 2025-12-15 21:02:55 on dist-test-slave-bjsf"
server_key: "576c1554a8ee6e83a0681aea96b5b0c1"
server_key_iv: "6d1612537f1d00c03f71c238eabb331d"
server_key_version: "encryptionkey@0"
I20251215 21:02:55.180189 4928 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20251215 21:02:55.182108 4943 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251215 21:02:55.183069 4928 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20251215 21:02:55.183158 4928 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765832506145157-29124-0/minicluster-data/ts-0/data,/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765832506145157-29124-0/minicluster-data/ts-0/wal
uuid: "38e72b9cde82428ea8acd83a5194f75d"
format_stamp: "Formatted at 2025-12-15 21:02:55 on dist-test-slave-bjsf"
server_key: "576c1554a8ee6e83a0681aea96b5b0c1"
server_key_iv: "6d1612537f1d00c03f71c238eabb331d"
server_key_version: "encryptionkey@0"
I20251215 21:02:55.183247 4928 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765832506145157-29124-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765832506145157-29124-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765832506145157-29124-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251215 21:02:55.199951 4928 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251215 21:02:55.202533 4928 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251215 21:02:55.202709 4928 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251215 21:02:55.204053 4928 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251215 21:02:55.204097 4928 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251215 21:02:55.204136 4928 ts_tablet_manager.cc:616] Registered 0 tablets
I20251215 21:02:55.204164 4928 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Dec 15 21:02:55 dist-test-slave-bjsf krb5kdc[4806](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765832575, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.113.1@KRBTEST.COM for oryx/127.28.113.62@KRBTEST.COM
I20251215 21:02:55.213404 4928 rpc_server.cc:307] RPC server started. Bound to: 127.28.113.1:45679
I20251215 21:02:55.213712 5058 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.28.113.1:45679 every 8 connection(s)
I20251215 21:02:55.214550 4928 server_base.cc:1179] Dumped server information to /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765832506145157-29124-0/minicluster-data/ts-0/data/info.pb
I20251215 21:02:55.216976 29124 external_mini_cluster.cc:1428] Started /tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu as pid 4928
I20251215 21:02:55.217082 29124 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765832506145157-29124-0/minicluster-data/ts-0/wal/instance
I20251215 21:02:55.217315 4903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:02:55.204705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.113.1:43715 (local address 127.28.113.62:38827)
1215 21:02:55.204854 (+ 149us) server_negotiation.cc:207] Beginning negotiation
1215 21:02:55.204858 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1215 21:02:55.205502 (+ 644us) server_negotiation.cc:408] Connection header received
1215 21:02:55.206382 (+ 880us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1215 21:02:55.206386 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1215 21:02:55.206440 (+ 54us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1215 21:02:55.206552 (+ 112us) server_negotiation.cc:227] Negotiated authn=SASL
1215 21:02:55.208088 (+ 1536us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:02:55.208865 (+ 777us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1215 21:02:55.209738 (+ 873us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:02:55.209927 (+ 189us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:02:55.213025 (+ 3098us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1215 21:02:55.213050 (+ 25us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1215 21:02:55.213053 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1215 21:02:55.213091 (+ 38us) server_negotiation.cc:923] calling sasl_server_start()
1215 21:02:55.214787 (+ 1696us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1215 21:02:55.215612 (+ 825us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1215 21:02:55.215616 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1215 21:02:55.215618 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1215 21:02:55.215671 (+ 53us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1215 21:02:55.215999 (+ 328us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1215 21:02:55.216003 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1215 21:02:55.216005 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1215 21:02:55.216230 (+ 225us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1215 21:02:55.216330 (+ 100us) server_negotiation.cc:1036] Waiting for connection context
1215 21:02:55.216973 (+ 643us) server_negotiation.cc:300] Negotiation successful
1215 21:02:55.217194 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20251215 21:02:55.217332 29124 external_mini_cluster.cc:1468] Setting key 7d463f7e82c444a98a4230c0bc9f9aeb
I20251215 21:02:55.217865 4952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:02:55.204925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.113.62:38827 (local address 127.28.113.1:43715)
1215 21:02:55.205359 (+ 434us) negotiation.cc:107] Waiting for socket to connect
1215 21:02:55.205397 (+ 38us) client_negotiation.cc:174] Beginning negotiation
1215 21:02:55.206163 (+ 766us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1215 21:02:55.206697 (+ 534us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1215 21:02:55.206707 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
1215 21:02:55.207228 (+ 521us) client_negotiation.cc:189] Negotiated authn=SASL
1215 21:02:55.207915 (+ 687us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:02:55.207931 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:02:55.209002 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1215 21:02:55.209006 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1215 21:02:55.209617 (+ 611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:02:55.209628 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:02:55.209853 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:02:55.210499 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1215 21:02:55.210535 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
1215 21:02:55.212870 (+ 2335us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1215 21:02:55.214930 (+ 2060us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1215 21:02:55.214935 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1215 21:02:55.214944 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
1215 21:02:55.215516 (+ 572us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1215 21:02:55.215773 (+ 257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1215 21:02:55.215776 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1215 21:02:55.215778 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1215 21:02:55.215891 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1215 21:02:55.216353 (+ 462us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1215 21:02:55.216361 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1215 21:02:55.216622 (+ 261us) client_negotiation.cc:769] Sending connection context
1215 21:02:55.216866 (+ 244us) client_negotiation.cc:240] Negotiation successful
1215 21:02:55.217148 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":124,"threads_started":1}
I20251215 21:02:55.219045 5059 heartbeater.cc:344] Connected to a master server at 127.28.113.62:38827
I20251215 21:02:55.219283 5059 heartbeater.cc:461] Registering TS with master...
I20251215 21:02:55.219861 5059 heartbeater.cc:507] Master 127.28.113.62:38827 requested a full tablet report, sending...
I20251215 21:02:55.221393 4854 ts_manager.cc:194] Registered new tserver with Master: 38e72b9cde82428ea8acd83a5194f75d (127.28.113.1:45679)
I20251215 21:02:55.222877 4854 master_service.cc:502] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.28.113.1@KRBTEST.COM'} at 127.28.113.1:43715
WARNING: no policy specified for oryx/127.28.113.2@KRBTEST.COM; defaulting to no policy
I20251215 21:02:55.228260 5066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:02:55.221301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.113.62:38827 (local address 127.28.113.1:48877)
1215 21:02:55.221730 (+ 429us) negotiation.cc:107] Waiting for socket to connect
1215 21:02:55.221744 (+ 14us) client_negotiation.cc:174] Beginning negotiation
1215 21:02:55.221838 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1215 21:02:55.222099 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1215 21:02:55.222102 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1215 21:02:55.222302 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
1215 21:02:55.222593 (+ 291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:02:55.222600 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:02:55.223599 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1215 21:02:55.223602 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1215 21:02:55.224168 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:02:55.224178 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:02:55.224292 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:02:55.224975 (+ 683us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1215 21:02:55.224990 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
1215 21:02:55.225313 (+ 323us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1215 21:02:55.226983 (+ 1670us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1215 21:02:55.226986 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1215 21:02:55.226989 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
1215 21:02:55.227147 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1215 21:02:55.227388 (+ 241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1215 21:02:55.227391 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1215 21:02:55.227392 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1215 21:02:55.227439 (+ 47us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1215 21:02:55.227750 (+ 311us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1215 21:02:55.227753 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1215 21:02:55.227830 (+ 77us) client_negotiation.cc:769] Sending connection context
1215 21:02:55.227921 (+ 91us) client_negotiation.cc:240] Negotiation successful
1215 21:02:55.228076 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":351,"thread_start_us":83,"threads_started":1}
I20251215 21:02:55.228322 4903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:02:55.221382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.113.1:48877 (local address 127.28.113.62:38827)
1215 21:02:55.221511 (+ 129us) server_negotiation.cc:207] Beginning negotiation
1215 21:02:55.221515 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1215 21:02:55.221824 (+ 309us) server_negotiation.cc:408] Connection header received
1215 21:02:55.221926 (+ 102us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1215 21:02:55.221930 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1215 21:02:55.221982 (+ 52us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1215 21:02:55.222075 (+ 93us) server_negotiation.cc:227] Negotiated authn=SASL
1215 21:02:55.222747 (+ 672us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:02:55.223484 (+ 737us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1215 21:02:55.224286 (+ 802us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:02:55.224472 (+ 186us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:02:55.225447 (+ 975us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1215 21:02:55.225466 (+ 19us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1215 21:02:55.225470 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1215 21:02:55.225498 (+ 28us) server_negotiation.cc:923] calling sasl_server_start()
1215 21:02:55.226885 (+ 1387us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1215 21:02:55.227256 (+ 371us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1215 21:02:55.227259 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1215 21:02:55.227261 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1215 21:02:55.227301 (+ 40us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1215 21:02:55.227525 (+ 224us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1215 21:02:55.227529 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1215 21:02:55.227531 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1215 21:02:55.227659 (+ 128us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1215 21:02:55.227742 (+ 83us) server_negotiation.cc:1036] Waiting for connection context
1215 21:02:55.228012 (+ 270us) server_negotiation.cc:300] Negotiation successful
1215 21:02:55.228197 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.28.113.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.28.113.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765832506145157-29124-0/krb5kdc/oryx.keytab.
Entry for principal oryx/127.28.113.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765832506145157-29124-0/krb5kdc/oryx.keytab.
WARNING: no policy specified for HTTP/127.28.113.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.28.113.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.28.113.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765832506145157-29124-0/krb5kdc/oryx.keytab.
Entry for principal HTTP/127.28.113.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765832506145157-29124-0/krb5kdc/oryx.keytab.
I20251215 21:02:55.259790 29124 external_mini_cluster.cc:1366] Running /tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu
/tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765832506145157-29124-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765832506145157-29124-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765832506145157-29124-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765832506145157-29124-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.113.2:0
--local_ip_for_outbound_sockets=127.28.113.2
--webserver_interface=127.28.113.2
--webserver_port=0
--tserver_master_addrs=127.28.113.62:38827
--keytab_file=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765832506145157-29124-0/krb5kdc/oryx.keytab
--principal=oryx/127.28.113.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.28.113.20:35731
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765832506145157-29124-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765832506145157-29124-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765832506145157-29124-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251215 21:02:55.364140 5070 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251215 21:02:55.364387 5070 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251215 21:02:55.364468 5070 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251215 21:02:55.367935 5070 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251215 21:02:55.368039 5070 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251215 21:02:55.368115 5070 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20251215 21:02:55.368181 5070 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.113.2
I20251215 21:02:55.372450 5070 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.113.20:35731
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765832506145157-29124-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765832506145157-29124-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765832506145157-29124-0/krb5kdc/oryx.keytab
--principal=oryx/127.28.113.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.113.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765832506145157-29124-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.113.2
--webserver_port=0
--webserver_require_spnego=true
--enable_txn_system_client_init=true
--tserver_master_addrs=127.28.113.62:38827
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5070
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.113.2
--log_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765832506145157-29124-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision dc741732bb6a40213e11a13869fad06a49a209a3
build type DEBUG
built by None at 15 Dec 2025 20:43:14 UTC on 24a791456cd2
build id 9530
I20251215 21:02:55.373514 5070 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251215 21:02:55.374351 5070 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251215 21:02:55.380682 5078 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251215 21:02:55.380741 5075 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251215 21:02:55.380679 5076 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251215 21:02:55.380956 5070 server_base.cc:1047] running on GCE node
I20251215 21:02:55.381325 5070 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251215 21:02:55.381837 5070 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251215 21:02:55.382995 5070 hybrid_clock.cc:648] HybridClock initialized: now 1765832575382975 us; error 38 us; skew 500 ppm
Dec 15 21:02:55 dist-test-slave-bjsf krb5kdc[4806](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765832575, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.113.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251215 21:02:55.385531 5070 init.cc:377] Logged in from keytab as oryx/127.28.113.2@KRBTEST.COM (short username oryx)
I20251215 21:02:55.386538 5070 webserver.cc:492] Webserver started at http://127.28.113.2:36841/ using document root <none> and password file <none>
I20251215 21:02:55.387101 5070 fs_manager.cc:362] Metadata directory not provided
I20251215 21:02:55.387152 5070 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251215 21:02:55.387324 5070 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251215 21:02:55.389055 5070 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765832506145157-29124-0/minicluster-data/ts-1/data/instance:
uuid: "f11866f255744d2fb31b4a6c2bcf6109"
format_stamp: "Formatted at 2025-12-15 21:02:55 on dist-test-slave-bjsf"
server_key: "365dd2243c509cd880afbb278ee7b4e8"
server_key_iv: "3949624ee2b9a50817375ba1b25f1a1a"
server_key_version: "encryptionkey@0"
I20251215 21:02:55.389494 5070 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765832506145157-29124-0/minicluster-data/ts-1/wal/instance:
uuid: "f11866f255744d2fb31b4a6c2bcf6109"
format_stamp: "Formatted at 2025-12-15 21:02:55 on dist-test-slave-bjsf"
server_key: "365dd2243c509cd880afbb278ee7b4e8"
server_key_iv: "3949624ee2b9a50817375ba1b25f1a1a"
server_key_version: "encryptionkey@0"
I20251215 21:02:55.392648 5070 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20251215 21:02:55.394663 5085 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251215 21:02:55.395596 5070 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20251215 21:02:55.395692 5070 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765832506145157-29124-0/minicluster-data/ts-1/data,/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765832506145157-29124-0/minicluster-data/ts-1/wal
uuid: "f11866f255744d2fb31b4a6c2bcf6109"
format_stamp: "Formatted at 2025-12-15 21:02:55 on dist-test-slave-bjsf"
server_key: "365dd2243c509cd880afbb278ee7b4e8"
server_key_iv: "3949624ee2b9a50817375ba1b25f1a1a"
server_key_version: "encryptionkey@0"
I20251215 21:02:55.395772 5070 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765832506145157-29124-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765832506145157-29124-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765832506145157-29124-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251215 21:02:55.404762 5070 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251215 21:02:55.407073 5070 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251215 21:02:55.407230 5070 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251215 21:02:55.408618 5070 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251215 21:02:55.408667 5070 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251215 21:02:55.408715 5070 ts_tablet_manager.cc:616] Registered 0 tablets
I20251215 21:02:55.408735 5070 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Dec 15 21:02:55 dist-test-slave-bjsf krb5kdc[4806](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765832575, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.113.2@KRBTEST.COM for oryx/127.28.113.62@KRBTEST.COM
I20251215 21:02:55.418339 5070 rpc_server.cc:307] RPC server started. Bound to: 127.28.113.2:44919
I20251215 21:02:55.418962 5200 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.28.113.2:44919 every 8 connection(s)
I20251215 21:02:55.419476 5070 server_base.cc:1179] Dumped server information to /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765832506145157-29124-0/minicluster-data/ts-1/data/info.pb
I20251215 21:02:55.420799 4903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:02:55.409151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.113.2:32847 (local address 127.28.113.62:38827)
1215 21:02:55.409270 (+ 119us) server_negotiation.cc:207] Beginning negotiation
1215 21:02:55.409274 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1215 21:02:55.410025 (+ 751us) server_negotiation.cc:408] Connection header received
1215 21:02:55.410873 (+ 848us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1215 21:02:55.410876 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1215 21:02:55.410936 (+ 60us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1215 21:02:55.411007 (+ 71us) server_negotiation.cc:227] Negotiated authn=SASL
1215 21:02:55.412637 (+ 1630us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:02:55.413399 (+ 762us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1215 21:02:55.414330 (+ 931us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:02:55.414536 (+ 206us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:02:55.417055 (+ 2519us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1215 21:02:55.417083 (+ 28us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1215 21:02:55.417087 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1215 21:02:55.417122 (+ 35us) server_negotiation.cc:923] calling sasl_server_start()
1215 21:02:55.418740 (+ 1618us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1215 21:02:55.419316 (+ 576us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1215 21:02:55.419321 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1215 21:02:55.419323 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1215 21:02:55.419377 (+ 54us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1215 21:02:55.419619 (+ 242us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1215 21:02:55.419622 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1215 21:02:55.419624 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1215 21:02:55.419825 (+ 201us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1215 21:02:55.419930 (+ 105us) server_negotiation.cc:1036] Waiting for connection context
1215 21:02:55.420512 (+ 582us) server_negotiation.cc:300] Negotiation successful
1215 21:02:55.420642 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20251215 21:02:55.421352 5094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:02:55.409461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.113.62:38827 (local address 127.28.113.2:32847)
1215 21:02:55.409890 (+ 429us) negotiation.cc:107] Waiting for socket to connect
1215 21:02:55.409924 (+ 34us) client_negotiation.cc:174] Beginning negotiation
1215 21:02:55.410656 (+ 732us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1215 21:02:55.411179 (+ 523us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1215 21:02:55.411186 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
1215 21:02:55.411702 (+ 516us) client_negotiation.cc:189] Negotiated authn=SASL
1215 21:02:55.412453 (+ 751us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:02:55.412467 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:02:55.413510 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1215 21:02:55.413514 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1215 21:02:55.414167 (+ 653us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:02:55.414179 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:02:55.414406 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:02:55.415062 (+ 656us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1215 21:02:55.415098 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
1215 21:02:55.416870 (+ 1772us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1215 21:02:55.418874 (+ 2004us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1215 21:02:55.418880 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1215 21:02:55.418891 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
1215 21:02:55.419204 (+ 313us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1215 21:02:55.419467 (+ 263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1215 21:02:55.419469 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1215 21:02:55.419471 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1215 21:02:55.419545 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1215 21:02:55.419935 (+ 390us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1215 21:02:55.419941 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1215 21:02:55.420226 (+ 285us) client_negotiation.cc:769] Sending connection context
1215 21:02:55.420445 (+ 219us) client_negotiation.cc:240] Negotiation successful
1215 21:02:55.420716 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":119,"threads_started":1}
I20251215 21:02:55.422415 5201 heartbeater.cc:344] Connected to a master server at 127.28.113.62:38827
I20251215 21:02:55.422569 5201 heartbeater.cc:461] Registering TS with master...
I20251215 21:02:55.423174 5201 heartbeater.cc:507] Master 127.28.113.62:38827 requested a full tablet report, sending...
I20251215 21:02:55.424257 4853 ts_manager.cc:194] Registered new tserver with Master: f11866f255744d2fb31b4a6c2bcf6109 (127.28.113.2:44919)
I20251215 21:02:55.424470 29124 external_mini_cluster.cc:1428] Started /tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu as pid 5070
I20251215 21:02:55.424585 29124 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765832506145157-29124-0/minicluster-data/ts-1/wal/instance
I20251215 21:02:55.424850 29124 external_mini_cluster.cc:1468] Setting key 1c77f80e167ab6f2aa85910da4cd9ec2
I20251215 21:02:55.425009 4853 master_service.cc:502] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.28.113.2@KRBTEST.COM'} at 127.28.113.2:32847
I20251215 21:02:55.433185 5207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:02:55.425399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.113.62:38827 (local address 127.28.113.2:33955)
1215 21:02:55.427143 (+ 1744us) negotiation.cc:107] Waiting for socket to connect
1215 21:02:55.427156 (+ 13us) client_negotiation.cc:174] Beginning negotiation
1215 21:02:55.427247 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1215 21:02:55.427469 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1215 21:02:55.427471 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1215 21:02:55.427663 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
1215 21:02:55.427936 (+ 273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:02:55.427944 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:02:55.428746 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1215 21:02:55.428748 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1215 21:02:55.429124 (+ 376us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:02:55.429131 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:02:55.429217 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:02:55.429861 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1215 21:02:55.429875 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
1215 21:02:55.430159 (+ 284us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1215 21:02:55.431921 (+ 1762us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1215 21:02:55.431924 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1215 21:02:55.431926 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1215 21:02:55.432126 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1215 21:02:55.432359 (+ 233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1215 21:02:55.432361 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1215 21:02:55.432362 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1215 21:02:55.432400 (+ 38us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1215 21:02:55.432778 (+ 378us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1215 21:02:55.432780 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1215 21:02:55.432855 (+ 75us) client_negotiation.cc:769] Sending connection context
1215 21:02:55.432931 (+ 76us) client_negotiation.cc:240] Negotiation successful
1215 21:02:55.433030 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1639,"thread_start_us":110,"threads_started":1}
I20251215 21:02:55.433291 4903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:02:55.425715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.113.2:33955 (local address 127.28.113.62:38827)
1215 21:02:55.425831 (+ 116us) server_negotiation.cc:207] Beginning negotiation
1215 21:02:55.425834 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1215 21:02:55.427238 (+ 1404us) server_negotiation.cc:408] Connection header received
1215 21:02:55.427323 (+ 85us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1215 21:02:55.427326 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1215 21:02:55.427374 (+ 48us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1215 21:02:55.427449 (+ 75us) server_negotiation.cc:227] Negotiated authn=SASL
1215 21:02:55.428068 (+ 619us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:02:55.428642 (+ 574us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1215 21:02:55.429227 (+ 585us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:02:55.429404 (+ 177us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:02:55.430289 (+ 885us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1215 21:02:55.430309 (+ 20us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1215 21:02:55.430312 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1215 21:02:55.430340 (+ 28us) server_negotiation.cc:923] calling sasl_server_start()
1215 21:02:55.431804 (+ 1464us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1215 21:02:55.432217 (+ 413us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1215 21:02:55.432221 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1215 21:02:55.432224 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
1215 21:02:55.432275 (+ 51us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1215 21:02:55.432474 (+ 199us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1215 21:02:55.432478 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1215 21:02:55.432480 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1215 21:02:55.432696 (+ 216us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1215 21:02:55.432800 (+ 104us) server_negotiation.cc:1036] Waiting for connection context
1215 21:02:55.433012 (+ 212us) server_negotiation.cc:300] Negotiation successful
1215 21:02:55.433139 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
WARNING: no policy specified for oryx/127.28.113.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.28.113.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.28.113.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765832506145157-29124-0/krb5kdc/oryx.keytab.
Entry for principal oryx/127.28.113.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765832506145157-29124-0/krb5kdc/oryx.keytab.
WARNING: no policy specified for HTTP/127.28.113.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.28.113.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.28.113.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765832506145157-29124-0/krb5kdc/oryx.keytab.
Entry for principal HTTP/127.28.113.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765832506145157-29124-0/krb5kdc/oryx.keytab.
I20251215 21:02:55.467834 29124 external_mini_cluster.cc:1366] Running /tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu
/tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765832506145157-29124-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765832506145157-29124-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765832506145157-29124-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765832506145157-29124-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.113.3:0
--local_ip_for_outbound_sockets=127.28.113.3
--webserver_interface=127.28.113.3
--webserver_port=0
--tserver_master_addrs=127.28.113.62:38827
--keytab_file=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765832506145157-29124-0/krb5kdc/oryx.keytab
--principal=oryx/127.28.113.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.28.113.20:35731
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765832506145157-29124-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765832506145157-29124-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765832506145157-29124-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251215 21:02:55.568452 5212 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251215 21:02:55.568655 5212 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251215 21:02:55.568722 5212 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251215 21:02:55.571877 5212 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251215 21:02:55.571933 5212 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251215 21:02:55.572017 5212 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20251215 21:02:55.572078 5212 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.113.3
I20251215 21:02:55.576097 5212 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.113.20:35731
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765832506145157-29124-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765832506145157-29124-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765832506145157-29124-0/krb5kdc/oryx.keytab
--principal=oryx/127.28.113.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.113.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765832506145157-29124-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.113.3
--webserver_port=0
--webserver_require_spnego=true
--enable_txn_system_client_init=true
--tserver_master_addrs=127.28.113.62:38827
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5212
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.113.3
--log_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765832506145157-29124-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision dc741732bb6a40213e11a13869fad06a49a209a3
build type DEBUG
built by None at 15 Dec 2025 20:43:14 UTC on 24a791456cd2
build id 9530
I20251215 21:02:55.577070 5212 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251215 21:02:55.577884 5212 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251215 21:02:55.583922 5220 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251215 21:02:55.583931 5217 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251215 21:02:55.584190 5212 server_base.cc:1047] running on GCE node
W20251215 21:02:55.584203 5218 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251215 21:02:55.584618 5212 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251215 21:02:55.585126 5212 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251215 21:02:55.586277 5212 hybrid_clock.cc:648] HybridClock initialized: now 1765832575586266 us; error 27 us; skew 500 ppm
Dec 15 21:02:55 dist-test-slave-bjsf krb5kdc[4806](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765832575, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.113.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251215 21:02:55.588742 5212 init.cc:377] Logged in from keytab as oryx/127.28.113.3@KRBTEST.COM (short username oryx)
I20251215 21:02:55.589710 5212 webserver.cc:492] Webserver started at http://127.28.113.3:40257/ using document root <none> and password file <none>
I20251215 21:02:55.590220 5212 fs_manager.cc:362] Metadata directory not provided
I20251215 21:02:55.590267 5212 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251215 21:02:55.590431 5212 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251215 21:02:55.592063 5212 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765832506145157-29124-0/minicluster-data/ts-2/data/instance:
uuid: "fe1797652ab94dff90667ef356a6c2ae"
format_stamp: "Formatted at 2025-12-15 21:02:55 on dist-test-slave-bjsf"
server_key: "a2238b21402c1911d5f55081c6dd1376"
server_key_iv: "cd0ccde4344d197fd010002bac0de75f"
server_key_version: "encryptionkey@0"
I20251215 21:02:55.592494 5212 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765832506145157-29124-0/minicluster-data/ts-2/wal/instance:
uuid: "fe1797652ab94dff90667ef356a6c2ae"
format_stamp: "Formatted at 2025-12-15 21:02:55 on dist-test-slave-bjsf"
server_key: "a2238b21402c1911d5f55081c6dd1376"
server_key_iv: "cd0ccde4344d197fd010002bac0de75f"
server_key_version: "encryptionkey@0"
I20251215 21:02:55.595534 5212 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20251215 21:02:55.597589 5227 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251215 21:02:55.598508 5212 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20251215 21:02:55.598603 5212 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765832506145157-29124-0/minicluster-data/ts-2/data,/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765832506145157-29124-0/minicluster-data/ts-2/wal
uuid: "fe1797652ab94dff90667ef356a6c2ae"
format_stamp: "Formatted at 2025-12-15 21:02:55 on dist-test-slave-bjsf"
server_key: "a2238b21402c1911d5f55081c6dd1376"
server_key_iv: "cd0ccde4344d197fd010002bac0de75f"
server_key_version: "encryptionkey@0"
I20251215 21:02:55.598681 5212 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765832506145157-29124-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765832506145157-29124-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765832506145157-29124-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251215 21:02:55.645248 5212 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251215 21:02:55.650310 5212 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251215 21:02:55.650487 5212 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251215 21:02:55.651855 5212 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251215 21:02:55.651912 5212 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251215 21:02:55.651984 5212 ts_tablet_manager.cc:616] Registered 0 tablets
I20251215 21:02:55.652016 5212 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Dec 15 21:02:55 dist-test-slave-bjsf krb5kdc[4806](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765832575, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.113.3@KRBTEST.COM for oryx/127.28.113.62@KRBTEST.COM
I20251215 21:02:55.662921 5212 rpc_server.cc:307] RPC server started. Bound to: 127.28.113.3:46345
I20251215 21:02:55.663273 4903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:02:55.652586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.113.3:45069 (local address 127.28.113.62:38827)
1215 21:02:55.652707 (+ 121us) server_negotiation.cc:207] Beginning negotiation
1215 21:02:55.652711 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1215 21:02:55.653479 (+ 768us) server_negotiation.cc:408] Connection header received
1215 21:02:55.654302 (+ 823us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1215 21:02:55.654306 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1215 21:02:55.654365 (+ 59us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1215 21:02:55.654439 (+ 74us) server_negotiation.cc:227] Negotiated authn=SASL
1215 21:02:55.655928 (+ 1489us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:02:55.656601 (+ 673us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1215 21:02:55.657266 (+ 665us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:02:55.657475 (+ 209us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:02:55.659902 (+ 2427us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1215 21:02:55.659930 (+ 28us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1215 21:02:55.659934 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1215 21:02:55.659994 (+ 60us) server_negotiation.cc:923] calling sasl_server_start()
1215 21:02:55.661430 (+ 1436us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1215 21:02:55.661870 (+ 440us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1215 21:02:55.661873 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1215 21:02:55.661874 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1215 21:02:55.661916 (+ 42us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1215 21:02:55.662170 (+ 254us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1215 21:02:55.662173 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1215 21:02:55.662175 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1215 21:02:55.662322 (+ 147us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1215 21:02:55.662404 (+ 82us) server_negotiation.cc:1036] Waiting for connection context
1215 21:02:55.662978 (+ 574us) server_negotiation.cc:300] Negotiation successful
1215 21:02:55.663103 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20251215 21:02:55.663793 5236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:02:55.652907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.113.62:38827 (local address 127.28.113.3:45069)
1215 21:02:55.653343 (+ 436us) negotiation.cc:107] Waiting for socket to connect
1215 21:02:55.653376 (+ 33us) client_negotiation.cc:174] Beginning negotiation
1215 21:02:55.654104 (+ 728us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1215 21:02:55.654603 (+ 499us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1215 21:02:55.654611 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
1215 21:02:55.655120 (+ 509us) client_negotiation.cc:189] Negotiated authn=SASL
1215 21:02:55.655788 (+ 668us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:02:55.655806 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:02:55.656719 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1215 21:02:55.656722 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1215 21:02:55.657149 (+ 427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:02:55.657157 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:02:55.657361 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:02:55.657966 (+ 605us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1215 21:02:55.657988 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
1215 21:02:55.659738 (+ 1750us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1215 21:02:55.661544 (+ 1806us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1215 21:02:55.661550 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1215 21:02:55.661560 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
1215 21:02:55.661763 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1215 21:02:55.661998 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1215 21:02:55.662001 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1215 21:02:55.662003 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1215 21:02:55.662081 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1215 21:02:55.662425 (+ 344us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1215 21:02:55.662432 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1215 21:02:55.662672 (+ 240us) client_negotiation.cc:769] Sending connection context
1215 21:02:55.662905 (+ 233us) client_negotiation.cc:240] Negotiation successful
1215 21:02:55.663146 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":139,"threads_started":1}
I20251215 21:02:55.663985 5212 server_base.cc:1179] Dumped server information to /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765832506145157-29124-0/minicluster-data/ts-2/data/info.pb
I20251215 21:02:55.664749 5342 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.28.113.3:46345 every 8 connection(s)
I20251215 21:02:55.665524 5343 heartbeater.cc:344] Connected to a master server at 127.28.113.62:38827
I20251215 21:02:55.665673 5343 heartbeater.cc:461] Registering TS with master...
I20251215 21:02:55.666152 5343 heartbeater.cc:507] Master 127.28.113.62:38827 requested a full tablet report, sending...
I20251215 21:02:55.667106 4853 ts_manager.cc:194] Registered new tserver with Master: fe1797652ab94dff90667ef356a6c2ae (127.28.113.3:46345)
I20251215 21:02:55.667714 4853 master_service.cc:502] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.28.113.3@KRBTEST.COM'} at 127.28.113.3:45069
I20251215 21:02:55.672484 5349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:02:55.666883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.113.62:38827 (local address 127.28.113.3:33989)
1215 21:02:55.667131 (+ 248us) negotiation.cc:107] Waiting for socket to connect
1215 21:02:55.667145 (+ 14us) client_negotiation.cc:174] Beginning negotiation
1215 21:02:55.667250 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1215 21:02:55.667483 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1215 21:02:55.667485 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1215 21:02:55.667642 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
1215 21:02:55.667879 (+ 237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:02:55.667887 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:02:55.668664 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1215 21:02:55.668668 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1215 21:02:55.669045 (+ 377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:02:55.669052 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:02:55.669150 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:02:55.669665 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1215 21:02:55.669677 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
1215 21:02:55.669936 (+ 259us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1215 21:02:55.671323 (+ 1387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1215 21:02:55.671327 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1215 21:02:55.671329 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1215 21:02:55.671475 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1215 21:02:55.671687 (+ 212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1215 21:02:55.671690 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1215 21:02:55.671692 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1215 21:02:55.671731 (+ 39us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1215 21:02:55.672043 (+ 312us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1215 21:02:55.672046 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1215 21:02:55.672129 (+ 83us) client_negotiation.cc:769] Sending connection context
1215 21:02:55.672212 (+ 83us) client_negotiation.cc:240] Negotiation successful
1215 21:02:55.672323 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":86,"threads_started":1}
I20251215 21:02:55.672485 4903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:02:55.666967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.113.3:33989 (local address 127.28.113.62:38827)
1215 21:02:55.667090 (+ 123us) server_negotiation.cc:207] Beginning negotiation
1215 21:02:55.667094 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1215 21:02:55.667229 (+ 135us) server_negotiation.cc:408] Connection header received
1215 21:02:55.667332 (+ 103us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1215 21:02:55.667335 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1215 21:02:55.667377 (+ 42us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1215 21:02:55.667453 (+ 76us) server_negotiation.cc:227] Negotiated authn=SASL
1215 21:02:55.668021 (+ 568us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:02:55.668555 (+ 534us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1215 21:02:55.669159 (+ 604us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:02:55.669340 (+ 181us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:02:55.670061 (+ 721us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1215 21:02:55.670074 (+ 13us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1215 21:02:55.670076 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1215 21:02:55.670096 (+ 20us) server_negotiation.cc:923] calling sasl_server_start()
1215 21:02:55.671219 (+ 1123us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1215 21:02:55.671565 (+ 346us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1215 21:02:55.671568 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1215 21:02:55.671570 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1215 21:02:55.671607 (+ 37us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1215 21:02:55.671809 (+ 202us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1215 21:02:55.671812 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1215 21:02:55.671814 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1215 21:02:55.671934 (+ 120us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1215 21:02:55.672046 (+ 112us) server_negotiation.cc:1036] Waiting for connection context
1215 21:02:55.672282 (+ 236us) server_negotiation.cc:300] Negotiation successful
1215 21:02:55.672369 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20251215 21:02:55.673394 29124 external_mini_cluster.cc:1428] Started /tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu as pid 5212
I20251215 21:02:55.673514 29124 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765832506145157-29124-0/minicluster-data/ts-2/wal/instance
I20251215 21:02:55.673759 29124 external_mini_cluster.cc:1468] Setting key 8809a10b6a06333bffdf7aabecf7395c
I20251215 21:02:55.675664 29124 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
Dec 15 21:02:55 dist-test-slave-bjsf krb5kdc[4806](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765832575, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Dec 15 21:02:55 dist-test-slave-bjsf krb5kdc[4806](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.28.113.62@KRBTEST.COM, Server not found in Kerberos database
Dec 15 21:02:55 dist-test-slave-bjsf krb5kdc[4806](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.28.113.62@KRBTEST.COM, Server not found in Kerberos database
W20251215 21:02:55.695513 4903 negotiation.cc:337] Failed RPC negotiation. Trace:
1215 21:02:55.690566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34148 (local address 127.28.113.62:38827)
1215 21:02:55.690684 (+ 118us) server_negotiation.cc:207] Beginning negotiation
1215 21:02:55.690688 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1215 21:02:55.690701 (+ 13us) server_negotiation.cc:408] Connection header received
1215 21:02:55.690739 (+ 38us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1215 21:02:55.690742 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1215 21:02:55.690781 (+ 39us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1215 21:02:55.690845 (+ 64us) server_negotiation.cc:227] Negotiated authn=SASL
1215 21:02:55.691548 (+ 703us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:02:55.692070 (+ 522us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1215 21:02:55.692685 (+ 615us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:02:55.692807 (+ 122us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:02:55.695394 (+ 2587us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34148: BlockingRecv error: recv got EOF from 127.0.0.1:34148 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Dec 15 21:02:55 dist-test-slave-bjsf krb5kdc[4806](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765832575, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.28.113.62@KRBTEST.COM
I20251215 21:02:55.703107 4903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:02:55.696740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34158 (local address 127.28.113.62:38827)
1215 21:02:55.696873 (+ 133us) server_negotiation.cc:207] Beginning negotiation
1215 21:02:55.696877 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1215 21:02:55.696946 (+ 69us) server_negotiation.cc:408] Connection header received
1215 21:02:55.697054 (+ 108us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1215 21:02:55.697057 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1215 21:02:55.697097 (+ 40us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1215 21:02:55.697161 (+ 64us) server_negotiation.cc:227] Negotiated authn=SASL
1215 21:02:55.697799 (+ 638us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:02:55.698245 (+ 446us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1215 21:02:55.698830 (+ 585us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:02:55.699001 (+ 171us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:02:55.700756 (+ 1755us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1215 21:02:55.700771 (+ 15us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1215 21:02:55.700773 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1215 21:02:55.700811 (+ 38us) server_negotiation.cc:923] calling sasl_server_start()
1215 21:02:55.701889 (+ 1078us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1215 21:02:55.702253 (+ 364us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1215 21:02:55.702256 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1215 21:02:55.702257 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1215 21:02:55.702311 (+ 54us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1215 21:02:55.702490 (+ 179us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1215 21:02:55.702492 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1215 21:02:55.702493 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1215 21:02:55.702612 (+ 119us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1215 21:02:55.702722 (+ 110us) server_negotiation.cc:1036] Waiting for connection context
1215 21:02:55.702902 (+ 180us) server_negotiation.cc:300] Negotiation successful
1215 21:02:55.703003 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20251215 21:02:55.705317 4853 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34158:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251215 21:02:55.707525 4853 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251215 21:02:55.719606 5365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:02:55.714841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.113.3:46345 (local address 127.0.0.1:50932)
1215 21:02:55.715158 (+ 317us) negotiation.cc:107] Waiting for socket to connect
1215 21:02:55.715197 (+ 39us) client_negotiation.cc:174] Beginning negotiation
1215 21:02:55.715331 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1215 21:02:55.716458 (+ 1127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1215 21:02:55.716464 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
1215 21:02:55.716486 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1215 21:02:55.716767 (+ 281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:02:55.716775 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:02:55.718143 (+ 1368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1215 21:02:55.718146 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1215 21:02:55.719047 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:02:55.719056 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:02:55.719164 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:02:55.719213 (+ 49us) client_negotiation.cc:769] Sending connection context
1215 21:02:55.719298 (+ 85us) client_negotiation.cc:240] Negotiation successful
1215 21:02:55.719382 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"spinlock_wait_cycles":1280,"thread_start_us":72,"threads_started":1}
I20251215 21:02:55.720009 5368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:02:55.715279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.113.1:45679 (local address 127.0.0.1:41348)
1215 21:02:55.715752 (+ 473us) negotiation.cc:107] Waiting for socket to connect
1215 21:02:55.715765 (+ 13us) client_negotiation.cc:174] Beginning negotiation
1215 21:02:55.715845 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1215 21:02:55.717109 (+ 1264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1215 21:02:55.717112 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1215 21:02:55.717128 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1215 21:02:55.717315 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:02:55.717321 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:02:55.718267 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1215 21:02:55.718271 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1215 21:02:55.719103 (+ 832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:02:55.719112 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:02:55.719216 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:02:55.719234 (+ 18us) client_negotiation.cc:769] Sending connection context
1215 21:02:55.719296 (+ 62us) client_negotiation.cc:240] Negotiation successful
1215 21:02:55.719383 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":406,"thread_start_us":81,"threads_started":1}
I20251215 21:02:55.720038 5367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:02:55.715447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.113.2:44919 (local address 127.0.0.1:35118)
1215 21:02:55.715908 (+ 461us) negotiation.cc:107] Waiting for socket to connect
1215 21:02:55.715920 (+ 12us) client_negotiation.cc:174] Beginning negotiation
1215 21:02:55.716020 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1215 21:02:55.716494 (+ 474us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1215 21:02:55.716497 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1215 21:02:55.716515 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1215 21:02:55.716753 (+ 238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:02:55.716760 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:02:55.718291 (+ 1531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1215 21:02:55.718294 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1215 21:02:55.719110 (+ 816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:02:55.719118 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:02:55.719196 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:02:55.719216 (+ 20us) client_negotiation.cc:769] Sending connection context
1215 21:02:55.719298 (+ 82us) client_negotiation.cc:240] Negotiation successful
1215 21:02:55.719382 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":398,"spinlock_wait_cycles":768,"thread_start_us":66,"threads_started":1}
I20251215 21:02:55.720113 5366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:02:55.714963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50932 (local address 127.28.113.3:46345)
1215 21:02:55.715325 (+ 362us) server_negotiation.cc:207] Beginning negotiation
1215 21:02:55.715331 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1215 21:02:55.715355 (+ 24us) server_negotiation.cc:408] Connection header received
1215 21:02:55.716123 (+ 768us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1215 21:02:55.716130 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
1215 21:02:55.716311 (+ 181us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1215 21:02:55.716443 (+ 132us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1215 21:02:55.716910 (+ 467us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:02:55.718026 (+ 1116us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1215 21:02:55.719193 (+ 1167us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:02:55.719683 (+ 490us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:02:55.719791 (+ 108us) server_negotiation.cc:1036] Waiting for connection context
1215 21:02:55.719847 (+ 56us) server_negotiation.cc:300] Negotiation successful
1215 21:02:55.719923 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":272,"thread_start_us":98,"threads_started":1}
I20251215 21:02:55.720441 5370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:02:55.715736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35118 (local address 127.28.113.2:44919)
1215 21:02:55.716090 (+ 354us) server_negotiation.cc:207] Beginning negotiation
1215 21:02:55.716095 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1215 21:02:55.716112 (+ 17us) server_negotiation.cc:408] Connection header received
1215 21:02:55.716187 (+ 75us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1215 21:02:55.716192 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1215 21:02:55.716365 (+ 173us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1215 21:02:55.716496 (+ 131us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1215 21:02:55.716863 (+ 367us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:02:55.718026 (+ 1163us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1215 21:02:55.719471 (+ 1445us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:02:55.720102 (+ 631us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:02:55.720200 (+ 98us) server_negotiation.cc:1036] Waiting for connection context
1215 21:02:55.720258 (+ 58us) server_negotiation.cc:300] Negotiation successful
1215 21:02:55.720333 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":275,"thread_start_us":91,"threads_started":1}
I20251215 21:02:55.721164 5369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:02:55.715601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41348 (local address 127.28.113.1:45679)
1215 21:02:55.716724 (+ 1123us) server_negotiation.cc:207] Beginning negotiation
1215 21:02:55.716730 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1215 21:02:55.716749 (+ 19us) server_negotiation.cc:408] Connection header received
1215 21:02:55.716837 (+ 88us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1215 21:02:55.716842 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1215 21:02:55.716985 (+ 143us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1215 21:02:55.717102 (+ 117us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1215 21:02:55.717423 (+ 321us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:02:55.718134 (+ 711us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1215 21:02:55.719525 (+ 1391us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:02:55.720880 (+ 1355us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:02:55.720958 (+ 78us) server_negotiation.cc:1036] Waiting for connection context
1215 21:02:55.721005 (+ 47us) server_negotiation.cc:300] Negotiation successful
1215 21:02:55.721062 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1029,"thread_start_us":74,"threads_started":1}
I20251215 21:02:55.723239 4993 tablet_service.cc:1505] Processing CreateTablet for tablet 23b0bbea258a4b91933d8a01a21598c7 (DEFAULT_TABLE table=test-table [id=a158679eea3d4885b8fe75ab75765981]), partition=RANGE (key) PARTITION UNBOUNDED
I20251215 21:02:55.723196 5135 tablet_service.cc:1505] Processing CreateTablet for tablet 23b0bbea258a4b91933d8a01a21598c7 (DEFAULT_TABLE table=test-table [id=a158679eea3d4885b8fe75ab75765981]), partition=RANGE (key) PARTITION UNBOUNDED
I20251215 21:02:55.723181 5277 tablet_service.cc:1505] Processing CreateTablet for tablet 23b0bbea258a4b91933d8a01a21598c7 (DEFAULT_TABLE table=test-table [id=a158679eea3d4885b8fe75ab75765981]), partition=RANGE (key) PARTITION UNBOUNDED
I20251215 21:02:55.724104 4993 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 23b0bbea258a4b91933d8a01a21598c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 21:02:55.724110 5277 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 23b0bbea258a4b91933d8a01a21598c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 21:02:55.724110 5135 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 23b0bbea258a4b91933d8a01a21598c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 21:02:55.729329 5371 tablet_bootstrap.cc:492] T 23b0bbea258a4b91933d8a01a21598c7 P 38e72b9cde82428ea8acd83a5194f75d: Bootstrap starting.
I20251215 21:02:55.730417 5372 tablet_bootstrap.cc:492] T 23b0bbea258a4b91933d8a01a21598c7 P fe1797652ab94dff90667ef356a6c2ae: Bootstrap starting.
I20251215 21:02:55.730564 5373 tablet_bootstrap.cc:492] T 23b0bbea258a4b91933d8a01a21598c7 P f11866f255744d2fb31b4a6c2bcf6109: Bootstrap starting.
I20251215 21:02:55.731756 5371 tablet_bootstrap.cc:654] T 23b0bbea258a4b91933d8a01a21598c7 P 38e72b9cde82428ea8acd83a5194f75d: Neither blocks nor log segments found. Creating new log.
I20251215 21:02:55.732420 5372 tablet_bootstrap.cc:654] T 23b0bbea258a4b91933d8a01a21598c7 P fe1797652ab94dff90667ef356a6c2ae: Neither blocks nor log segments found. Creating new log.
I20251215 21:02:55.732420 5373 tablet_bootstrap.cc:654] T 23b0bbea258a4b91933d8a01a21598c7 P f11866f255744d2fb31b4a6c2bcf6109: Neither blocks nor log segments found. Creating new log.
I20251215 21:02:55.732445 5371 log.cc:826] T 23b0bbea258a4b91933d8a01a21598c7 P 38e72b9cde82428ea8acd83a5194f75d: Log is configured to *not* fsync() on all Append() calls
I20251215 21:02:55.733596 5372 log.cc:826] T 23b0bbea258a4b91933d8a01a21598c7 P fe1797652ab94dff90667ef356a6c2ae: Log is configured to *not* fsync() on all Append() calls
I20251215 21:02:55.733596 5373 log.cc:826] T 23b0bbea258a4b91933d8a01a21598c7 P f11866f255744d2fb31b4a6c2bcf6109: Log is configured to *not* fsync() on all Append() calls
I20251215 21:02:55.733945 5371 tablet_bootstrap.cc:492] T 23b0bbea258a4b91933d8a01a21598c7 P 38e72b9cde82428ea8acd83a5194f75d: No bootstrap required, opened a new log
I20251215 21:02:55.734144 5371 ts_tablet_manager.cc:1403] T 23b0bbea258a4b91933d8a01a21598c7 P 38e72b9cde82428ea8acd83a5194f75d: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20251215 21:02:55.735157 5372 tablet_bootstrap.cc:492] T 23b0bbea258a4b91933d8a01a21598c7 P fe1797652ab94dff90667ef356a6c2ae: No bootstrap required, opened a new log
I20251215 21:02:55.735327 5372 ts_tablet_manager.cc:1403] T 23b0bbea258a4b91933d8a01a21598c7 P fe1797652ab94dff90667ef356a6c2ae: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20251215 21:02:55.735419 5373 tablet_bootstrap.cc:492] T 23b0bbea258a4b91933d8a01a21598c7 P f11866f255744d2fb31b4a6c2bcf6109: No bootstrap required, opened a new log
I20251215 21:02:55.735743 5373 ts_tablet_manager.cc:1403] T 23b0bbea258a4b91933d8a01a21598c7 P f11866f255744d2fb31b4a6c2bcf6109: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20251215 21:02:55.737382 5371 raft_consensus.cc:359] T 23b0bbea258a4b91933d8a01a21598c7 P 38e72b9cde82428ea8acd83a5194f75d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe1797652ab94dff90667ef356a6c2ae" member_type: VOTER last_known_addr { host: "127.28.113.3" port: 46345 } } peers { permanent_uuid: "f11866f255744d2fb31b4a6c2bcf6109" member_type: VOTER last_known_addr { host: "127.28.113.2" port: 44919 } } peers { permanent_uuid: "38e72b9cde82428ea8acd83a5194f75d" member_type: VOTER last_known_addr { host: "127.28.113.1" port: 45679 } }
I20251215 21:02:55.737597 5371 raft_consensus.cc:385] T 23b0bbea258a4b91933d8a01a21598c7 P 38e72b9cde82428ea8acd83a5194f75d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251215 21:02:55.737648 5371 raft_consensus.cc:740] T 23b0bbea258a4b91933d8a01a21598c7 P 38e72b9cde82428ea8acd83a5194f75d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 38e72b9cde82428ea8acd83a5194f75d, State: Initialized, Role: FOLLOWER
I20251215 21:02:55.737802 5372 raft_consensus.cc:359] T 23b0bbea258a4b91933d8a01a21598c7 P fe1797652ab94dff90667ef356a6c2ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe1797652ab94dff90667ef356a6c2ae" member_type: VOTER last_known_addr { host: "127.28.113.3" port: 46345 } } peers { permanent_uuid: "f11866f255744d2fb31b4a6c2bcf6109" member_type: VOTER last_known_addr { host: "127.28.113.2" port: 44919 } } peers { permanent_uuid: "38e72b9cde82428ea8acd83a5194f75d" member_type: VOTER last_known_addr { host: "127.28.113.1" port: 45679 } }
I20251215 21:02:55.738004 5372 raft_consensus.cc:385] T 23b0bbea258a4b91933d8a01a21598c7 P fe1797652ab94dff90667ef356a6c2ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251215 21:02:55.738083 5372 raft_consensus.cc:740] T 23b0bbea258a4b91933d8a01a21598c7 P fe1797652ab94dff90667ef356a6c2ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe1797652ab94dff90667ef356a6c2ae, State: Initialized, Role: FOLLOWER
I20251215 21:02:55.738075 5371 consensus_queue.cc:260] T 23b0bbea258a4b91933d8a01a21598c7 P 38e72b9cde82428ea8acd83a5194f75d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe1797652ab94dff90667ef356a6c2ae" member_type: VOTER last_known_addr { host: "127.28.113.3" port: 46345 } } peers { permanent_uuid: "f11866f255744d2fb31b4a6c2bcf6109" member_type: VOTER last_known_addr { host: "127.28.113.2" port: 44919 } } peers { permanent_uuid: "38e72b9cde82428ea8acd83a5194f75d" member_type: VOTER last_known_addr { host: "127.28.113.1" port: 45679 } }
I20251215 21:02:55.738510 5372 consensus_queue.cc:260] T 23b0bbea258a4b91933d8a01a21598c7 P fe1797652ab94dff90667ef356a6c2ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe1797652ab94dff90667ef356a6c2ae" member_type: VOTER last_known_addr { host: "127.28.113.3" port: 46345 } } peers { permanent_uuid: "f11866f255744d2fb31b4a6c2bcf6109" member_type: VOTER last_known_addr { host: "127.28.113.2" port: 44919 } } peers { permanent_uuid: "38e72b9cde82428ea8acd83a5194f75d" member_type: VOTER last_known_addr { host: "127.28.113.1" port: 45679 } }
I20251215 21:02:55.738516 5373 raft_consensus.cc:359] T 23b0bbea258a4b91933d8a01a21598c7 P f11866f255744d2fb31b4a6c2bcf6109 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe1797652ab94dff90667ef356a6c2ae" member_type: VOTER last_known_addr { host: "127.28.113.3" port: 46345 } } peers { permanent_uuid: "f11866f255744d2fb31b4a6c2bcf6109" member_type: VOTER last_known_addr { host: "127.28.113.2" port: 44919 } } peers { permanent_uuid: "38e72b9cde82428ea8acd83a5194f75d" member_type: VOTER last_known_addr { host: "127.28.113.1" port: 45679 } }
I20251215 21:02:55.738675 5373 raft_consensus.cc:385] T 23b0bbea258a4b91933d8a01a21598c7 P f11866f255744d2fb31b4a6c2bcf6109 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251215 21:02:55.738723 5373 raft_consensus.cc:740] T 23b0bbea258a4b91933d8a01a21598c7 P f11866f255744d2fb31b4a6c2bcf6109 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f11866f255744d2fb31b4a6c2bcf6109, State: Initialized, Role: FOLLOWER
I20251215 21:02:55.738719 5059 heartbeater.cc:499] Master 127.28.113.62:38827 was elected leader, sending a full tablet report...
I20251215 21:02:55.738871 5371 ts_tablet_manager.cc:1434] T 23b0bbea258a4b91933d8a01a21598c7 P 38e72b9cde82428ea8acd83a5194f75d: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20251215 21:02:55.739092 5373 consensus_queue.cc:260] T 23b0bbea258a4b91933d8a01a21598c7 P f11866f255744d2fb31b4a6c2bcf6109 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe1797652ab94dff90667ef356a6c2ae" member_type: VOTER last_known_addr { host: "127.28.113.3" port: 46345 } } peers { permanent_uuid: "f11866f255744d2fb31b4a6c2bcf6109" member_type: VOTER last_known_addr { host: "127.28.113.2" port: 44919 } } peers { permanent_uuid: "38e72b9cde82428ea8acd83a5194f75d" member_type: VOTER last_known_addr { host: "127.28.113.1" port: 45679 } }
I20251215 21:02:55.739285 5372 ts_tablet_manager.cc:1434] T 23b0bbea258a4b91933d8a01a21598c7 P fe1797652ab94dff90667ef356a6c2ae: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20251215 21:02:55.739576 5343 heartbeater.cc:499] Master 127.28.113.62:38827 was elected leader, sending a full tablet report...
I20251215 21:02:55.739611 5201 heartbeater.cc:499] Master 127.28.113.62:38827 was elected leader, sending a full tablet report...
I20251215 21:02:55.739714 5373 ts_tablet_manager.cc:1434] T 23b0bbea258a4b91933d8a01a21598c7 P f11866f255744d2fb31b4a6c2bcf6109: Time spent starting tablet: real 0.004s user 0.000s sys 0.004s
I20251215 21:02:55.754153 5379 raft_consensus.cc:493] T 23b0bbea258a4b91933d8a01a21598c7 P f11866f255744d2fb31b4a6c2bcf6109 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 21:02:55.754277 5379 raft_consensus.cc:515] T 23b0bbea258a4b91933d8a01a21598c7 P f11866f255744d2fb31b4a6c2bcf6109 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe1797652ab94dff90667ef356a6c2ae" member_type: VOTER last_known_addr { host: "127.28.113.3" port: 46345 } } peers { permanent_uuid: "f11866f255744d2fb31b4a6c2bcf6109" member_type: VOTER last_known_addr { host: "127.28.113.2" port: 44919 } } peers { permanent_uuid: "38e72b9cde82428ea8acd83a5194f75d" member_type: VOTER last_known_addr { host: "127.28.113.1" port: 45679 } }
I20251215 21:02:55.755183 5379 leader_election.cc:290] T 23b0bbea258a4b91933d8a01a21598c7 P f11866f255744d2fb31b4a6c2bcf6109 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe1797652ab94dff90667ef356a6c2ae (127.28.113.3:46345), 38e72b9cde82428ea8acd83a5194f75d (127.28.113.1:45679)
I20251215 21:02:55.757745 5094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:02:55.755385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.113.3:46345 (local address 127.28.113.2:46767)
1215 21:02:55.755504 (+ 119us) negotiation.cc:107] Waiting for socket to connect
1215 21:02:55.755519 (+ 15us) client_negotiation.cc:174] Beginning negotiation
1215 21:02:55.755602 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1215 21:02:55.755860 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1215 21:02:55.755863 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1215 21:02:55.755883 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1215 21:02:55.756122 (+ 239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:02:55.756129 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:02:55.756842 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1215 21:02:55.756845 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1215 21:02:55.757395 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:02:55.757403 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:02:55.757501 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:02:55.757524 (+ 23us) client_negotiation.cc:769] Sending connection context
1215 21:02:55.757573 (+ 49us) client_negotiation.cc:240] Negotiation successful
1215 21:02:55.757628 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20251215 21:02:55.758147 5380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:02:55.755385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.113.1:45679 (local address 127.28.113.2:35681)
1215 21:02:55.755780 (+ 395us) negotiation.cc:107] Waiting for socket to connect
1215 21:02:55.755789 (+ 9us) client_negotiation.cc:174] Beginning negotiation
1215 21:02:55.755858 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1215 21:02:55.756117 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1215 21:02:55.756120 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1215 21:02:55.756137 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1215 21:02:55.756367 (+ 230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:02:55.756373 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:02:55.757042 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1215 21:02:55.757044 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1215 21:02:55.757608 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:02:55.757615 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:02:55.757712 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:02:55.757729 (+ 17us) client_negotiation.cc:769] Sending connection context
1215 21:02:55.757776 (+ 47us) client_negotiation.cc:240] Negotiation successful
1215 21:02:55.758009 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":343,"mutex_wait_us":155,"thread_start_us":118,"threads_started":1}
I20251215 21:02:55.758183 5366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:02:55.755475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.113.2:46767 (local address 127.28.113.3:46345)
1215 21:02:55.755606 (+ 131us) server_negotiation.cc:207] Beginning negotiation
1215 21:02:55.755610 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1215 21:02:55.755626 (+ 16us) server_negotiation.cc:408] Connection header received
1215 21:02:55.755676 (+ 50us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1215 21:02:55.755679 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1215 21:02:55.755727 (+ 48us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1215 21:02:55.755805 (+ 78us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1215 21:02:55.756233 (+ 428us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:02:55.756722 (+ 489us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1215 21:02:55.757515 (+ 793us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:02:55.757966 (+ 451us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:02:55.758002 (+ 36us) server_negotiation.cc:1036] Waiting for connection context
1215 21:02:55.758048 (+ 46us) server_negotiation.cc:300] Negotiation successful
1215 21:02:55.758094 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20251215 21:02:55.758414 5369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:02:55.755498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.113.2:35681 (local address 127.28.113.1:45679)
1215 21:02:55.755606 (+ 108us) server_negotiation.cc:207] Beginning negotiation
1215 21:02:55.755610 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1215 21:02:55.755853 (+ 243us) server_negotiation.cc:408] Connection header received
1215 21:02:55.755950 (+ 97us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1215 21:02:55.755988 (+ 38us) server_negotiation.cc:462] Received NEGOTIATE request from client
1215 21:02:55.756035 (+ 47us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1215 21:02:55.756096 (+ 61us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1215 21:02:55.756471 (+ 375us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:02:55.756938 (+ 467us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1215 21:02:55.757724 (+ 786us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:02:55.758174 (+ 450us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:02:55.758209 (+ 35us) server_negotiation.cc:1036] Waiting for connection context
1215 21:02:55.758258 (+ 49us) server_negotiation.cc:300] Negotiation successful
1215 21:02:55.758314 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20251215 21:02:55.758679 5297 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "23b0bbea258a4b91933d8a01a21598c7" candidate_uuid: "f11866f255744d2fb31b4a6c2bcf6109" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe1797652ab94dff90667ef356a6c2ae" is_pre_election: true
I20251215 21:02:55.758893 5297 raft_consensus.cc:2468] T 23b0bbea258a4b91933d8a01a21598c7 P fe1797652ab94dff90667ef356a6c2ae [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f11866f255744d2fb31b4a6c2bcf6109 in term 0.
I20251215 21:02:55.758888 5013 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "23b0bbea258a4b91933d8a01a21598c7" candidate_uuid: "f11866f255744d2fb31b4a6c2bcf6109" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "38e72b9cde82428ea8acd83a5194f75d" is_pre_election: true
I20251215 21:02:55.759079 5013 raft_consensus.cc:2468] T 23b0bbea258a4b91933d8a01a21598c7 P 38e72b9cde82428ea8acd83a5194f75d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f11866f255744d2fb31b4a6c2bcf6109 in term 0.
I20251215 21:02:55.759277 5088 leader_election.cc:304] T 23b0bbea258a4b91933d8a01a21598c7 P f11866f255744d2fb31b4a6c2bcf6109 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f11866f255744d2fb31b4a6c2bcf6109, fe1797652ab94dff90667ef356a6c2ae; no voters:
I20251215 21:02:55.759461 5379 raft_consensus.cc:2804] T 23b0bbea258a4b91933d8a01a21598c7 P f11866f255744d2fb31b4a6c2bcf6109 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251215 21:02:55.759527 5379 raft_consensus.cc:493] T 23b0bbea258a4b91933d8a01a21598c7 P f11866f255744d2fb31b4a6c2bcf6109 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 21:02:55.759567 5379 raft_consensus.cc:3060] T 23b0bbea258a4b91933d8a01a21598c7 P f11866f255744d2fb31b4a6c2bcf6109 [term 0 FOLLOWER]: Advancing to term 1
I20251215 21:02:55.760427 5379 raft_consensus.cc:515] T 23b0bbea258a4b91933d8a01a21598c7 P f11866f255744d2fb31b4a6c2bcf6109 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe1797652ab94dff90667ef356a6c2ae" member_type: VOTER last_known_addr { host: "127.28.113.3" port: 46345 } } peers { permanent_uuid: "f11866f255744d2fb31b4a6c2bcf6109" member_type: VOTER last_known_addr { host: "127.28.113.2" port: 44919 } } peers { permanent_uuid: "38e72b9cde82428ea8acd83a5194f75d" member_type: VOTER last_known_addr { host: "127.28.113.1" port: 45679 } }
I20251215 21:02:55.760721 5379 leader_election.cc:290] T 23b0bbea258a4b91933d8a01a21598c7 P f11866f255744d2fb31b4a6c2bcf6109 [CANDIDATE]: Term 1 election: Requested vote from peers fe1797652ab94dff90667ef356a6c2ae (127.28.113.3:46345), 38e72b9cde82428ea8acd83a5194f75d (127.28.113.1:45679)
I20251215 21:02:55.761018 5297 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "23b0bbea258a4b91933d8a01a21598c7" candidate_uuid: "f11866f255744d2fb31b4a6c2bcf6109" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe1797652ab94dff90667ef356a6c2ae"
I20251215 21:02:55.761072 5013 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "23b0bbea258a4b91933d8a01a21598c7" candidate_uuid: "f11866f255744d2fb31b4a6c2bcf6109" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "38e72b9cde82428ea8acd83a5194f75d"
I20251215 21:02:55.761132 5297 raft_consensus.cc:3060] T 23b0bbea258a4b91933d8a01a21598c7 P fe1797652ab94dff90667ef356a6c2ae [term 0 FOLLOWER]: Advancing to term 1
I20251215 21:02:55.761196 5013 raft_consensus.cc:3060] T 23b0bbea258a4b91933d8a01a21598c7 P 38e72b9cde82428ea8acd83a5194f75d [term 0 FOLLOWER]: Advancing to term 1
I20251215 21:02:55.762168 5297 raft_consensus.cc:2468] T 23b0bbea258a4b91933d8a01a21598c7 P fe1797652ab94dff90667ef356a6c2ae [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f11866f255744d2fb31b4a6c2bcf6109 in term 1.
I20251215 21:02:55.762171 5013 raft_consensus.cc:2468] T 23b0bbea258a4b91933d8a01a21598c7 P 38e72b9cde82428ea8acd83a5194f75d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f11866f255744d2fb31b4a6c2bcf6109 in term 1.
I20251215 21:02:55.762420 5088 leader_election.cc:304] T 23b0bbea258a4b91933d8a01a21598c7 P f11866f255744d2fb31b4a6c2bcf6109 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f11866f255744d2fb31b4a6c2bcf6109, fe1797652ab94dff90667ef356a6c2ae; no voters:
I20251215 21:02:55.762562 5379 raft_consensus.cc:2804] T 23b0bbea258a4b91933d8a01a21598c7 P f11866f255744d2fb31b4a6c2bcf6109 [term 1 FOLLOWER]: Leader election won for term 1
I20251215 21:02:55.762754 5379 raft_consensus.cc:697] T 23b0bbea258a4b91933d8a01a21598c7 P f11866f255744d2fb31b4a6c2bcf6109 [term 1 LEADER]: Becoming Leader. State: Replica: f11866f255744d2fb31b4a6c2bcf6109, State: Running, Role: LEADER
I20251215 21:02:55.763044 5379 consensus_queue.cc:237] T 23b0bbea258a4b91933d8a01a21598c7 P f11866f255744d2fb31b4a6c2bcf6109 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe1797652ab94dff90667ef356a6c2ae" member_type: VOTER last_known_addr { host: "127.28.113.3" port: 46345 } } peers { permanent_uuid: "f11866f255744d2fb31b4a6c2bcf6109" member_type: VOTER last_known_addr { host: "127.28.113.2" port: 44919 } } peers { permanent_uuid: "38e72b9cde82428ea8acd83a5194f75d" member_type: VOTER last_known_addr { host: "127.28.113.1" port: 45679 } }
I20251215 21:02:55.765749 4852 catalog_manager.cc:5654] T 23b0bbea258a4b91933d8a01a21598c7 P f11866f255744d2fb31b4a6c2bcf6109 reported cstate change: term changed from 0 to 1, leader changed from <none> to f11866f255744d2fb31b4a6c2bcf6109 (127.28.113.2). New cstate: current_term: 1 leader_uuid: "f11866f255744d2fb31b4a6c2bcf6109" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe1797652ab94dff90667ef356a6c2ae" member_type: VOTER last_known_addr { host: "127.28.113.3" port: 46345 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f11866f255744d2fb31b4a6c2bcf6109" member_type: VOTER last_known_addr { host: "127.28.113.2" port: 44919 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "38e72b9cde82428ea8acd83a5194f75d" member_type: VOTER last_known_addr { host: "127.28.113.1" port: 45679 } health_report { overall_health: UNKNOWN } } }
I20251215 21:02:55.780894 5370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:02:55.777989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35134 (local address 127.28.113.2:44919)
1215 21:02:55.778123 (+ 134us) server_negotiation.cc:207] Beginning negotiation
1215 21:02:55.778128 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1215 21:02:55.778141 (+ 13us) server_negotiation.cc:408] Connection header received
1215 21:02:55.778280 (+ 139us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1215 21:02:55.778282 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1215 21:02:55.778333 (+ 51us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1215 21:02:55.778411 (+ 78us) server_negotiation.cc:227] Negotiated authn=TOKEN
1215 21:02:55.778779 (+ 368us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:02:55.779255 (+ 476us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1215 21:02:55.779874 (+ 619us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:02:55.780080 (+ 206us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:02:55.780133 (+ 53us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1215 21:02:55.780526 (+ 393us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1215 21:02:55.780624 (+ 98us) server_negotiation.cc:1036] Waiting for connection context
1215 21:02:55.780725 (+ 101us) server_negotiation.cc:300] Negotiation successful
1215 21:02:55.780769 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20251215 21:02:55.785635 5013 raft_consensus.cc:1275] T 23b0bbea258a4b91933d8a01a21598c7 P 38e72b9cde82428ea8acd83a5194f75d [term 1 FOLLOWER]: Refusing update from remote peer f11866f255744d2fb31b4a6c2bcf6109: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251215 21:02:55.785636 5297 raft_consensus.cc:1275] T 23b0bbea258a4b91933d8a01a21598c7 P fe1797652ab94dff90667ef356a6c2ae [term 1 FOLLOWER]: Refusing update from remote peer f11866f255744d2fb31b4a6c2bcf6109: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251215 21:02:55.786207 5381 consensus_queue.cc:1048] T 23b0bbea258a4b91933d8a01a21598c7 P f11866f255744d2fb31b4a6c2bcf6109 [LEADER]: Connected to new peer: Peer: permanent_uuid: "38e72b9cde82428ea8acd83a5194f75d" member_type: VOTER last_known_addr { host: "127.28.113.1" port: 45679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251215 21:02:55.786346 5379 consensus_queue.cc:1048] T 23b0bbea258a4b91933d8a01a21598c7 P f11866f255744d2fb31b4a6c2bcf6109 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe1797652ab94dff90667ef356a6c2ae" member_type: VOTER last_known_addr { host: "127.28.113.3" port: 46345 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251215 21:02:55.791939 5384 mvcc.cc:204] Tried to move back new op lower bound from 7232850230412337152 to 7232850230327795712. Current Snapshot: MvccSnapshot[applied={T|T < 7232850230412337152}]
I20251215 21:02:55.792649 5386 mvcc.cc:204] Tried to move back new op lower bound from 7232850230412337152 to 7232850230327795712. Current Snapshot: MvccSnapshot[applied={T|T < 7232850230412337152}]
I20251215 21:02:55.799815 4852 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34158:
table { table_name: "test-table" } modify_external_catalogs: true
I20251215 21:02:55.799950 4852 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34158:
table { table_name: "test-table" } modify_external_catalogs: true
I20251215 21:02:55.802146 4852 catalog_manager.cc:5941] T 00000000000000000000000000000000 P 4fc227cc99184e17b315174b90a73b5c: Sending DeleteTablet for 3 replicas of tablet 23b0bbea258a4b91933d8a01a21598c7
I20251215 21:02:55.802836 4993 tablet_service.cc:1552] Processing DeleteTablet for tablet 23b0bbea258a4b91933d8a01a21598c7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-15 21:02:55 UTC) from {username='oryx', principal='oryx/127.28.113.62@KRBTEST.COM'} at 127.0.0.1:41348
I20251215 21:02:55.802866 5135 tablet_service.cc:1552] Processing DeleteTablet for tablet 23b0bbea258a4b91933d8a01a21598c7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-15 21:02:55 UTC) from {username='oryx', principal='oryx/127.28.113.62@KRBTEST.COM'} at 127.0.0.1:35118
I20251215 21:02:55.803377 5277 tablet_service.cc:1552] Processing DeleteTablet for tablet 23b0bbea258a4b91933d8a01a21598c7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-15 21:02:55 UTC) from {username='oryx', principal='oryx/127.28.113.62@KRBTEST.COM'} at 127.0.0.1:50932
I20251215 21:02:55.803355 4852 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34158:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251215 21:02:55.803505 5394 tablet_replica.cc:333] T 23b0bbea258a4b91933d8a01a21598c7 P f11866f255744d2fb31b4a6c2bcf6109: stopping tablet replica
W20251215 21:02:55.803711 4852 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251215 21:02:55.803723 5394 raft_consensus.cc:2243] T 23b0bbea258a4b91933d8a01a21598c7 P f11866f255744d2fb31b4a6c2bcf6109 [term 1 LEADER]: Raft consensus shutting down.
I20251215 21:02:55.803870 5395 tablet_replica.cc:333] T 23b0bbea258a4b91933d8a01a21598c7 P fe1797652ab94dff90667ef356a6c2ae: stopping tablet replica
I20251215 21:02:55.804136 5394 raft_consensus.cc:2272] T 23b0bbea258a4b91933d8a01a21598c7 P f11866f255744d2fb31b4a6c2bcf6109 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251215 21:02:55.804162 5395 raft_consensus.cc:2243] T 23b0bbea258a4b91933d8a01a21598c7 P fe1797652ab94dff90667ef356a6c2ae [term 1 FOLLOWER]: Raft consensus shutting down.
I20251215 21:02:55.804468 5393 tablet_replica.cc:333] T 23b0bbea258a4b91933d8a01a21598c7 P 38e72b9cde82428ea8acd83a5194f75d: stopping tablet replica
I20251215 21:02:55.804677 5393 raft_consensus.cc:2243] T 23b0bbea258a4b91933d8a01a21598c7 P 38e72b9cde82428ea8acd83a5194f75d [term 1 FOLLOWER]: Raft consensus shutting down.
I20251215 21:02:55.804975 5393 raft_consensus.cc:2272] T 23b0bbea258a4b91933d8a01a21598c7 P 38e72b9cde82428ea8acd83a5194f75d [term 1 FOLLOWER]: Raft consensus is shut down!
I20251215 21:02:55.805490 5395 raft_consensus.cc:2272] T 23b0bbea258a4b91933d8a01a21598c7 P fe1797652ab94dff90667ef356a6c2ae [term 1 FOLLOWER]: Raft consensus is shut down!
I20251215 21:02:55.806247 5393 ts_tablet_manager.cc:1916] T 23b0bbea258a4b91933d8a01a21598c7 P 38e72b9cde82428ea8acd83a5194f75d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251215 21:02:55.806644 5395 ts_tablet_manager.cc:1916] T 23b0bbea258a4b91933d8a01a21598c7 P fe1797652ab94dff90667ef356a6c2ae: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251215 21:02:55.808666 5393 ts_tablet_manager.cc:1929] T 23b0bbea258a4b91933d8a01a21598c7 P 38e72b9cde82428ea8acd83a5194f75d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251215 21:02:55.808755 5393 log.cc:1199] T 23b0bbea258a4b91933d8a01a21598c7 P 38e72b9cde82428ea8acd83a5194f75d: Deleting WAL directory at /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765832506145157-29124-0/minicluster-data/ts-0/wal/wals/23b0bbea258a4b91933d8a01a21598c7
I20251215 21:02:55.808794 5395 ts_tablet_manager.cc:1929] T 23b0bbea258a4b91933d8a01a21598c7 P fe1797652ab94dff90667ef356a6c2ae: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251215 21:02:55.808854 5395 log.cc:1199] T 23b0bbea258a4b91933d8a01a21598c7 P fe1797652ab94dff90667ef356a6c2ae: Deleting WAL directory at /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765832506145157-29124-0/minicluster-data/ts-2/wal/wals/23b0bbea258a4b91933d8a01a21598c7
I20251215 21:02:55.809041 5393 ts_tablet_manager.cc:1950] T 23b0bbea258a4b91933d8a01a21598c7 P 38e72b9cde82428ea8acd83a5194f75d: Deleting consensus metadata
I20251215 21:02:55.809106 5395 ts_tablet_manager.cc:1950] T 23b0bbea258a4b91933d8a01a21598c7 P fe1797652ab94dff90667ef356a6c2ae: Deleting consensus metadata
I20251215 21:02:55.809543 5135 tablet_service.cc:1505] Processing CreateTablet for tablet fa857d00e2dd4965b89aa2213391e843 (DEFAULT_TABLE table=test-table [id=67cd9b8f91744369a13c2d39ba15caa0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251215 21:02:55.809543 5277 tablet_service.cc:1505] Processing CreateTablet for tablet fa857d00e2dd4965b89aa2213391e843 (DEFAULT_TABLE table=test-table [id=67cd9b8f91744369a13c2d39ba15caa0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251215 21:02:55.809777 5277 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fa857d00e2dd4965b89aa2213391e843. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 21:02:55.809777 5135 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fa857d00e2dd4965b89aa2213391e843. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 21:02:55.811831 5373 tablet_bootstrap.cc:492] T fa857d00e2dd4965b89aa2213391e843 P f11866f255744d2fb31b4a6c2bcf6109: Bootstrap starting.
I20251215 21:02:55.811864 4840 catalog_manager.cc:4985] TS fe1797652ab94dff90667ef356a6c2ae (127.28.113.3:46345): tablet 23b0bbea258a4b91933d8a01a21598c7 (table test-table [id=a158679eea3d4885b8fe75ab75765981]) successfully deleted
I20251215 21:02:55.812038 4839 catalog_manager.cc:4985] TS 38e72b9cde82428ea8acd83a5194f75d (127.28.113.1:45679): tablet 23b0bbea258a4b91933d8a01a21598c7 (table test-table [id=a158679eea3d4885b8fe75ab75765981]) successfully deleted
I20251215 21:02:55.812357 4993 tablet_service.cc:1505] Processing CreateTablet for tablet fa857d00e2dd4965b89aa2213391e843 (DEFAULT_TABLE table=test-table [id=67cd9b8f91744369a13c2d39ba15caa0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251215 21:02:55.812594 4993 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fa857d00e2dd4965b89aa2213391e843. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 21:02:55.812847 5373 tablet_bootstrap.cc:654] T fa857d00e2dd4965b89aa2213391e843 P f11866f255744d2fb31b4a6c2bcf6109: Neither blocks nor log segments found. Creating new log.
I20251215 21:02:55.813463 5372 tablet_bootstrap.cc:492] T fa857d00e2dd4965b89aa2213391e843 P fe1797652ab94dff90667ef356a6c2ae: Bootstrap starting.
I20251215 21:02:55.814033 5373 tablet_bootstrap.cc:492] T fa857d00e2dd4965b89aa2213391e843 P f11866f255744d2fb31b4a6c2bcf6109: No bootstrap required, opened a new log
I20251215 21:02:55.814116 5373 ts_tablet_manager.cc:1403] T fa857d00e2dd4965b89aa2213391e843 P f11866f255744d2fb31b4a6c2bcf6109: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20251215 21:02:55.814395 5372 tablet_bootstrap.cc:654] T fa857d00e2dd4965b89aa2213391e843 P fe1797652ab94dff90667ef356a6c2ae: Neither blocks nor log segments found. Creating new log.
I20251215 21:02:55.814553 5373 raft_consensus.cc:359] T fa857d00e2dd4965b89aa2213391e843 P f11866f255744d2fb31b4a6c2bcf6109 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38e72b9cde82428ea8acd83a5194f75d" member_type: VOTER last_known_addr { host: "127.28.113.1" port: 45679 } } peers { permanent_uuid: "f11866f255744d2fb31b4a6c2bcf6109" member_type: VOTER last_known_addr { host: "127.28.113.2" port: 44919 } } peers { permanent_uuid: "fe1797652ab94dff90667ef356a6c2ae" member_type: VOTER last_known_addr { host: "127.28.113.3" port: 46345 } }
I20251215 21:02:55.814656 5373 raft_consensus.cc:385] T fa857d00e2dd4965b89aa2213391e843 P f11866f255744d2fb31b4a6c2bcf6109 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251215 21:02:55.814687 5373 raft_consensus.cc:740] T fa857d00e2dd4965b89aa2213391e843 P f11866f255744d2fb31b4a6c2bcf6109 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f11866f255744d2fb31b4a6c2bcf6109, State: Initialized, Role: FOLLOWER
I20251215 21:02:55.814797 5373 consensus_queue.cc:260] T fa857d00e2dd4965b89aa2213391e843 P f11866f255744d2fb31b4a6c2bcf6109 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38e72b9cde82428ea8acd83a5194f75d" member_type: VOTER last_known_addr { host: "127.28.113.1" port: 45679 } } peers { permanent_uuid: "f11866f255744d2fb31b4a6c2bcf6109" member_type: VOTER last_known_addr { host: "127.28.113.2" port: 44919 } } peers { permanent_uuid: "fe1797652ab94dff90667ef356a6c2ae" member_type: VOTER last_known_addr { host: "127.28.113.3" port: 46345 } }
I20251215 21:02:55.815022 5373 ts_tablet_manager.cc:1434] T fa857d00e2dd4965b89aa2213391e843 P f11866f255744d2fb31b4a6c2bcf6109: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20251215 21:02:55.815111 5371 tablet_bootstrap.cc:492] T fa857d00e2dd4965b89aa2213391e843 P 38e72b9cde82428ea8acd83a5194f75d: Bootstrap starting.
I20251215 21:02:55.816082 5371 tablet_bootstrap.cc:654] T fa857d00e2dd4965b89aa2213391e843 P 38e72b9cde82428ea8acd83a5194f75d: Neither blocks nor log segments found. Creating new log.
I20251215 21:02:55.816387 5394 ts_tablet_manager.cc:1916] T 23b0bbea258a4b91933d8a01a21598c7 P f11866f255744d2fb31b4a6c2bcf6109: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251215 21:02:55.817507 5371 tablet_bootstrap.cc:492] T fa857d00e2dd4965b89aa2213391e843 P 38e72b9cde82428ea8acd83a5194f75d: No bootstrap required, opened a new log
I20251215 21:02:55.817585 5371 ts_tablet_manager.cc:1403] T fa857d00e2dd4965b89aa2213391e843 P 38e72b9cde82428ea8acd83a5194f75d: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20251215 21:02:55.817974 5371 raft_consensus.cc:359] T fa857d00e2dd4965b89aa2213391e843 P 38e72b9cde82428ea8acd83a5194f75d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38e72b9cde82428ea8acd83a5194f75d" member_type: VOTER last_known_addr { host: "127.28.113.1" port: 45679 } } peers { permanent_uuid: "f11866f255744d2fb31b4a6c2bcf6109" member_type: VOTER last_known_addr { host: "127.28.113.2" port: 44919 } } peers { permanent_uuid: "fe1797652ab94dff90667ef356a6c2ae" member_type: VOTER last_known_addr { host: "127.28.113.3" port: 46345 } }
I20251215 21:02:55.818084 5371 raft_consensus.cc:385] T fa857d00e2dd4965b89aa2213391e843 P 38e72b9cde82428ea8acd83a5194f75d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251215 21:02:55.818125 5371 raft_consensus.cc:740] T fa857d00e2dd4965b89aa2213391e843 P 38e72b9cde82428ea8acd83a5194f75d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 38e72b9cde82428ea8acd83a5194f75d, State: Initialized, Role: FOLLOWER
I20251215 21:02:55.818243 5371 consensus_queue.cc:260] T fa857d00e2dd4965b89aa2213391e843 P 38e72b9cde82428ea8acd83a5194f75d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38e72b9cde82428ea8acd83a5194f75d" member_type: VOTER last_known_addr { host: "127.28.113.1" port: 45679 } } peers { permanent_uuid: "f11866f255744d2fb31b4a6c2bcf6109" member_type: VOTER last_known_addr { host: "127.28.113.2" port: 44919 } } peers { permanent_uuid: "fe1797652ab94dff90667ef356a6c2ae" member_type: VOTER last_known_addr { host: "127.28.113.3" port: 46345 } }
I20251215 21:02:55.818482 5371 ts_tablet_manager.cc:1434] T fa857d00e2dd4965b89aa2213391e843 P 38e72b9cde82428ea8acd83a5194f75d: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20251215 21:02:55.819377 5372 tablet_bootstrap.cc:492] T fa857d00e2dd4965b89aa2213391e843 P fe1797652ab94dff90667ef356a6c2ae: No bootstrap required, opened a new log
I20251215 21:02:55.819459 5372 ts_tablet_manager.cc:1403] T fa857d00e2dd4965b89aa2213391e843 P fe1797652ab94dff90667ef356a6c2ae: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20251215 21:02:55.819913 5372 raft_consensus.cc:359] T fa857d00e2dd4965b89aa2213391e843 P fe1797652ab94dff90667ef356a6c2ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38e72b9cde82428ea8acd83a5194f75d" member_type: VOTER last_known_addr { host: "127.28.113.1" port: 45679 } } peers { permanent_uuid: "f11866f255744d2fb31b4a6c2bcf6109" member_type: VOTER last_known_addr { host: "127.28.113.2" port: 44919 } } peers { permanent_uuid: "fe1797652ab94dff90667ef356a6c2ae" member_type: VOTER last_known_addr { host: "127.28.113.3" port: 46345 } }
I20251215 21:02:55.820042 5372 raft_consensus.cc:385] T fa857d00e2dd4965b89aa2213391e843 P fe1797652ab94dff90667ef356a6c2ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251215 21:02:55.820080 5372 raft_consensus.cc:740] T fa857d00e2dd4965b89aa2213391e843 P fe1797652ab94dff90667ef356a6c2ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe1797652ab94dff90667ef356a6c2ae, State: Initialized, Role: FOLLOWER
I20251215 21:02:55.820231 5372 consensus_queue.cc:260] T fa857d00e2dd4965b89aa2213391e843 P fe1797652ab94dff90667ef356a6c2ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38e72b9cde82428ea8acd83a5194f75d" member_type: VOTER last_known_addr { host: "127.28.113.1" port: 45679 } } peers { permanent_uuid: "f11866f255744d2fb31b4a6c2bcf6109" member_type: VOTER last_known_addr { host: "127.28.113.2" port: 44919 } } peers { permanent_uuid: "fe1797652ab94dff90667ef356a6c2ae" member_type: VOTER last_known_addr { host: "127.28.113.3" port: 46345 } }
I20251215 21:02:55.820390 5394 ts_tablet_manager.cc:1929] T 23b0bbea258a4b91933d8a01a21598c7 P f11866f255744d2fb31b4a6c2bcf6109: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251215 21:02:55.820461 5372 ts_tablet_manager.cc:1434] T fa857d00e2dd4965b89aa2213391e843 P fe1797652ab94dff90667ef356a6c2ae: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20251215 21:02:55.820495 5394 log.cc:1199] T 23b0bbea258a4b91933d8a01a21598c7 P f11866f255744d2fb31b4a6c2bcf6109: Deleting WAL directory at /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765832506145157-29124-0/minicluster-data/ts-1/wal/wals/23b0bbea258a4b91933d8a01a21598c7
I20251215 21:02:55.820745 5394 ts_tablet_manager.cc:1950] T 23b0bbea258a4b91933d8a01a21598c7 P f11866f255744d2fb31b4a6c2bcf6109: Deleting consensus metadata
I20251215 21:02:55.821816 4840 catalog_manager.cc:4985] TS f11866f255744d2fb31b4a6c2bcf6109 (127.28.113.2:44919): tablet 23b0bbea258a4b91933d8a01a21598c7 (table test-table [id=a158679eea3d4885b8fe75ab75765981]) successfully deleted
I20251215 21:02:55.881217 5377 raft_consensus.cc:493] T fa857d00e2dd4965b89aa2213391e843 P 38e72b9cde82428ea8acd83a5194f75d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 21:02:55.881366 5377 raft_consensus.cc:515] T fa857d00e2dd4965b89aa2213391e843 P 38e72b9cde82428ea8acd83a5194f75d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38e72b9cde82428ea8acd83a5194f75d" member_type: VOTER last_known_addr { host: "127.28.113.1" port: 45679 } } peers { permanent_uuid: "f11866f255744d2fb31b4a6c2bcf6109" member_type: VOTER last_known_addr { host: "127.28.113.2" port: 44919 } } peers { permanent_uuid: "fe1797652ab94dff90667ef356a6c2ae" member_type: VOTER last_known_addr { host: "127.28.113.3" port: 46345 } }
I20251215 21:02:55.882570 5377 leader_election.cc:290] T fa857d00e2dd4965b89aa2213391e843 P 38e72b9cde82428ea8acd83a5194f75d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f11866f255744d2fb31b4a6c2bcf6109 (127.28.113.2:44919), fe1797652ab94dff90667ef356a6c2ae (127.28.113.3:46345)
I20251215 21:02:55.886045 5400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:02:55.883171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.113.3:46345 (local address 127.28.113.1:38357)
1215 21:02:55.883457 (+ 286us) negotiation.cc:107] Waiting for socket to connect
1215 21:02:55.883469 (+ 12us) client_negotiation.cc:174] Beginning negotiation
1215 21:02:55.883551 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1215 21:02:55.883791 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1215 21:02:55.883794 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1215 21:02:55.883810 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1215 21:02:55.884060 (+ 250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:02:55.884067 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:02:55.884789 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1215 21:02:55.884792 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1215 21:02:55.885628 (+ 836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:02:55.885638 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:02:55.885746 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:02:55.885771 (+ 25us) client_negotiation.cc:769] Sending connection context
1215 21:02:55.885820 (+ 49us) client_negotiation.cc:240] Negotiation successful
1215 21:02:55.885877 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":83,"threads_started":1}
I20251215 21:02:55.886502 5366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:02:55.883293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.113.1:38357 (local address 127.28.113.3:46345)
1215 21:02:55.883407 (+ 114us) server_negotiation.cc:207] Beginning negotiation
1215 21:02:55.883411 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1215 21:02:55.883537 (+ 126us) server_negotiation.cc:408] Connection header received
1215 21:02:55.883636 (+ 99us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1215 21:02:55.883639 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1215 21:02:55.883687 (+ 48us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1215 21:02:55.883768 (+ 81us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1215 21:02:55.884206 (+ 438us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:02:55.884675 (+ 469us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1215 21:02:55.885767 (+ 1092us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:02:55.886248 (+ 481us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:02:55.886287 (+ 39us) server_negotiation.cc:1036] Waiting for connection context
1215 21:02:55.886335 (+ 48us) server_negotiation.cc:300] Negotiation successful
1215 21:02:55.886389 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20251215 21:02:55.886855 5297 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fa857d00e2dd4965b89aa2213391e843" candidate_uuid: "38e72b9cde82428ea8acd83a5194f75d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe1797652ab94dff90667ef356a6c2ae" is_pre_election: true
I20251215 21:02:55.886999 5297 raft_consensus.cc:2468] T fa857d00e2dd4965b89aa2213391e843 P fe1797652ab94dff90667ef356a6c2ae [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38e72b9cde82428ea8acd83a5194f75d in term 0.
I20251215 21:02:55.887369 4946 leader_election.cc:304] T fa857d00e2dd4965b89aa2213391e843 P 38e72b9cde82428ea8acd83a5194f75d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38e72b9cde82428ea8acd83a5194f75d, fe1797652ab94dff90667ef356a6c2ae; no voters:
I20251215 21:02:55.887557 5370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:02:55.882915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.113.1:37731 (local address 127.28.113.2:44919)
1215 21:02:55.883067 (+ 152us) server_negotiation.cc:207] Beginning negotiation
1215 21:02:55.883072 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1215 21:02:55.883195 (+ 123us) server_negotiation.cc:408] Connection header received
1215 21:02:55.883405 (+ 210us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1215 21:02:55.883409 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1215 21:02:55.883461 (+ 52us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1215 21:02:55.883511 (+ 50us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1215 21:02:55.884132 (+ 621us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:02:55.884881 (+ 749us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1215 21:02:55.886633 (+ 1752us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:02:55.887132 (+ 499us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:02:55.887167 (+ 35us) server_negotiation.cc:1036] Waiting for connection context
1215 21:02:55.887384 (+ 217us) server_negotiation.cc:300] Negotiation successful
1215 21:02:55.887451 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20251215 21:02:55.887600 5377 raft_consensus.cc:2804] T fa857d00e2dd4965b89aa2213391e843 P 38e72b9cde82428ea8acd83a5194f75d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251215 21:02:55.887653 5377 raft_consensus.cc:493] T fa857d00e2dd4965b89aa2213391e843 P 38e72b9cde82428ea8acd83a5194f75d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 21:02:55.887691 5377 raft_consensus.cc:3060] T fa857d00e2dd4965b89aa2213391e843 P 38e72b9cde82428ea8acd83a5194f75d [term 0 FOLLOWER]: Advancing to term 1
I20251215 21:02:55.887588 5399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:02:55.882789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.113.2:44919 (local address 127.28.113.1:37731)
1215 21:02:55.883116 (+ 327us) negotiation.cc:107] Waiting for socket to connect
1215 21:02:55.883131 (+ 15us) client_negotiation.cc:174] Beginning negotiation
1215 21:02:55.883275 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1215 21:02:55.883650 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1215 21:02:55.883652 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1215 21:02:55.883668 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1215 21:02:55.884003 (+ 335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:02:55.884011 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:02:55.884989 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1215 21:02:55.884993 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1215 21:02:55.886495 (+ 1502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:02:55.886506 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:02:55.887241 (+ 735us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:02:55.887257 (+ 16us) client_negotiation.cc:769] Sending connection context
1215 21:02:55.887332 (+ 75us) client_negotiation.cc:240] Negotiation successful
1215 21:02:55.887393 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":80,"threads_started":1}
I20251215 21:02:55.888803 5377 raft_consensus.cc:515] T fa857d00e2dd4965b89aa2213391e843 P 38e72b9cde82428ea8acd83a5194f75d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38e72b9cde82428ea8acd83a5194f75d" member_type: VOTER last_known_addr { host: "127.28.113.1" port: 45679 } } peers { permanent_uuid: "f11866f255744d2fb31b4a6c2bcf6109" member_type: VOTER last_known_addr { host: "127.28.113.2" port: 44919 } } peers { permanent_uuid: "fe1797652ab94dff90667ef356a6c2ae" member_type: VOTER last_known_addr { host: "127.28.113.3" port: 46345 } }
I20251215 21:02:55.889154 5377 leader_election.cc:290] T fa857d00e2dd4965b89aa2213391e843 P 38e72b9cde82428ea8acd83a5194f75d [CANDIDATE]: Term 1 election: Requested vote from peers f11866f255744d2fb31b4a6c2bcf6109 (127.28.113.2:44919), fe1797652ab94dff90667ef356a6c2ae (127.28.113.3:46345)
I20251215 21:02:55.889539 5297 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fa857d00e2dd4965b89aa2213391e843" candidate_uuid: "38e72b9cde82428ea8acd83a5194f75d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe1797652ab94dff90667ef356a6c2ae"
I20251215 21:02:55.889551 5155 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fa857d00e2dd4965b89aa2213391e843" candidate_uuid: "38e72b9cde82428ea8acd83a5194f75d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f11866f255744d2fb31b4a6c2bcf6109" is_pre_election: true
I20251215 21:02:55.889639 5297 raft_consensus.cc:3060] T fa857d00e2dd4965b89aa2213391e843 P fe1797652ab94dff90667ef356a6c2ae [term 0 FOLLOWER]: Advancing to term 1
I20251215 21:02:55.889767 5155 raft_consensus.cc:2468] T fa857d00e2dd4965b89aa2213391e843 P f11866f255744d2fb31b4a6c2bcf6109 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38e72b9cde82428ea8acd83a5194f75d in term 0.
I20251215 21:02:55.889868 5154 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fa857d00e2dd4965b89aa2213391e843" candidate_uuid: "38e72b9cde82428ea8acd83a5194f75d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f11866f255744d2fb31b4a6c2bcf6109"
I20251215 21:02:55.889948 5154 raft_consensus.cc:3060] T fa857d00e2dd4965b89aa2213391e843 P f11866f255744d2fb31b4a6c2bcf6109 [term 0 FOLLOWER]: Advancing to term 1
I20251215 21:02:55.890635 5297 raft_consensus.cc:2468] T fa857d00e2dd4965b89aa2213391e843 P fe1797652ab94dff90667ef356a6c2ae [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38e72b9cde82428ea8acd83a5194f75d in term 1.
I20251215 21:02:55.890733 5154 raft_consensus.cc:2468] T fa857d00e2dd4965b89aa2213391e843 P f11866f255744d2fb31b4a6c2bcf6109 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38e72b9cde82428ea8acd83a5194f75d in term 1.
I20251215 21:02:55.890957 4946 leader_election.cc:304] T fa857d00e2dd4965b89aa2213391e843 P 38e72b9cde82428ea8acd83a5194f75d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38e72b9cde82428ea8acd83a5194f75d, fe1797652ab94dff90667ef356a6c2ae; no voters:
I20251215 21:02:55.891139 5377 raft_consensus.cc:2804] T fa857d00e2dd4965b89aa2213391e843 P 38e72b9cde82428ea8acd83a5194f75d [term 1 FOLLOWER]: Leader election won for term 1
I20251215 21:02:55.891342 5377 raft_consensus.cc:697] T fa857d00e2dd4965b89aa2213391e843 P 38e72b9cde82428ea8acd83a5194f75d [term 1 LEADER]: Becoming Leader. State: Replica: 38e72b9cde82428ea8acd83a5194f75d, State: Running, Role: LEADER
I20251215 21:02:55.891592 5377 consensus_queue.cc:237] T fa857d00e2dd4965b89aa2213391e843 P 38e72b9cde82428ea8acd83a5194f75d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38e72b9cde82428ea8acd83a5194f75d" member_type: VOTER last_known_addr { host: "127.28.113.1" port: 45679 } } peers { permanent_uuid: "f11866f255744d2fb31b4a6c2bcf6109" member_type: VOTER last_known_addr { host: "127.28.113.2" port: 44919 } } peers { permanent_uuid: "fe1797652ab94dff90667ef356a6c2ae" member_type: VOTER last_known_addr { host: "127.28.113.3" port: 46345 } }
I20251215 21:02:55.893898 4853 catalog_manager.cc:5654] T fa857d00e2dd4965b89aa2213391e843 P 38e72b9cde82428ea8acd83a5194f75d reported cstate change: term changed from 0 to 1, leader changed from <none> to 38e72b9cde82428ea8acd83a5194f75d (127.28.113.1). New cstate: current_term: 1 leader_uuid: "38e72b9cde82428ea8acd83a5194f75d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38e72b9cde82428ea8acd83a5194f75d" member_type: VOTER last_known_addr { host: "127.28.113.1" port: 45679 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f11866f255744d2fb31b4a6c2bcf6109" member_type: VOTER last_known_addr { host: "127.28.113.2" port: 44919 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fe1797652ab94dff90667ef356a6c2ae" member_type: VOTER last_known_addr { host: "127.28.113.3" port: 46345 } health_report { overall_health: UNKNOWN } } }
W20251215 21:02:55.914882 5344 tablet.cc:2378] T fa857d00e2dd4965b89aa2213391e843 P fe1797652ab94dff90667ef356a6c2ae: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20251215 21:02:55.920969 5202 tablet.cc:2378] T fa857d00e2dd4965b89aa2213391e843 P f11866f255744d2fb31b4a6c2bcf6109: Can't schedule compaction. Clean time has not been advanced past its initial value.
Dec 15 21:02:55 dist-test-slave-bjsf krb5kdc[4806](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765832574, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.113.62@KRBTEST.COM for oryx/127.28.113.62@KRBTEST.COM
I20251215 21:02:55.941659 5408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:02:55.933739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.113.62:38827 (local address 127.0.0.1:34162)
1215 21:02:55.933985 (+ 246us) negotiation.cc:107] Waiting for socket to connect
1215 21:02:55.933997 (+ 12us) client_negotiation.cc:174] Beginning negotiation
1215 21:02:55.934099 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1215 21:02:55.934361 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1215 21:02:55.934364 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1215 21:02:55.934731 (+ 367us) client_negotiation.cc:189] Negotiated authn=SASL
1215 21:02:55.934926 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:02:55.934933 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:02:55.935929 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1215 21:02:55.935932 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1215 21:02:55.936341 (+ 409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:02:55.936350 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:02:55.936439 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:02:55.936998 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1215 21:02:55.937011 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
1215 21:02:55.938539 (+ 1528us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1215 21:02:55.940365 (+ 1826us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1215 21:02:55.940372 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1215 21:02:55.940376 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
1215 21:02:55.940569 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1215 21:02:55.940830 (+ 261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1215 21:02:55.940833 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1215 21:02:55.940834 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1215 21:02:55.940877 (+ 43us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1215 21:02:55.941250 (+ 373us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1215 21:02:55.941255 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1215 21:02:55.941343 (+ 88us) client_negotiation.cc:769] Sending connection context
1215 21:02:55.941427 (+ 84us) client_negotiation.cc:240] Negotiation successful
1215 21:02:55.941526 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":116,"threads_started":1}
I20251215 21:02:55.941803 4903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:02:55.933651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34162 (local address 127.28.113.62:38827)
1215 21:02:55.933785 (+ 134us) server_negotiation.cc:207] Beginning negotiation
1215 21:02:55.933789 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1215 21:02:55.934083 (+ 294us) server_negotiation.cc:408] Connection header received
1215 21:02:55.934198 (+ 115us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1215 21:02:55.934202 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1215 21:02:55.934258 (+ 56us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1215 21:02:55.934354 (+ 96us) server_negotiation.cc:227] Negotiated authn=SASL
1215 21:02:55.935060 (+ 706us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:02:55.935817 (+ 757us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1215 21:02:55.936470 (+ 653us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:02:55.936654 (+ 184us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:02:55.938668 (+ 2014us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1215 21:02:55.938686 (+ 18us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1215 21:02:55.938690 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1215 21:02:55.938716 (+ 26us) server_negotiation.cc:923] calling sasl_server_start()
1215 21:02:55.940261 (+ 1545us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1215 21:02:55.940676 (+ 415us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1215 21:02:55.940679 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1215 21:02:55.940681 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1215 21:02:55.940734 (+ 53us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1215 21:02:55.940961 (+ 227us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1215 21:02:55.940964 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1215 21:02:55.940966 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1215 21:02:55.941158 (+ 192us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1215 21:02:55.941261 (+ 103us) server_negotiation.cc:1036] Waiting for connection context
1215 21:02:55.941514 (+ 253us) server_negotiation.cc:300] Negotiation successful
1215 21:02:55.941627 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20251215 21:02:55.945494 4853 catalog_manager.cc:2257] Servicing CreateTable request from {username='oryx', principal='oryx/127.28.113.62@KRBTEST.COM'} at 127.0.0.1:34162:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20251215 21:02:55.946167 4853 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251215 21:02:55.951119 5135 tablet_service.cc:1505] Processing CreateTablet for tablet a50663f3fbe14634800d2e38c7413d61 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=84f383077e0545528ae87b91e9aa1690]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20251215 21:02:55.951119 4993 tablet_service.cc:1505] Processing CreateTablet for tablet a50663f3fbe14634800d2e38c7413d61 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=84f383077e0545528ae87b91e9aa1690]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20251215 21:02:55.951484 4993 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a50663f3fbe14634800d2e38c7413d61. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 21:02:55.951483 5135 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a50663f3fbe14634800d2e38c7413d61. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 21:02:55.954056 5373 tablet_bootstrap.cc:492] T a50663f3fbe14634800d2e38c7413d61 P f11866f255744d2fb31b4a6c2bcf6109: Bootstrap starting.
I20251215 21:02:55.954294 5277 tablet_service.cc:1505] Processing CreateTablet for tablet a50663f3fbe14634800d2e38c7413d61 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=84f383077e0545528ae87b91e9aa1690]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20251215 21:02:55.954614 5277 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a50663f3fbe14634800d2e38c7413d61. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 21:02:55.955158 5373 tablet_bootstrap.cc:654] T a50663f3fbe14634800d2e38c7413d61 P f11866f255744d2fb31b4a6c2bcf6109: Neither blocks nor log segments found. Creating new log.
I20251215 21:02:55.956871 5372 tablet_bootstrap.cc:492] T a50663f3fbe14634800d2e38c7413d61 P fe1797652ab94dff90667ef356a6c2ae: Bootstrap starting.
I20251215 21:02:55.956902 5373 tablet_bootstrap.cc:492] T a50663f3fbe14634800d2e38c7413d61 P f11866f255744d2fb31b4a6c2bcf6109: No bootstrap required, opened a new log
I20251215 21:02:55.957000 5373 ts_tablet_manager.cc:1403] T a50663f3fbe14634800d2e38c7413d61 P f11866f255744d2fb31b4a6c2bcf6109: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.001s
I20251215 21:02:55.957463 5373 raft_consensus.cc:359] T a50663f3fbe14634800d2e38c7413d61 P f11866f255744d2fb31b4a6c2bcf6109 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38e72b9cde82428ea8acd83a5194f75d" member_type: VOTER last_known_addr { host: "127.28.113.1" port: 45679 } } peers { permanent_uuid: "f11866f255744d2fb31b4a6c2bcf6109" member_type: VOTER last_known_addr { host: "127.28.113.2" port: 44919 } } peers { permanent_uuid: "fe1797652ab94dff90667ef356a6c2ae" member_type: VOTER last_known_addr { host: "127.28.113.3" port: 46345 } }
I20251215 21:02:55.957567 5373 raft_consensus.cc:385] T a50663f3fbe14634800d2e38c7413d61 P f11866f255744d2fb31b4a6c2bcf6109 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251215 21:02:55.957599 5373 raft_consensus.cc:740] T a50663f3fbe14634800d2e38c7413d61 P f11866f255744d2fb31b4a6c2bcf6109 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f11866f255744d2fb31b4a6c2bcf6109, State: Initialized, Role: FOLLOWER
I20251215 21:02:55.957715 5372 tablet_bootstrap.cc:654] T a50663f3fbe14634800d2e38c7413d61 P fe1797652ab94dff90667ef356a6c2ae: Neither blocks nor log segments found. Creating new log.
I20251215 21:02:55.957731 5373 consensus_queue.cc:260] T a50663f3fbe14634800d2e38c7413d61 P f11866f255744d2fb31b4a6c2bcf6109 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38e72b9cde82428ea8acd83a5194f75d" member_type: VOTER last_known_addr { host: "127.28.113.1" port: 45679 } } peers { permanent_uuid: "f11866f255744d2fb31b4a6c2bcf6109" member_type: VOTER last_known_addr { host: "127.28.113.2" port: 44919 } } peers { permanent_uuid: "fe1797652ab94dff90667ef356a6c2ae" member_type: VOTER last_known_addr { host: "127.28.113.3" port: 46345 } }
I20251215 21:02:55.958006 5373 ts_tablet_manager.cc:1434] T a50663f3fbe14634800d2e38c7413d61 P f11866f255744d2fb31b4a6c2bcf6109: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20251215 21:02:55.958037 5381 tablet_replica.cc:442] T a50663f3fbe14634800d2e38c7413d61 P f11866f255744d2fb31b4a6c2bcf6109: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38e72b9cde82428ea8acd83a5194f75d" member_type: VOTER last_known_addr { host: "127.28.113.1" port: 45679 } } peers { permanent_uuid: "f11866f255744d2fb31b4a6c2bcf6109" member_type: VOTER last_known_addr { host: "127.28.113.2" port: 44919 } } peers { permanent_uuid: "fe1797652ab94dff90667ef356a6c2ae" member_type: VOTER last_known_addr { host: "127.28.113.3" port: 46345 } } }
I20251215 21:02:55.958153 5381 tablet_replica.cc:445] T a50663f3fbe14634800d2e38c7413d61 P f11866f255744d2fb31b4a6c2bcf6109: This TxnStatusTablet replica's current role is: FOLLOWER
I20251215 21:02:55.959353 5372 tablet_bootstrap.cc:492] T a50663f3fbe14634800d2e38c7413d61 P fe1797652ab94dff90667ef356a6c2ae: No bootstrap required, opened a new log
I20251215 21:02:55.959434 5372 ts_tablet_manager.cc:1403] T a50663f3fbe14634800d2e38c7413d61 P fe1797652ab94dff90667ef356a6c2ae: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20251215 21:02:55.959813 5372 raft_consensus.cc:359] T a50663f3fbe14634800d2e38c7413d61 P fe1797652ab94dff90667ef356a6c2ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38e72b9cde82428ea8acd83a5194f75d" member_type: VOTER last_known_addr { host: "127.28.113.1" port: 45679 } } peers { permanent_uuid: "f11866f255744d2fb31b4a6c2bcf6109" member_type: VOTER last_known_addr { host: "127.28.113.2" port: 44919 } } peers { permanent_uuid: "fe1797652ab94dff90667ef356a6c2ae" member_type: VOTER last_known_addr { host: "127.28.113.3" port: 46345 } }
I20251215 21:02:55.959913 5372 raft_consensus.cc:385] T a50663f3fbe14634800d2e38c7413d61 P fe1797652ab94dff90667ef356a6c2ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251215 21:02:55.959944 5372 raft_consensus.cc:740] T a50663f3fbe14634800d2e38c7413d61 P fe1797652ab94dff90667ef356a6c2ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe1797652ab94dff90667ef356a6c2ae, State: Initialized, Role: FOLLOWER
I20251215 21:02:55.960103 5372 consensus_queue.cc:260] T a50663f3fbe14634800d2e38c7413d61 P fe1797652ab94dff90667ef356a6c2ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38e72b9cde82428ea8acd83a5194f75d" member_type: VOTER last_known_addr { host: "127.28.113.1" port: 45679 } } peers { permanent_uuid: "f11866f255744d2fb31b4a6c2bcf6109" member_type: VOTER last_known_addr { host: "127.28.113.2" port: 44919 } } peers { permanent_uuid: "fe1797652ab94dff90667ef356a6c2ae" member_type: VOTER last_known_addr { host: "127.28.113.3" port: 46345 } }
I20251215 21:02:55.960328 5372 ts_tablet_manager.cc:1434] T a50663f3fbe14634800d2e38c7413d61 P fe1797652ab94dff90667ef356a6c2ae: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20251215 21:02:55.960855 5378 tablet_replica.cc:442] T a50663f3fbe14634800d2e38c7413d61 P fe1797652ab94dff90667ef356a6c2ae: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38e72b9cde82428ea8acd83a5194f75d" member_type: VOTER last_known_addr { host: "127.28.113.1" port: 45679 } } peers { permanent_uuid: "f11866f255744d2fb31b4a6c2bcf6109" member_type: VOTER last_known_addr { host: "127.28.113.2" port: 44919 } } peers { permanent_uuid: "fe1797652ab94dff90667ef356a6c2ae" member_type: VOTER last_known_addr { host: "127.28.113.3" port: 46345 } } }
I20251215 21:02:55.960973 5378 tablet_replica.cc:445] T a50663f3fbe14634800d2e38c7413d61 P fe1797652ab94dff90667ef356a6c2ae: This TxnStatusTablet replica's current role is: FOLLOWER
I20251215 21:02:55.964206 5371 tablet_bootstrap.cc:492] T a50663f3fbe14634800d2e38c7413d61 P 38e72b9cde82428ea8acd83a5194f75d: Bootstrap starting.
I20251215 21:02:55.965128 5371 tablet_bootstrap.cc:654] T a50663f3fbe14634800d2e38c7413d61 P 38e72b9cde82428ea8acd83a5194f75d: Neither blocks nor log segments found. Creating new log.
W20251215 21:02:55.965822 5060 tablet.cc:2378] T fa857d00e2dd4965b89aa2213391e843 P 38e72b9cde82428ea8acd83a5194f75d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20251215 21:02:55.966626 5371 tablet_bootstrap.cc:492] T a50663f3fbe14634800d2e38c7413d61 P 38e72b9cde82428ea8acd83a5194f75d: No bootstrap required, opened a new log
I20251215 21:02:55.966713 5371 ts_tablet_manager.cc:1403] T a50663f3fbe14634800d2e38c7413d61 P 38e72b9cde82428ea8acd83a5194f75d: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20251215 21:02:55.967146 5371 raft_consensus.cc:359] T a50663f3fbe14634800d2e38c7413d61 P 38e72b9cde82428ea8acd83a5194f75d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38e72b9cde82428ea8acd83a5194f75d" member_type: VOTER last_known_addr { host: "127.28.113.1" port: 45679 } } peers { permanent_uuid: "f11866f255744d2fb31b4a6c2bcf6109" member_type: VOTER last_known_addr { host: "127.28.113.2" port: 44919 } } peers { permanent_uuid: "fe1797652ab94dff90667ef356a6c2ae" member_type: VOTER last_known_addr { host: "127.28.113.3" port: 46345 } }
I20251215 21:02:55.967236 5371 raft_consensus.cc:385] T a50663f3fbe14634800d2e38c7413d61 P 38e72b9cde82428ea8acd83a5194f75d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251215 21:02:55.967257 5371 raft_consensus.cc:740] T a50663f3fbe14634800d2e38c7413d61 P 38e72b9cde82428ea8acd83a5194f75d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 38e72b9cde82428ea8acd83a5194f75d, State: Initialized, Role: FOLLOWER
I20251215 21:02:55.967362 5371 consensus_queue.cc:260] T a50663f3fbe14634800d2e38c7413d61 P 38e72b9cde82428ea8acd83a5194f75d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38e72b9cde82428ea8acd83a5194f75d" member_type: VOTER last_known_addr { host: "127.28.113.1" port: 45679 } } peers { permanent_uuid: "f11866f255744d2fb31b4a6c2bcf6109" member_type: VOTER last_known_addr { host: "127.28.113.2" port: 44919 } } peers { permanent_uuid: "fe1797652ab94dff90667ef356a6c2ae" member_type: VOTER last_known_addr { host: "127.28.113.3" port: 46345 } }
I20251215 21:02:55.967571 5377 tablet_replica.cc:442] T a50663f3fbe14634800d2e38c7413d61 P 38e72b9cde82428ea8acd83a5194f75d: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38e72b9cde82428ea8acd83a5194f75d" member_type: VOTER last_known_addr { host: "127.28.113.1" port: 45679 } } peers { permanent_uuid: "f11866f255744d2fb31b4a6c2bcf6109" member_type: VOTER last_known_addr { host: "127.28.113.2" port: 44919 } } peers { permanent_uuid: "fe1797652ab94dff90667ef356a6c2ae" member_type: VOTER last_known_addr { host: "127.28.113.3" port: 46345 } } }
I20251215 21:02:55.967681 5377 tablet_replica.cc:445] T a50663f3fbe14634800d2e38c7413d61 P 38e72b9cde82428ea8acd83a5194f75d: This TxnStatusTablet replica's current role is: FOLLOWER
I20251215 21:02:55.967617 5371 ts_tablet_manager.cc:1434] T a50663f3fbe14634800d2e38c7413d61 P 38e72b9cde82428ea8acd83a5194f75d: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20251215 21:02:56.161451 5381 raft_consensus.cc:493] T a50663f3fbe14634800d2e38c7413d61 P f11866f255744d2fb31b4a6c2bcf6109 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 21:02:56.161597 5381 raft_consensus.cc:515] T a50663f3fbe14634800d2e38c7413d61 P f11866f255744d2fb31b4a6c2bcf6109 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38e72b9cde82428ea8acd83a5194f75d" member_type: VOTER last_known_addr { host: "127.28.113.1" port: 45679 } } peers { permanent_uuid: "f11866f255744d2fb31b4a6c2bcf6109" member_type: VOTER last_known_addr { host: "127.28.113.2" port: 44919 } } peers { permanent_uuid: "fe1797652ab94dff90667ef356a6c2ae" member_type: VOTER last_known_addr { host: "127.28.113.3" port: 46345 } }
I20251215 21:02:56.162060 5381 leader_election.cc:290] T a50663f3fbe14634800d2e38c7413d61 P f11866f255744d2fb31b4a6c2bcf6109 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 38e72b9cde82428ea8acd83a5194f75d (127.28.113.1:45679), fe1797652ab94dff90667ef356a6c2ae (127.28.113.3:46345)
I20251215 21:02:56.162510 5297 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a50663f3fbe14634800d2e38c7413d61" candidate_uuid: "f11866f255744d2fb31b4a6c2bcf6109" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe1797652ab94dff90667ef356a6c2ae" is_pre_election: true
I20251215 21:02:56.162503 5013 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a50663f3fbe14634800d2e38c7413d61" candidate_uuid: "f11866f255744d2fb31b4a6c2bcf6109" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "38e72b9cde82428ea8acd83a5194f75d" is_pre_election: true
I20251215 21:02:56.162667 5297 raft_consensus.cc:2468] T a50663f3fbe14634800d2e38c7413d61 P fe1797652ab94dff90667ef356a6c2ae [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f11866f255744d2fb31b4a6c2bcf6109 in term 0.
I20251215 21:02:56.162673 5013 raft_consensus.cc:2468] T a50663f3fbe14634800d2e38c7413d61 P 38e72b9cde82428ea8acd83a5194f75d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f11866f255744d2fb31b4a6c2bcf6109 in term 0.
I20251215 21:02:56.162962 5088 leader_election.cc:304] T a50663f3fbe14634800d2e38c7413d61 P f11866f255744d2fb31b4a6c2bcf6109 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f11866f255744d2fb31b4a6c2bcf6109, fe1797652ab94dff90667ef356a6c2ae; no voters:
I20251215 21:02:56.163136 5381 raft_consensus.cc:2804] T a50663f3fbe14634800d2e38c7413d61 P f11866f255744d2fb31b4a6c2bcf6109 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251215 21:02:56.163192 5381 raft_consensus.cc:493] T a50663f3fbe14634800d2e38c7413d61 P f11866f255744d2fb31b4a6c2bcf6109 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 21:02:56.163228 5381 raft_consensus.cc:3060] T a50663f3fbe14634800d2e38c7413d61 P f11866f255744d2fb31b4a6c2bcf6109 [term 0 FOLLOWER]: Advancing to term 1
I20251215 21:02:56.164186 5381 raft_consensus.cc:515] T a50663f3fbe14634800d2e38c7413d61 P f11866f255744d2fb31b4a6c2bcf6109 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38e72b9cde82428ea8acd83a5194f75d" member_type: VOTER last_known_addr { host: "127.28.113.1" port: 45679 } } peers { permanent_uuid: "f11866f255744d2fb31b4a6c2bcf6109" member_type: VOTER last_known_addr { host: "127.28.113.2" port: 44919 } } peers { permanent_uuid: "fe1797652ab94dff90667ef356a6c2ae" member_type: VOTER last_known_addr { host: "127.28.113.3" port: 46345 } }
I20251215 21:02:56.164539 5381 leader_election.cc:290] T a50663f3fbe14634800d2e38c7413d61 P f11866f255744d2fb31b4a6c2bcf6109 [CANDIDATE]: Term 1 election: Requested vote from peers 38e72b9cde82428ea8acd83a5194f75d (127.28.113.1:45679), fe1797652ab94dff90667ef356a6c2ae (127.28.113.3:46345)
I20251215 21:02:56.164898 5013 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a50663f3fbe14634800d2e38c7413d61" candidate_uuid: "f11866f255744d2fb31b4a6c2bcf6109" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "38e72b9cde82428ea8acd83a5194f75d"
I20251215 21:02:56.164918 5297 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a50663f3fbe14634800d2e38c7413d61" candidate_uuid: "f11866f255744d2fb31b4a6c2bcf6109" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe1797652ab94dff90667ef356a6c2ae"
I20251215 21:02:56.165011 5297 raft_consensus.cc:3060] T a50663f3fbe14634800d2e38c7413d61 P fe1797652ab94dff90667ef356a6c2ae [term 0 FOLLOWER]: Advancing to term 1
I20251215 21:02:56.165009 5013 raft_consensus.cc:3060] T a50663f3fbe14634800d2e38c7413d61 P 38e72b9cde82428ea8acd83a5194f75d [term 0 FOLLOWER]: Advancing to term 1
I20251215 21:02:56.165925 5297 raft_consensus.cc:2468] T a50663f3fbe14634800d2e38c7413d61 P fe1797652ab94dff90667ef356a6c2ae [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f11866f255744d2fb31b4a6c2bcf6109 in term 1.
I20251215 21:02:56.165932 5013 raft_consensus.cc:2468] T a50663f3fbe14634800d2e38c7413d61 P 38e72b9cde82428ea8acd83a5194f75d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f11866f255744d2fb31b4a6c2bcf6109 in term 1.
I20251215 21:02:56.166213 5088 leader_election.cc:304] T a50663f3fbe14634800d2e38c7413d61 P f11866f255744d2fb31b4a6c2bcf6109 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f11866f255744d2fb31b4a6c2bcf6109, fe1797652ab94dff90667ef356a6c2ae; no voters:
I20251215 21:02:56.166363 5381 raft_consensus.cc:2804] T a50663f3fbe14634800d2e38c7413d61 P f11866f255744d2fb31b4a6c2bcf6109 [term 1 FOLLOWER]: Leader election won for term 1
I20251215 21:02:56.166446 5381 raft_consensus.cc:697] T a50663f3fbe14634800d2e38c7413d61 P f11866f255744d2fb31b4a6c2bcf6109 [term 1 LEADER]: Becoming Leader. State: Replica: f11866f255744d2fb31b4a6c2bcf6109, State: Running, Role: LEADER
I20251215 21:02:56.166581 5381 consensus_queue.cc:237] T a50663f3fbe14634800d2e38c7413d61 P f11866f255744d2fb31b4a6c2bcf6109 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38e72b9cde82428ea8acd83a5194f75d" member_type: VOTER last_known_addr { host: "127.28.113.1" port: 45679 } } peers { permanent_uuid: "f11866f255744d2fb31b4a6c2bcf6109" member_type: VOTER last_known_addr { host: "127.28.113.2" port: 44919 } } peers { permanent_uuid: "fe1797652ab94dff90667ef356a6c2ae" member_type: VOTER last_known_addr { host: "127.28.113.3" port: 46345 } }
I20251215 21:02:56.167207 5379 tablet_replica.cc:442] T a50663f3fbe14634800d2e38c7413d61 P f11866f255744d2fb31b4a6c2bcf6109: TxnStatusTablet state changed. Reason: New leader f11866f255744d2fb31b4a6c2bcf6109. Latest consensus state: current_term: 1 leader_uuid: "f11866f255744d2fb31b4a6c2bcf6109" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38e72b9cde82428ea8acd83a5194f75d" member_type: VOTER last_known_addr { host: "127.28.113.1" port: 45679 } } peers { permanent_uuid: "f11866f255744d2fb31b4a6c2bcf6109" member_type: VOTER last_known_addr { host: "127.28.113.2" port: 44919 } } peers { permanent_uuid: "fe1797652ab94dff90667ef356a6c2ae" member_type: VOTER last_known_addr { host: "127.28.113.3" port: 46345 } } }
I20251215 21:02:56.167302 5379 tablet_replica.cc:445] T a50663f3fbe14634800d2e38c7413d61 P f11866f255744d2fb31b4a6c2bcf6109: This TxnStatusTablet replica's current role is: LEADER
I20251215 21:02:56.167513 5413 txn_status_manager.cc:874] Waiting until node catch up with all replicated operations in previous term...
I20251215 21:02:56.167655 5413 txn_status_manager.cc:930] Loading transaction status metadata into memory...
I20251215 21:02:56.168287 4852 catalog_manager.cc:5654] T a50663f3fbe14634800d2e38c7413d61 P f11866f255744d2fb31b4a6c2bcf6109 reported cstate change: term changed from 0 to 1, leader changed from <none> to f11866f255744d2fb31b4a6c2bcf6109 (127.28.113.2). New cstate: current_term: 1 leader_uuid: "f11866f255744d2fb31b4a6c2bcf6109" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38e72b9cde82428ea8acd83a5194f75d" member_type: VOTER last_known_addr { host: "127.28.113.1" port: 45679 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f11866f255744d2fb31b4a6c2bcf6109" member_type: VOTER last_known_addr { host: "127.28.113.2" port: 44919 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fe1797652ab94dff90667ef356a6c2ae" member_type: VOTER last_known_addr { host: "127.28.113.3" port: 46345 } health_report { overall_health: UNKNOWN } } }
I20251215 21:02:56.205189 5370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:02:56.202316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35150 (local address 127.28.113.2:44919)
1215 21:02:56.202482 (+ 166us) server_negotiation.cc:207] Beginning negotiation
1215 21:02:56.202486 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1215 21:02:56.202501 (+ 15us) server_negotiation.cc:408] Connection header received
1215 21:02:56.202556 (+ 55us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1215 21:02:56.202559 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1215 21:02:56.202615 (+ 56us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1215 21:02:56.202718 (+ 103us) server_negotiation.cc:227] Negotiated authn=TOKEN
1215 21:02:56.203177 (+ 459us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:02:56.203644 (+ 467us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1215 21:02:56.204571 (+ 927us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:02:56.204711 (+ 140us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:02:56.204751 (+ 40us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1215 21:02:56.204861 (+ 110us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1215 21:02:56.204926 (+ 65us) server_negotiation.cc:1036] Waiting for connection context
1215 21:02:56.205040 (+ 114us) server_negotiation.cc:300] Negotiation successful
1215 21:02:56.205081 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20251215 21:02:56.205237 5408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:02:56.202184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.113.2:44919 (local address 127.0.0.1:35150)
1215 21:02:56.202332 (+ 148us) negotiation.cc:107] Waiting for socket to connect
1215 21:02:56.202348 (+ 16us) client_negotiation.cc:174] Beginning negotiation
1215 21:02:56.202469 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1215 21:02:56.202764 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1215 21:02:56.202768 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
1215 21:02:56.202779 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
1215 21:02:56.203050 (+ 271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:02:56.203058 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:02:56.203752 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1215 21:02:56.203756 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1215 21:02:56.204427 (+ 671us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:02:56.204438 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:02:56.204559 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:02:56.204610 (+ 51us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
1215 21:02:56.204953 (+ 343us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
1215 21:02:56.204965 (+ 12us) client_negotiation.cc:769] Sending connection context
1215 21:02:56.205029 (+ 64us) client_negotiation.cc:240] Negotiation successful
1215 21:02:56.205103 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
I20251215 21:02:56.207698 5013 raft_consensus.cc:1275] T a50663f3fbe14634800d2e38c7413d61 P 38e72b9cde82428ea8acd83a5194f75d [term 1 FOLLOWER]: Refusing update from remote peer f11866f255744d2fb31b4a6c2bcf6109: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251215 21:02:56.207724 5297 raft_consensus.cc:1275] T a50663f3fbe14634800d2e38c7413d61 P fe1797652ab94dff90667ef356a6c2ae [term 1 FOLLOWER]: Refusing update from remote peer f11866f255744d2fb31b4a6c2bcf6109: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251215 21:02:56.208134 5379 consensus_queue.cc:1048] T a50663f3fbe14634800d2e38c7413d61 P f11866f255744d2fb31b4a6c2bcf6109 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe1797652ab94dff90667ef356a6c2ae" member_type: VOTER last_known_addr { host: "127.28.113.3" port: 46345 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251215 21:02:56.208276 5381 consensus_queue.cc:1048] T a50663f3fbe14634800d2e38c7413d61 P f11866f255744d2fb31b4a6c2bcf6109 [LEADER]: Connected to new peer: Peer: permanent_uuid: "38e72b9cde82428ea8acd83a5194f75d" member_type: VOTER last_known_addr { host: "127.28.113.1" port: 45679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251215 21:02:56.209622 5378 tablet_replica.cc:442] T a50663f3fbe14634800d2e38c7413d61 P fe1797652ab94dff90667ef356a6c2ae: TxnStatusTablet state changed. Reason: New leader f11866f255744d2fb31b4a6c2bcf6109. Latest consensus state: current_term: 1 leader_uuid: "f11866f255744d2fb31b4a6c2bcf6109" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38e72b9cde82428ea8acd83a5194f75d" member_type: VOTER last_known_addr { host: "127.28.113.1" port: 45679 } } peers { permanent_uuid: "f11866f255744d2fb31b4a6c2bcf6109" member_type: VOTER last_known_addr { host: "127.28.113.2" port: 44919 } } peers { permanent_uuid: "fe1797652ab94dff90667ef356a6c2ae" member_type: VOTER last_known_addr { host: "127.28.113.3" port: 46345 } } }
I20251215 21:02:56.209731 5378 tablet_replica.cc:445] T a50663f3fbe14634800d2e38c7413d61 P fe1797652ab94dff90667ef356a6c2ae: This TxnStatusTablet replica's current role is: FOLLOWER
I20251215 21:02:56.209928 5377 tablet_replica.cc:442] T a50663f3fbe14634800d2e38c7413d61 P 38e72b9cde82428ea8acd83a5194f75d: TxnStatusTablet state changed. Reason: New leader f11866f255744d2fb31b4a6c2bcf6109. Latest consensus state: current_term: 1 leader_uuid: "f11866f255744d2fb31b4a6c2bcf6109" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38e72b9cde82428ea8acd83a5194f75d" member_type: VOTER last_known_addr { host: "127.28.113.1" port: 45679 } } peers { permanent_uuid: "f11866f255744d2fb31b4a6c2bcf6109" member_type: VOTER last_known_addr { host: "127.28.113.2" port: 44919 } } peers { permanent_uuid: "fe1797652ab94dff90667ef356a6c2ae" member_type: VOTER last_known_addr { host: "127.28.113.3" port: 46345 } } }
I20251215 21:02:56.210042 5377 tablet_replica.cc:445] T a50663f3fbe14634800d2e38c7413d61 P 38e72b9cde82428ea8acd83a5194f75d: This TxnStatusTablet replica's current role is: FOLLOWER
I20251215 21:02:56.211889 5381 tablet_replica.cc:442] T a50663f3fbe14634800d2e38c7413d61 P f11866f255744d2fb31b4a6c2bcf6109: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f11866f255744d2fb31b4a6c2bcf6109" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38e72b9cde82428ea8acd83a5194f75d" member_type: VOTER last_known_addr { host: "127.28.113.1" port: 45679 } } peers { permanent_uuid: "f11866f255744d2fb31b4a6c2bcf6109" member_type: VOTER last_known_addr { host: "127.28.113.2" port: 44919 } } peers { permanent_uuid: "fe1797652ab94dff90667ef356a6c2ae" member_type: VOTER last_known_addr { host: "127.28.113.3" port: 46345 } } }
I20251215 21:02:56.212016 5381 tablet_replica.cc:445] T a50663f3fbe14634800d2e38c7413d61 P f11866f255744d2fb31b4a6c2bcf6109: This TxnStatusTablet replica's current role is: LEADER
I20251215 21:02:56.212112 5381 tablet_replica.cc:442] T a50663f3fbe14634800d2e38c7413d61 P f11866f255744d2fb31b4a6c2bcf6109: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f11866f255744d2fb31b4a6c2bcf6109" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38e72b9cde82428ea8acd83a5194f75d" member_type: VOTER last_known_addr { host: "127.28.113.1" port: 45679 } } peers { permanent_uuid: "f11866f255744d2fb31b4a6c2bcf6109" member_type: VOTER last_known_addr { host: "127.28.113.2" port: 44919 } } peers { permanent_uuid: "fe1797652ab94dff90667ef356a6c2ae" member_type: VOTER last_known_addr { host: "127.28.113.3" port: 46345 } } }
I20251215 21:02:56.212158 5381 tablet_replica.cc:445] T a50663f3fbe14634800d2e38c7413d61 P f11866f255744d2fb31b4a6c2bcf6109: This TxnStatusTablet replica's current role is: LEADER
I20251215 21:02:56.212913 5378 tablet_replica.cc:442] T a50663f3fbe14634800d2e38c7413d61 P fe1797652ab94dff90667ef356a6c2ae: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f11866f255744d2fb31b4a6c2bcf6109" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38e72b9cde82428ea8acd83a5194f75d" member_type: VOTER last_known_addr { host: "127.28.113.1" port: 45679 } } peers { permanent_uuid: "f11866f255744d2fb31b4a6c2bcf6109" member_type: VOTER last_known_addr { host: "127.28.113.2" port: 44919 } } peers { permanent_uuid: "fe1797652ab94dff90667ef356a6c2ae" member_type: VOTER last_known_addr { host: "127.28.113.3" port: 46345 } } }
I20251215 21:02:56.213001 5378 tablet_replica.cc:445] T a50663f3fbe14634800d2e38c7413d61 P fe1797652ab94dff90667ef356a6c2ae: This TxnStatusTablet replica's current role is: FOLLOWER
I20251215 21:02:56.213958 5377 tablet_replica.cc:442] T a50663f3fbe14634800d2e38c7413d61 P 38e72b9cde82428ea8acd83a5194f75d: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f11866f255744d2fb31b4a6c2bcf6109" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38e72b9cde82428ea8acd83a5194f75d" member_type: VOTER last_known_addr { host: "127.28.113.1" port: 45679 } } peers { permanent_uuid: "f11866f255744d2fb31b4a6c2bcf6109" member_type: VOTER last_known_addr { host: "127.28.113.2" port: 44919 } } peers { permanent_uuid: "fe1797652ab94dff90667ef356a6c2ae" member_type: VOTER last_known_addr { host: "127.28.113.3" port: 46345 } } }
I20251215 21:02:56.214051 5377 tablet_replica.cc:445] T a50663f3fbe14634800d2e38c7413d61 P 38e72b9cde82428ea8acd83a5194f75d: This TxnStatusTablet replica's current role is: FOLLOWER
I20251215 21:02:56.214869 5385 mvcc.cc:204] Tried to move back new op lower bound from 7232850232142450688 to 7232850231979606016. Current Snapshot: MvccSnapshot[applied={T|T < 7232850232142450688 or (T in {7232850232142450688})}]
I20251215 21:02:56.221413 5369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:02:56.217937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41350 (local address 127.28.113.1:45679)
1215 21:02:56.218046 (+ 109us) server_negotiation.cc:207] Beginning negotiation
1215 21:02:56.218062 (+ 16us) server_negotiation.cc:400] Waiting for connection header
1215 21:02:56.218075 (+ 13us) server_negotiation.cc:408] Connection header received
1215 21:02:56.218151 (+ 76us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1215 21:02:56.218154 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1215 21:02:56.218218 (+ 64us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1215 21:02:56.218308 (+ 90us) server_negotiation.cc:227] Negotiated authn=TOKEN
1215 21:02:56.218796 (+ 488us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:02:56.219460 (+ 664us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1215 21:02:56.220373 (+ 913us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:02:56.220547 (+ 174us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:02:56.220599 (+ 52us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1215 21:02:56.221040 (+ 441us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1215 21:02:56.221128 (+ 88us) server_negotiation.cc:1036] Waiting for connection context
1215 21:02:56.221247 (+ 119us) server_negotiation.cc:300] Negotiation successful
1215 21:02:56.221295 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20251215 21:02:56.230989 5370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:02:56.228134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.113.1:48519 (local address 127.28.113.2:44919)
1215 21:02:56.228311 (+ 177us) server_negotiation.cc:207] Beginning negotiation
1215 21:02:56.228315 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1215 21:02:56.228411 (+ 96us) server_negotiation.cc:408] Connection header received
1215 21:02:56.228541 (+ 130us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1215 21:02:56.228543 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1215 21:02:56.228587 (+ 44us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1215 21:02:56.228650 (+ 63us) server_negotiation.cc:227] Negotiated authn=TOKEN
1215 21:02:56.229088 (+ 438us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:02:56.229538 (+ 450us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1215 21:02:56.230443 (+ 905us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:02:56.230579 (+ 136us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:02:56.230618 (+ 39us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1215 21:02:56.230689 (+ 71us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1215 21:02:56.230754 (+ 65us) server_negotiation.cc:1036] Waiting for connection context
1215 21:02:56.230844 (+ 90us) server_negotiation.cc:300] Negotiation successful
1215 21:02:56.230884 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105}
I20251215 21:02:56.231055 5421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:02:56.228047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.113.2:44919 (local address 127.28.113.1:48519)
1215 21:02:56.228329 (+ 282us) negotiation.cc:107] Waiting for socket to connect
1215 21:02:56.228345 (+ 16us) client_negotiation.cc:174] Beginning negotiation
1215 21:02:56.228456 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1215 21:02:56.228670 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1215 21:02:56.228673 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1215 21:02:56.228680 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
1215 21:02:56.228982 (+ 302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:02:56.228990 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:02:56.229625 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1215 21:02:56.229628 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1215 21:02:56.230307 (+ 679us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:02:56.230317 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:02:56.230444 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:02:56.230487 (+ 43us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
1215 21:02:56.230769 (+ 282us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
1215 21:02:56.230774 (+ 5us) client_negotiation.cc:769] Sending connection context
1215 21:02:56.230834 (+ 60us) client_negotiation.cc:240] Negotiation successful
1215 21:02:56.230899 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":84,"threads_started":1}
I20251215 21:02:56.238746 5297 raft_consensus.cc:1275] T fa857d00e2dd4965b89aa2213391e843 P fe1797652ab94dff90667ef356a6c2ae [term 1 FOLLOWER]: Refusing update from remote peer 38e72b9cde82428ea8acd83a5194f75d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251215 21:02:56.238950 5154 raft_consensus.cc:1275] T fa857d00e2dd4965b89aa2213391e843 P f11866f255744d2fb31b4a6c2bcf6109 [term 1 FOLLOWER]: Refusing update from remote peer 38e72b9cde82428ea8acd83a5194f75d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251215 21:02:56.239192 5377 consensus_queue.cc:1048] T fa857d00e2dd4965b89aa2213391e843 P 38e72b9cde82428ea8acd83a5194f75d [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe1797652ab94dff90667ef356a6c2ae" member_type: VOTER last_known_addr { host: "127.28.113.3" port: 46345 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251215 21:02:56.239511 5377 consensus_queue.cc:1048] T fa857d00e2dd4965b89aa2213391e843 P 38e72b9cde82428ea8acd83a5194f75d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f11866f255744d2fb31b4a6c2bcf6109" member_type: VOTER last_known_addr { host: "127.28.113.2" port: 44919 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251215 21:02:56.261291 5369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:02:56.257882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.113.2:51413 (local address 127.28.113.1:45679)
1215 21:02:56.257998 (+ 116us) server_negotiation.cc:207] Beginning negotiation
1215 21:02:56.258002 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1215 21:02:56.258316 (+ 314us) server_negotiation.cc:408] Connection header received
1215 21:02:56.258412 (+ 96us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1215 21:02:56.258414 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1215 21:02:56.258459 (+ 45us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1215 21:02:56.258509 (+ 50us) server_negotiation.cc:227] Negotiated authn=TOKEN
1215 21:02:56.258954 (+ 445us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:02:56.259555 (+ 601us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1215 21:02:56.260760 (+ 1205us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:02:56.260895 (+ 135us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:02:56.260933 (+ 38us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1215 21:02:56.260992 (+ 59us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1215 21:02:56.261051 (+ 59us) server_negotiation.cc:1036] Waiting for connection context
1215 21:02:56.261148 (+ 97us) server_negotiation.cc:300] Negotiation successful
1215 21:02:56.261189 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20251215 21:02:56.261353 5426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:02:56.257821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.113.1:45679 (local address 127.28.113.2:51413)
1215 21:02:56.258215 (+ 394us) negotiation.cc:107] Waiting for socket to connect
1215 21:02:56.258231 (+ 16us) client_negotiation.cc:174] Beginning negotiation
1215 21:02:56.258333 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1215 21:02:56.258540 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1215 21:02:56.258543 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1215 21:02:56.258551 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
1215 21:02:56.258818 (+ 267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:02:56.258824 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:02:56.259658 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1215 21:02:56.259663 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1215 21:02:56.260651 (+ 988us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:02:56.260661 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:02:56.260773 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:02:56.260817 (+ 44us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
1215 21:02:56.261074 (+ 257us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
1215 21:02:56.261080 (+ 6us) client_negotiation.cc:769] Sending connection context
1215 21:02:56.261140 (+ 60us) client_negotiation.cc:240] Negotiation successful
1215 21:02:56.261213 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":105,"threads_started":1}
I20251215 21:02:56.282142 4852 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34158:
table { table_name: "test-table" } modify_external_catalogs: true
I20251215 21:02:56.282286 4852 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34158:
table { table_name: "test-table" } modify_external_catalogs: true
I20251215 21:02:56.284277 4852 catalog_manager.cc:5941] T 00000000000000000000000000000000 P 4fc227cc99184e17b315174b90a73b5c: Sending DeleteTablet for 3 replicas of tablet fa857d00e2dd4965b89aa2213391e843
I20251215 21:02:56.284787 4993 tablet_service.cc:1552] Processing DeleteTablet for tablet fa857d00e2dd4965b89aa2213391e843 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-15 21:02:56 UTC) from {username='oryx', principal='oryx/127.28.113.62@KRBTEST.COM'} at 127.0.0.1:41348
I20251215 21:02:56.284987 5393 tablet_replica.cc:333] T fa857d00e2dd4965b89aa2213391e843 P 38e72b9cde82428ea8acd83a5194f75d: stopping tablet replica
I20251215 21:02:56.285120 5393 raft_consensus.cc:2243] T fa857d00e2dd4965b89aa2213391e843 P 38e72b9cde82428ea8acd83a5194f75d [term 1 LEADER]: Raft consensus shutting down.
I20251215 21:02:56.285122 5277 tablet_service.cc:1552] Processing DeleteTablet for tablet fa857d00e2dd4965b89aa2213391e843 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-15 21:02:56 UTC) from {username='oryx', principal='oryx/127.28.113.62@KRBTEST.COM'} at 127.0.0.1:50932
I20251215 21:02:56.285411 5395 tablet_replica.cc:333] T fa857d00e2dd4965b89aa2213391e843 P fe1797652ab94dff90667ef356a6c2ae: stopping tablet replica
I20251215 21:02:56.285410 5393 raft_consensus.cc:2272] T fa857d00e2dd4965b89aa2213391e843 P 38e72b9cde82428ea8acd83a5194f75d [term 1 FOLLOWER]: Raft consensus is shut down!
I20251215 21:02:56.285523 5395 raft_consensus.cc:2243] T fa857d00e2dd4965b89aa2213391e843 P fe1797652ab94dff90667ef356a6c2ae [term 1 FOLLOWER]: Raft consensus shutting down.
I20251215 21:02:56.285634 5395 raft_consensus.cc:2272] T fa857d00e2dd4965b89aa2213391e843 P fe1797652ab94dff90667ef356a6c2ae [term 1 FOLLOWER]: Raft consensus is shut down!
I20251215 21:02:56.285979 5135 tablet_service.cc:1552] Processing DeleteTablet for tablet fa857d00e2dd4965b89aa2213391e843 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-15 21:02:56 UTC) from {username='oryx', principal='oryx/127.28.113.62@KRBTEST.COM'} at 127.0.0.1:35118
I20251215 21:02:56.286161 5394 tablet_replica.cc:333] T fa857d00e2dd4965b89aa2213391e843 P f11866f255744d2fb31b4a6c2bcf6109: stopping tablet replica
I20251215 21:02:56.286221 5393 ts_tablet_manager.cc:1916] T fa857d00e2dd4965b89aa2213391e843 P 38e72b9cde82428ea8acd83a5194f75d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251215 21:02:56.286252 5394 raft_consensus.cc:2243] T fa857d00e2dd4965b89aa2213391e843 P f11866f255744d2fb31b4a6c2bcf6109 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251215 21:02:56.286347 5394 raft_consensus.cc:2272] T fa857d00e2dd4965b89aa2213391e843 P f11866f255744d2fb31b4a6c2bcf6109 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251215 21:02:56.286370 5395 ts_tablet_manager.cc:1916] T fa857d00e2dd4965b89aa2213391e843 P fe1797652ab94dff90667ef356a6c2ae: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251215 21:02:56.288875 5395 ts_tablet_manager.cc:1929] T fa857d00e2dd4965b89aa2213391e843 P fe1797652ab94dff90667ef356a6c2ae: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20251215 21:02:56.288952 5395 log.cc:1199] T fa857d00e2dd4965b89aa2213391e843 P fe1797652ab94dff90667ef356a6c2ae: Deleting WAL directory at /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765832506145157-29124-0/minicluster-data/ts-2/wal/wals/fa857d00e2dd4965b89aa2213391e843
I20251215 21:02:56.289124 5393 ts_tablet_manager.cc:1929] T fa857d00e2dd4965b89aa2213391e843 P 38e72b9cde82428ea8acd83a5194f75d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20251215 21:02:56.289188 5395 ts_tablet_manager.cc:1950] T fa857d00e2dd4965b89aa2213391e843 P fe1797652ab94dff90667ef356a6c2ae: Deleting consensus metadata
I20251215 21:02:56.289274 5393 log.cc:1199] T fa857d00e2dd4965b89aa2213391e843 P 38e72b9cde82428ea8acd83a5194f75d: Deleting WAL directory at /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765832506145157-29124-0/minicluster-data/ts-0/wal/wals/fa857d00e2dd4965b89aa2213391e843
I20251215 21:02:56.289526 5393 ts_tablet_manager.cc:1950] T fa857d00e2dd4965b89aa2213391e843 P 38e72b9cde82428ea8acd83a5194f75d: Deleting consensus metadata
I20251215 21:02:56.289896 4840 catalog_manager.cc:4985] TS fe1797652ab94dff90667ef356a6c2ae (127.28.113.3:46345): tablet fa857d00e2dd4965b89aa2213391e843 (table test-table [id=67cd9b8f91744369a13c2d39ba15caa0]) successfully deleted
I20251215 21:02:56.290357 4839 catalog_manager.cc:4985] TS 38e72b9cde82428ea8acd83a5194f75d (127.28.113.1:45679): tablet fa857d00e2dd4965b89aa2213391e843 (table test-table [id=67cd9b8f91744369a13c2d39ba15caa0]) successfully deleted
I20251215 21:02:56.296716 5394 ts_tablet_manager.cc:1916] T fa857d00e2dd4965b89aa2213391e843 P f11866f255744d2fb31b4a6c2bcf6109: Deleting tablet data with delete state TABLET_DATA_DELETED
Dec 15 21:02:56 dist-test-slave-bjsf krb5kdc[4806](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765832576, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
I20251215 21:02:56.299453 5394 ts_tablet_manager.cc:1929] T fa857d00e2dd4965b89aa2213391e843 P f11866f255744d2fb31b4a6c2bcf6109: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20251215 21:02:56.299644 5394 log.cc:1199] T fa857d00e2dd4965b89aa2213391e843 P f11866f255744d2fb31b4a6c2bcf6109: Deleting WAL directory at /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765832506145157-29124-0/minicluster-data/ts-1/wal/wals/fa857d00e2dd4965b89aa2213391e843
I20251215 21:02:56.299937 5394 ts_tablet_manager.cc:1950] T fa857d00e2dd4965b89aa2213391e843 P f11866f255744d2fb31b4a6c2bcf6109: Deleting consensus metadata
I20251215 21:02:56.301033 4840 catalog_manager.cc:4985] TS f11866f255744d2fb31b4a6c2bcf6109 (127.28.113.2:44919): tablet fa857d00e2dd4965b89aa2213391e843 (table test-table [id=67cd9b8f91744369a13c2d39ba15caa0]) successfully deleted
Dec 15 21:02:56 dist-test-slave-bjsf krb5kdc[4806](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.28.113.62@KRBTEST.COM, Server not found in Kerberos database
Dec 15 21:02:56 dist-test-slave-bjsf krb5kdc[4806](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.28.113.62@KRBTEST.COM, Server not found in Kerberos database
W20251215 21:02:56.309742 4903 negotiation.cc:337] Failed RPC negotiation. Trace:
1215 21:02:56.304495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34168 (local address 127.28.113.62:38827)
1215 21:02:56.304650 (+ 155us) server_negotiation.cc:207] Beginning negotiation
1215 21:02:56.304656 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1215 21:02:56.304830 (+ 174us) server_negotiation.cc:408] Connection header received
1215 21:02:56.304945 (+ 115us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1215 21:02:56.304949 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1215 21:02:56.304997 (+ 48us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1215 21:02:56.305062 (+ 65us) server_negotiation.cc:227] Negotiated authn=SASL
1215 21:02:56.305824 (+ 762us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:02:56.306326 (+ 502us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1215 21:02:56.306941 (+ 615us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:02:56.307113 (+ 172us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:02:56.309627 (+ 2514us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34168: BlockingRecv error: recv got EOF from 127.0.0.1:34168 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Dec 15 21:02:56 dist-test-slave-bjsf krb5kdc[4806](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765832576, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.28.113.62@KRBTEST.COM
I20251215 21:02:56.318734 4903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:02:56.311345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34176 (local address 127.28.113.62:38827)
1215 21:02:56.311457 (+ 112us) server_negotiation.cc:207] Beginning negotiation
1215 21:02:56.311461 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1215 21:02:56.311772 (+ 311us) server_negotiation.cc:408] Connection header received
1215 21:02:56.312073 (+ 301us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1215 21:02:56.312077 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1215 21:02:56.312125 (+ 48us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1215 21:02:56.312190 (+ 65us) server_negotiation.cc:227] Negotiated authn=SASL
1215 21:02:56.312831 (+ 641us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:02:56.313308 (+ 477us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1215 21:02:56.313896 (+ 588us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:02:56.314081 (+ 185us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:02:56.316061 (+ 1980us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1215 21:02:56.316080 (+ 19us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1215 21:02:56.316082 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1215 21:02:56.316107 (+ 25us) server_negotiation.cc:923] calling sasl_server_start()
1215 21:02:56.317405 (+ 1298us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1215 21:02:56.317743 (+ 338us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1215 21:02:56.317747 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1215 21:02:56.317748 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1215 21:02:56.317788 (+ 40us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1215 21:02:56.318010 (+ 222us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1215 21:02:56.318013 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1215 21:02:56.318015 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1215 21:02:56.318160 (+ 145us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1215 21:02:56.318274 (+ 114us) server_negotiation.cc:1036] Waiting for connection context
1215 21:02:56.318467 (+ 193us) server_negotiation.cc:300] Negotiation successful
1215 21:02:56.318573 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20251215 21:02:56.321087 4852 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34176:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251215 21:02:56.321532 4852 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251215 21:02:56.326253 4993 tablet_service.cc:1505] Processing CreateTablet for tablet a76d075cb870424094e1ee3129135a70 (DEFAULT_TABLE table=test-table [id=9c8290cb2de94ef381e6fb70fd5af287]), partition=RANGE (key) PARTITION UNBOUNDED
I20251215 21:02:56.326563 4993 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a76d075cb870424094e1ee3129135a70. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 21:02:56.326851 5135 tablet_service.cc:1505] Processing CreateTablet for tablet a76d075cb870424094e1ee3129135a70 (DEFAULT_TABLE table=test-table [id=9c8290cb2de94ef381e6fb70fd5af287]), partition=RANGE (key) PARTITION UNBOUNDED
I20251215 21:02:56.327090 5135 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a76d075cb870424094e1ee3129135a70. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 21:02:56.328226 5277 tablet_service.cc:1505] Processing CreateTablet for tablet a76d075cb870424094e1ee3129135a70 (DEFAULT_TABLE table=test-table [id=9c8290cb2de94ef381e6fb70fd5af287]), partition=RANGE (key) PARTITION UNBOUNDED
I20251215 21:02:56.328470 5277 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a76d075cb870424094e1ee3129135a70. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 21:02:56.328850 5371 tablet_bootstrap.cc:492] T a76d075cb870424094e1ee3129135a70 P 38e72b9cde82428ea8acd83a5194f75d: Bootstrap starting.
I20251215 21:02:56.329766 5371 tablet_bootstrap.cc:654] T a76d075cb870424094e1ee3129135a70 P 38e72b9cde82428ea8acd83a5194f75d: Neither blocks nor log segments found. Creating new log.
I20251215 21:02:56.330559 5372 tablet_bootstrap.cc:492] T a76d075cb870424094e1ee3129135a70 P fe1797652ab94dff90667ef356a6c2ae: Bootstrap starting.
I20251215 21:02:56.331197 5371 tablet_bootstrap.cc:492] T a76d075cb870424094e1ee3129135a70 P 38e72b9cde82428ea8acd83a5194f75d: No bootstrap required, opened a new log
I20251215 21:02:56.331270 5371 ts_tablet_manager.cc:1403] T a76d075cb870424094e1ee3129135a70 P 38e72b9cde82428ea8acd83a5194f75d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20251215 21:02:56.331462 5372 tablet_bootstrap.cc:654] T a76d075cb870424094e1ee3129135a70 P fe1797652ab94dff90667ef356a6c2ae: Neither blocks nor log segments found. Creating new log.
I20251215 21:02:56.331715 5371 raft_consensus.cc:359] T a76d075cb870424094e1ee3129135a70 P 38e72b9cde82428ea8acd83a5194f75d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38e72b9cde82428ea8acd83a5194f75d" member_type: VOTER last_known_addr { host: "127.28.113.1" port: 45679 } } peers { permanent_uuid: "f11866f255744d2fb31b4a6c2bcf6109" member_type: VOTER last_known_addr { host: "127.28.113.2" port: 44919 } } peers { permanent_uuid: "fe1797652ab94dff90667ef356a6c2ae" member_type: VOTER last_known_addr { host: "127.28.113.3" port: 46345 } }
I20251215 21:02:56.331813 5371 raft_consensus.cc:385] T a76d075cb870424094e1ee3129135a70 P 38e72b9cde82428ea8acd83a5194f75d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251215 21:02:56.331842 5371 raft_consensus.cc:740] T a76d075cb870424094e1ee3129135a70 P 38e72b9cde82428ea8acd83a5194f75d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 38e72b9cde82428ea8acd83a5194f75d, State: Initialized, Role: FOLLOWER
I20251215 21:02:56.331974 5371 consensus_queue.cc:260] T a76d075cb870424094e1ee3129135a70 P 38e72b9cde82428ea8acd83a5194f75d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38e72b9cde82428ea8acd83a5194f75d" member_type: VOTER last_known_addr { host: "127.28.113.1" port: 45679 } } peers { permanent_uuid: "f11866f255744d2fb31b4a6c2bcf6109" member_type: VOTER last_known_addr { host: "127.28.113.2" port: 44919 } } peers { permanent_uuid: "fe1797652ab94dff90667ef356a6c2ae" member_type: VOTER last_known_addr { host: "127.28.113.3" port: 46345 } }
I20251215 21:02:56.332237 5371 ts_tablet_manager.cc:1434] T a76d075cb870424094e1ee3129135a70 P 38e72b9cde82428ea8acd83a5194f75d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20251215 21:02:56.333796 5373 tablet_bootstrap.cc:492] T a76d075cb870424094e1ee3129135a70 P f11866f255744d2fb31b4a6c2bcf6109: Bootstrap starting.
I20251215 21:02:56.334681 5373 tablet_bootstrap.cc:654] T a76d075cb870424094e1ee3129135a70 P f11866f255744d2fb31b4a6c2bcf6109: Neither blocks nor log segments found. Creating new log.
I20251215 21:02:56.335654 5372 tablet_bootstrap.cc:492] T a76d075cb870424094e1ee3129135a70 P fe1797652ab94dff90667ef356a6c2ae: No bootstrap required, opened a new log
I20251215 21:02:56.335738 5372 ts_tablet_manager.cc:1403] T a76d075cb870424094e1ee3129135a70 P fe1797652ab94dff90667ef356a6c2ae: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20251215 21:02:56.336258 5372 raft_consensus.cc:359] T a76d075cb870424094e1ee3129135a70 P fe1797652ab94dff90667ef356a6c2ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38e72b9cde82428ea8acd83a5194f75d" member_type: VOTER last_known_addr { host: "127.28.113.1" port: 45679 } } peers { permanent_uuid: "f11866f255744d2fb31b4a6c2bcf6109" member_type: VOTER last_known_addr { host: "127.28.113.2" port: 44919 } } peers { permanent_uuid: "fe1797652ab94dff90667ef356a6c2ae" member_type: VOTER last_known_addr { host: "127.28.113.3" port: 46345 } }
I20251215 21:02:56.336366 5372 raft_consensus.cc:385] T a76d075cb870424094e1ee3129135a70 P fe1797652ab94dff90667ef356a6c2ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251215 21:02:56.336401 5372 raft_consensus.cc:740] T a76d075cb870424094e1ee3129135a70 P fe1797652ab94dff90667ef356a6c2ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe1797652ab94dff90667ef356a6c2ae, State: Initialized, Role: FOLLOWER
I20251215 21:02:56.336521 5372 consensus_queue.cc:260] T a76d075cb870424094e1ee3129135a70 P fe1797652ab94dff90667ef356a6c2ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38e72b9cde82428ea8acd83a5194f75d" member_type: VOTER last_known_addr { host: "127.28.113.1" port: 45679 } } peers { permanent_uuid: "f11866f255744d2fb31b4a6c2bcf6109" member_type: VOTER last_known_addr { host: "127.28.113.2" port: 44919 } } peers { permanent_uuid: "fe1797652ab94dff90667ef356a6c2ae" member_type: VOTER last_known_addr { host: "127.28.113.3" port: 46345 } }
I20251215 21:02:56.336684 5373 tablet_bootstrap.cc:492] T a76d075cb870424094e1ee3129135a70 P f11866f255744d2fb31b4a6c2bcf6109: No bootstrap required, opened a new log
I20251215 21:02:56.336756 5372 ts_tablet_manager.cc:1434] T a76d075cb870424094e1ee3129135a70 P fe1797652ab94dff90667ef356a6c2ae: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20251215 21:02:56.336767 5373 ts_tablet_manager.cc:1403] T a76d075cb870424094e1ee3129135a70 P f11866f255744d2fb31b4a6c2bcf6109: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20251215 21:02:56.337169 5373 raft_consensus.cc:359] T a76d075cb870424094e1ee3129135a70 P f11866f255744d2fb31b4a6c2bcf6109 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38e72b9cde82428ea8acd83a5194f75d" member_type: VOTER last_known_addr { host: "127.28.113.1" port: 45679 } } peers { permanent_uuid: "f11866f255744d2fb31b4a6c2bcf6109" member_type: VOTER last_known_addr { host: "127.28.113.2" port: 44919 } } peers { permanent_uuid: "fe1797652ab94dff90667ef356a6c2ae" member_type: VOTER last_known_addr { host: "127.28.113.3" port: 46345 } }
I20251215 21:02:56.337275 5373 raft_consensus.cc:385] T a76d075cb870424094e1ee3129135a70 P f11866f255744d2fb31b4a6c2bcf6109 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251215 21:02:56.337306 5373 raft_consensus.cc:740] T a76d075cb870424094e1ee3129135a70 P f11866f255744d2fb31b4a6c2bcf6109 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f11866f255744d2fb31b4a6c2bcf6109, State: Initialized, Role: FOLLOWER
I20251215 21:02:56.337427 5373 consensus_queue.cc:260] T a76d075cb870424094e1ee3129135a70 P f11866f255744d2fb31b4a6c2bcf6109 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38e72b9cde82428ea8acd83a5194f75d" member_type: VOTER last_known_addr { host: "127.28.113.1" port: 45679 } } peers { permanent_uuid: "f11866f255744d2fb31b4a6c2bcf6109" member_type: VOTER last_known_addr { host: "127.28.113.2" port: 44919 } } peers { permanent_uuid: "fe1797652ab94dff90667ef356a6c2ae" member_type: VOTER last_known_addr { host: "127.28.113.3" port: 46345 } }
I20251215 21:02:56.337661 5373 ts_tablet_manager.cc:1434] T a76d075cb870424094e1ee3129135a70 P f11866f255744d2fb31b4a6c2bcf6109: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20251215 21:02:56.338580 5418 raft_consensus.cc:493] T a76d075cb870424094e1ee3129135a70 P f11866f255744d2fb31b4a6c2bcf6109 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 21:02:56.338657 5418 raft_consensus.cc:515] T a76d075cb870424094e1ee3129135a70 P f11866f255744d2fb31b4a6c2bcf6109 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38e72b9cde82428ea8acd83a5194f75d" member_type: VOTER last_known_addr { host: "127.28.113.1" port: 45679 } } peers { permanent_uuid: "f11866f255744d2fb31b4a6c2bcf6109" member_type: VOTER last_known_addr { host: "127.28.113.2" port: 44919 } } peers { permanent_uuid: "fe1797652ab94dff90667ef356a6c2ae" member_type: VOTER last_known_addr { host: "127.28.113.3" port: 46345 } }
I20251215 21:02:56.339031 5418 leader_election.cc:290] T a76d075cb870424094e1ee3129135a70 P f11866f255744d2fb31b4a6c2bcf6109 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 38e72b9cde82428ea8acd83a5194f75d (127.28.113.1:45679), fe1797652ab94dff90667ef356a6c2ae (127.28.113.3:46345)
I20251215 21:02:56.339468 5297 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a76d075cb870424094e1ee3129135a70" candidate_uuid: "f11866f255744d2fb31b4a6c2bcf6109" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe1797652ab94dff90667ef356a6c2ae" is_pre_election: true
I20251215 21:02:56.339569 5297 raft_consensus.cc:2468] T a76d075cb870424094e1ee3129135a70 P fe1797652ab94dff90667ef356a6c2ae [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f11866f255744d2fb31b4a6c2bcf6109 in term 0.
I20251215 21:02:56.339620 5013 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a76d075cb870424094e1ee3129135a70" candidate_uuid: "f11866f255744d2fb31b4a6c2bcf6109" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "38e72b9cde82428ea8acd83a5194f75d" is_pre_election: true
I20251215 21:02:56.339738 5013 raft_consensus.cc:2468] T a76d075cb870424094e1ee3129135a70 P 38e72b9cde82428ea8acd83a5194f75d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f11866f255744d2fb31b4a6c2bcf6109 in term 0.
I20251215 21:02:56.339993 5088 leader_election.cc:304] T a76d075cb870424094e1ee3129135a70 P f11866f255744d2fb31b4a6c2bcf6109 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f11866f255744d2fb31b4a6c2bcf6109, fe1797652ab94dff90667ef356a6c2ae; no voters:
I20251215 21:02:56.340297 5418 raft_consensus.cc:2804] T a76d075cb870424094e1ee3129135a70 P f11866f255744d2fb31b4a6c2bcf6109 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251215 21:02:56.340348 5418 raft_consensus.cc:493] T a76d075cb870424094e1ee3129135a70 P f11866f255744d2fb31b4a6c2bcf6109 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 21:02:56.340389 5418 raft_consensus.cc:3060] T a76d075cb870424094e1ee3129135a70 P f11866f255744d2fb31b4a6c2bcf6109 [term 0 FOLLOWER]: Advancing to term 1
I20251215 21:02:56.341168 5418 raft_consensus.cc:515] T a76d075cb870424094e1ee3129135a70 P f11866f255744d2fb31b4a6c2bcf6109 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38e72b9cde82428ea8acd83a5194f75d" member_type: VOTER last_known_addr { host: "127.28.113.1" port: 45679 } } peers { permanent_uuid: "f11866f255744d2fb31b4a6c2bcf6109" member_type: VOTER last_known_addr { host: "127.28.113.2" port: 44919 } } peers { permanent_uuid: "fe1797652ab94dff90667ef356a6c2ae" member_type: VOTER last_known_addr { host: "127.28.113.3" port: 46345 } }
I20251215 21:02:56.341461 5418 leader_election.cc:290] T a76d075cb870424094e1ee3129135a70 P f11866f255744d2fb31b4a6c2bcf6109 [CANDIDATE]: Term 1 election: Requested vote from peers 38e72b9cde82428ea8acd83a5194f75d (127.28.113.1:45679), fe1797652ab94dff90667ef356a6c2ae (127.28.113.3:46345)
I20251215 21:02:56.341735 5013 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a76d075cb870424094e1ee3129135a70" candidate_uuid: "f11866f255744d2fb31b4a6c2bcf6109" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "38e72b9cde82428ea8acd83a5194f75d"
I20251215 21:02:56.341848 5013 raft_consensus.cc:3060] T a76d075cb870424094e1ee3129135a70 P 38e72b9cde82428ea8acd83a5194f75d [term 0 FOLLOWER]: Advancing to term 1
I20251215 21:02:56.341878 5297 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a76d075cb870424094e1ee3129135a70" candidate_uuid: "f11866f255744d2fb31b4a6c2bcf6109" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe1797652ab94dff90667ef356a6c2ae"
I20251215 21:02:56.341971 5297 raft_consensus.cc:3060] T a76d075cb870424094e1ee3129135a70 P fe1797652ab94dff90667ef356a6c2ae [term 0 FOLLOWER]: Advancing to term 1
I20251215 21:02:56.343046 5013 raft_consensus.cc:2468] T a76d075cb870424094e1ee3129135a70 P 38e72b9cde82428ea8acd83a5194f75d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f11866f255744d2fb31b4a6c2bcf6109 in term 1.
I20251215 21:02:56.343082 5297 raft_consensus.cc:2468] T a76d075cb870424094e1ee3129135a70 P fe1797652ab94dff90667ef356a6c2ae [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f11866f255744d2fb31b4a6c2bcf6109 in term 1.
I20251215 21:02:56.343333 5088 leader_election.cc:304] T a76d075cb870424094e1ee3129135a70 P f11866f255744d2fb31b4a6c2bcf6109 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f11866f255744d2fb31b4a6c2bcf6109, fe1797652ab94dff90667ef356a6c2ae; no voters:
I20251215 21:02:56.343530 5418 raft_consensus.cc:2804] T a76d075cb870424094e1ee3129135a70 P f11866f255744d2fb31b4a6c2bcf6109 [term 1 FOLLOWER]: Leader election won for term 1
I20251215 21:02:56.343617 5418 raft_consensus.cc:697] T a76d075cb870424094e1ee3129135a70 P f11866f255744d2fb31b4a6c2bcf6109 [term 1 LEADER]: Becoming Leader. State: Replica: f11866f255744d2fb31b4a6c2bcf6109, State: Running, Role: LEADER
I20251215 21:02:56.343745 5418 consensus_queue.cc:237] T a76d075cb870424094e1ee3129135a70 P f11866f255744d2fb31b4a6c2bcf6109 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38e72b9cde82428ea8acd83a5194f75d" member_type: VOTER last_known_addr { host: "127.28.113.1" port: 45679 } } peers { permanent_uuid: "f11866f255744d2fb31b4a6c2bcf6109" member_type: VOTER last_known_addr { host: "127.28.113.2" port: 44919 } } peers { permanent_uuid: "fe1797652ab94dff90667ef356a6c2ae" member_type: VOTER last_known_addr { host: "127.28.113.3" port: 46345 } }
I20251215 21:02:56.345348 4854 catalog_manager.cc:5654] T a76d075cb870424094e1ee3129135a70 P f11866f255744d2fb31b4a6c2bcf6109 reported cstate change: term changed from 0 to 1, leader changed from <none> to f11866f255744d2fb31b4a6c2bcf6109 (127.28.113.2). New cstate: current_term: 1 leader_uuid: "f11866f255744d2fb31b4a6c2bcf6109" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38e72b9cde82428ea8acd83a5194f75d" member_type: VOTER last_known_addr { host: "127.28.113.1" port: 45679 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f11866f255744d2fb31b4a6c2bcf6109" member_type: VOTER last_known_addr { host: "127.28.113.2" port: 44919 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fe1797652ab94dff90667ef356a6c2ae" member_type: VOTER last_known_addr { host: "127.28.113.3" port: 46345 } health_report { overall_health: UNKNOWN } } }
I20251215 21:02:56.353008 5370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:02:56.350627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35158 (local address 127.28.113.2:44919)
1215 21:02:56.350734 (+ 107us) server_negotiation.cc:207] Beginning negotiation
1215 21:02:56.350738 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1215 21:02:56.350750 (+ 12us) server_negotiation.cc:408] Connection header received
1215 21:02:56.350863 (+ 113us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1215 21:02:56.350866 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1215 21:02:56.350919 (+ 53us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1215 21:02:56.350981 (+ 62us) server_negotiation.cc:227] Negotiated authn=TOKEN
1215 21:02:56.351308 (+ 327us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:02:56.351759 (+ 451us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1215 21:02:56.352416 (+ 657us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:02:56.352576 (+ 160us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:02:56.352615 (+ 39us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1215 21:02:56.352678 (+ 63us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1215 21:02:56.352737 (+ 59us) server_negotiation.cc:1036] Waiting for connection context
1215 21:02:56.352841 (+ 104us) server_negotiation.cc:300] Negotiation successful
1215 21:02:56.352883 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20251215 21:02:56.354807 5297 raft_consensus.cc:1275] T a76d075cb870424094e1ee3129135a70 P fe1797652ab94dff90667ef356a6c2ae [term 1 FOLLOWER]: Refusing update from remote peer f11866f255744d2fb31b4a6c2bcf6109: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251215 21:02:56.355237 5379 consensus_queue.cc:1048] T a76d075cb870424094e1ee3129135a70 P f11866f255744d2fb31b4a6c2bcf6109 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe1797652ab94dff90667ef356a6c2ae" member_type: VOTER last_known_addr { host: "127.28.113.3" port: 46345 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251215 21:02:56.357070 5013 raft_consensus.cc:1275] T a76d075cb870424094e1ee3129135a70 P 38e72b9cde82428ea8acd83a5194f75d [term 1 FOLLOWER]: Refusing update from remote peer f11866f255744d2fb31b4a6c2bcf6109: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251215 21:02:56.357524 5418 consensus_queue.cc:1048] T a76d075cb870424094e1ee3129135a70 P f11866f255744d2fb31b4a6c2bcf6109 [LEADER]: Connected to new peer: Peer: permanent_uuid: "38e72b9cde82428ea8acd83a5194f75d" member_type: VOTER last_known_addr { host: "127.28.113.1" port: 45679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251215 21:02:56.365020 4852 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34176:
table { table_name: "test-table" } modify_external_catalogs: true
I20251215 21:02:56.365144 4852 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34176:
table { table_name: "test-table" } modify_external_catalogs: true
I20251215 21:02:56.368722 4852 catalog_manager.cc:5941] T 00000000000000000000000000000000 P 4fc227cc99184e17b315174b90a73b5c: Sending DeleteTablet for 3 replicas of tablet a76d075cb870424094e1ee3129135a70
I20251215 21:02:56.369438 4993 tablet_service.cc:1552] Processing DeleteTablet for tablet a76d075cb870424094e1ee3129135a70 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-15 21:02:56 UTC) from {username='oryx', principal='oryx/127.28.113.62@KRBTEST.COM'} at 127.0.0.1:41348
I20251215 21:02:56.369632 4992 tablet_service.cc:1552] Processing DeleteTablet for tablet a76d075cb870424094e1ee3129135a70 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-15 21:02:56 UTC) from {username='oryx', principal='oryx/127.28.113.62@KRBTEST.COM'} at 127.0.0.1:41348
I20251215 21:02:56.369727 5277 tablet_service.cc:1552] Processing DeleteTablet for tablet a76d075cb870424094e1ee3129135a70 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-15 21:02:56 UTC) from {username='oryx', principal='oryx/127.28.113.62@KRBTEST.COM'} at 127.0.0.1:50932
I20251215 21:02:56.369906 5395 tablet_replica.cc:333] T a76d075cb870424094e1ee3129135a70 P fe1797652ab94dff90667ef356a6c2ae: stopping tablet replica
I20251215 21:02:56.369913 5393 tablet_replica.cc:333] T a76d075cb870424094e1ee3129135a70 P 38e72b9cde82428ea8acd83a5194f75d: stopping tablet replica
I20251215 21:02:56.370008 5395 raft_consensus.cc:2243] T a76d075cb870424094e1ee3129135a70 P fe1797652ab94dff90667ef356a6c2ae [term 1 FOLLOWER]: Raft consensus shutting down.
I20251215 21:02:56.370008 5393 raft_consensus.cc:2243] T a76d075cb870424094e1ee3129135a70 P 38e72b9cde82428ea8acd83a5194f75d [term 1 FOLLOWER]: Raft consensus shutting down.
I20251215 21:02:56.370121 5393 raft_consensus.cc:2272] T a76d075cb870424094e1ee3129135a70 P 38e72b9cde82428ea8acd83a5194f75d [term 1 FOLLOWER]: Raft consensus is shut down!
I20251215 21:02:56.370123 5395 raft_consensus.cc:2272] T a76d075cb870424094e1ee3129135a70 P fe1797652ab94dff90667ef356a6c2ae [term 1 FOLLOWER]: Raft consensus is shut down!
I20251215 21:02:56.370266 5135 tablet_service.cc:1552] Processing DeleteTablet for tablet a76d075cb870424094e1ee3129135a70 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-15 21:02:56 UTC) from {username='oryx', principal='oryx/127.28.113.62@KRBTEST.COM'} at 127.0.0.1:35118
I20251215 21:02:56.370414 5394 tablet_replica.cc:333] T a76d075cb870424094e1ee3129135a70 P f11866f255744d2fb31b4a6c2bcf6109: stopping tablet replica
I20251215 21:02:56.370503 5394 raft_consensus.cc:2243] T a76d075cb870424094e1ee3129135a70 P f11866f255744d2fb31b4a6c2bcf6109 [term 1 LEADER]: Raft consensus shutting down.
I20251215 21:02:56.370688 5394 raft_consensus.cc:2272] T a76d075cb870424094e1ee3129135a70 P f11866f255744d2fb31b4a6c2bcf6109 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251215 21:02:56.371764 5395 ts_tablet_manager.cc:1916] T a76d075cb870424094e1ee3129135a70 P fe1797652ab94dff90667ef356a6c2ae: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251215 21:02:56.371742 4854 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34176:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251215 21:02:56.372200 4854 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251215 21:02:56.373790 5395 ts_tablet_manager.cc:1929] T a76d075cb870424094e1ee3129135a70 P fe1797652ab94dff90667ef356a6c2ae: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251215 21:02:56.373840 5395 log.cc:1199] T a76d075cb870424094e1ee3129135a70 P fe1797652ab94dff90667ef356a6c2ae: Deleting WAL directory at /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765832506145157-29124-0/minicluster-data/ts-2/wal/wals/a76d075cb870424094e1ee3129135a70
I20251215 21:02:56.373998 5395 ts_tablet_manager.cc:1950] T a76d075cb870424094e1ee3129135a70 P fe1797652ab94dff90667ef356a6c2ae: Deleting consensus metadata
I20251215 21:02:56.375358 4840 catalog_manager.cc:4985] TS fe1797652ab94dff90667ef356a6c2ae (127.28.113.3:46345): tablet a76d075cb870424094e1ee3129135a70 (table test-table [id=9c8290cb2de94ef381e6fb70fd5af287]) successfully deleted
I20251215 21:02:56.377558 5394 ts_tablet_manager.cc:1916] T a76d075cb870424094e1ee3129135a70 P f11866f255744d2fb31b4a6c2bcf6109: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251215 21:02:56.379467 5394 ts_tablet_manager.cc:1929] T a76d075cb870424094e1ee3129135a70 P f11866f255744d2fb31b4a6c2bcf6109: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251215 21:02:56.379532 5394 log.cc:1199] T a76d075cb870424094e1ee3129135a70 P f11866f255744d2fb31b4a6c2bcf6109: Deleting WAL directory at /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765832506145157-29124-0/minicluster-data/ts-1/wal/wals/a76d075cb870424094e1ee3129135a70
I20251215 21:02:56.379719 5394 ts_tablet_manager.cc:1950] T a76d075cb870424094e1ee3129135a70 P f11866f255744d2fb31b4a6c2bcf6109: Deleting consensus metadata
I20251215 21:02:56.381506 4840 catalog_manager.cc:4985] TS f11866f255744d2fb31b4a6c2bcf6109 (127.28.113.2:44919): tablet a76d075cb870424094e1ee3129135a70 (table test-table [id=9c8290cb2de94ef381e6fb70fd5af287]) successfully deleted
I20251215 21:02:56.381801 4992 tablet_service.cc:1505] Processing CreateTablet for tablet 072037ebcc654a85a5115664fed3cbbf (DEFAULT_TABLE table=test-table [id=e2512ff87ea54becb5d0fe3e92cb9c84]), partition=RANGE (key) PARTITION UNBOUNDED
I20251215 21:02:56.381960 5277 tablet_service.cc:1505] Processing CreateTablet for tablet 072037ebcc654a85a5115664fed3cbbf (DEFAULT_TABLE table=test-table [id=e2512ff87ea54becb5d0fe3e92cb9c84]), partition=RANGE (key) PARTITION UNBOUNDED
I20251215 21:02:56.382097 4992 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 072037ebcc654a85a5115664fed3cbbf. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 21:02:56.382196 5277 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 072037ebcc654a85a5115664fed3cbbf. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 21:02:56.384352 5135 tablet_service.cc:1505] Processing CreateTablet for tablet 072037ebcc654a85a5115664fed3cbbf (DEFAULT_TABLE table=test-table [id=e2512ff87ea54becb5d0fe3e92cb9c84]), partition=RANGE (key) PARTITION UNBOUNDED
I20251215 21:02:56.384557 5371 tablet_bootstrap.cc:492] T 072037ebcc654a85a5115664fed3cbbf P 38e72b9cde82428ea8acd83a5194f75d: Bootstrap starting.
I20251215 21:02:56.384578 5135 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 072037ebcc654a85a5115664fed3cbbf. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 21:02:56.385489 5371 tablet_bootstrap.cc:654] T 072037ebcc654a85a5115664fed3cbbf P 38e72b9cde82428ea8acd83a5194f75d: Neither blocks nor log segments found. Creating new log.
I20251215 21:02:56.386566 5373 tablet_bootstrap.cc:492] T 072037ebcc654a85a5115664fed3cbbf P f11866f255744d2fb31b4a6c2bcf6109: Bootstrap starting.
I20251215 21:02:56.387372 5373 tablet_bootstrap.cc:654] T 072037ebcc654a85a5115664fed3cbbf P f11866f255744d2fb31b4a6c2bcf6109: Neither blocks nor log segments found. Creating new log.
I20251215 21:02:56.388309 5372 tablet_bootstrap.cc:492] T 072037ebcc654a85a5115664fed3cbbf P fe1797652ab94dff90667ef356a6c2ae: Bootstrap starting.
I20251215 21:02:56.388844 5373 tablet_bootstrap.cc:492] T 072037ebcc654a85a5115664fed3cbbf P f11866f255744d2fb31b4a6c2bcf6109: No bootstrap required, opened a new log
I20251215 21:02:56.388914 5373 ts_tablet_manager.cc:1403] T 072037ebcc654a85a5115664fed3cbbf P f11866f255744d2fb31b4a6c2bcf6109: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20251215 21:02:56.389158 5372 tablet_bootstrap.cc:654] T 072037ebcc654a85a5115664fed3cbbf P fe1797652ab94dff90667ef356a6c2ae: Neither blocks nor log segments found. Creating new log.
I20251215 21:02:56.389312 5373 raft_consensus.cc:359] T 072037ebcc654a85a5115664fed3cbbf P f11866f255744d2fb31b4a6c2bcf6109 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38e72b9cde82428ea8acd83a5194f75d" member_type: VOTER last_known_addr { host: "127.28.113.1" port: 45679 } } peers { permanent_uuid: "f11866f255744d2fb31b4a6c2bcf6109" member_type: VOTER last_known_addr { host: "127.28.113.2" port: 44919 } } peers { permanent_uuid: "fe1797652ab94dff90667ef356a6c2ae" member_type: VOTER last_known_addr { host: "127.28.113.3" port: 46345 } }
I20251215 21:02:56.389413 5373 raft_consensus.cc:385] T 072037ebcc654a85a5115664fed3cbbf P f11866f255744d2fb31b4a6c2bcf6109 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251215 21:02:56.389441 5373 raft_consensus.cc:740] T 072037ebcc654a85a5115664fed3cbbf P f11866f255744d2fb31b4a6c2bcf6109 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f11866f255744d2fb31b4a6c2bcf6109, State: Initialized, Role: FOLLOWER
I20251215 21:02:56.389555 5373 consensus_queue.cc:260] T 072037ebcc654a85a5115664fed3cbbf P f11866f255744d2fb31b4a6c2bcf6109 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38e72b9cde82428ea8acd83a5194f75d" member_type: VOTER last_known_addr { host: "127.28.113.1" port: 45679 } } peers { permanent_uuid: "f11866f255744d2fb31b4a6c2bcf6109" member_type: VOTER last_known_addr { host: "127.28.113.2" port: 44919 } } peers { permanent_uuid: "fe1797652ab94dff90667ef356a6c2ae" member_type: VOTER last_known_addr { host: "127.28.113.3" port: 46345 } }
I20251215 21:02:56.389796 5373 ts_tablet_manager.cc:1434] T 072037ebcc654a85a5115664fed3cbbf P f11866f255744d2fb31b4a6c2bcf6109: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20251215 21:02:56.391652 5393 ts_tablet_manager.cc:1916] T a76d075cb870424094e1ee3129135a70 P 38e72b9cde82428ea8acd83a5194f75d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251215 21:02:56.392674 5371 tablet_bootstrap.cc:492] T 072037ebcc654a85a5115664fed3cbbf P 38e72b9cde82428ea8acd83a5194f75d: No bootstrap required, opened a new log
I20251215 21:02:56.392752 5371 ts_tablet_manager.cc:1403] T 072037ebcc654a85a5115664fed3cbbf P 38e72b9cde82428ea8acd83a5194f75d: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20251215 21:02:56.393183 5371 raft_consensus.cc:359] T 072037ebcc654a85a5115664fed3cbbf P 38e72b9cde82428ea8acd83a5194f75d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38e72b9cde82428ea8acd83a5194f75d" member_type: VOTER last_known_addr { host: "127.28.113.1" port: 45679 } } peers { permanent_uuid: "f11866f255744d2fb31b4a6c2bcf6109" member_type: VOTER last_known_addr { host: "127.28.113.2" port: 44919 } } peers { permanent_uuid: "fe1797652ab94dff90667ef356a6c2ae" member_type: VOTER last_known_addr { host: "127.28.113.3" port: 46345 } }
I20251215 21:02:56.393280 5371 raft_consensus.cc:385] T 072037ebcc654a85a5115664fed3cbbf P 38e72b9cde82428ea8acd83a5194f75d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251215 21:02:56.393311 5371 raft_consensus.cc:740] T 072037ebcc654a85a5115664fed3cbbf P 38e72b9cde82428ea8acd83a5194f75d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 38e72b9cde82428ea8acd83a5194f75d, State: Initialized, Role: FOLLOWER
I20251215 21:02:56.393420 5371 consensus_queue.cc:260] T 072037ebcc654a85a5115664fed3cbbf P 38e72b9cde82428ea8acd83a5194f75d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38e72b9cde82428ea8acd83a5194f75d" member_type: VOTER last_known_addr { host: "127.28.113.1" port: 45679 } } peers { permanent_uuid: "f11866f255744d2fb31b4a6c2bcf6109" member_type: VOTER last_known_addr { host: "127.28.113.2" port: 44919 } } peers { permanent_uuid: "fe1797652ab94dff90667ef356a6c2ae" member_type: VOTER last_known_addr { host: "127.28.113.3" port: 46345 } }
I20251215 21:02:56.393587 5393 ts_tablet_manager.cc:1929] T a76d075cb870424094e1ee3129135a70 P 38e72b9cde82428ea8acd83a5194f75d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251215 21:02:56.393639 5393 log.cc:1199] T a76d075cb870424094e1ee3129135a70 P 38e72b9cde82428ea8acd83a5194f75d: Deleting WAL directory at /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765832506145157-29124-0/minicluster-data/ts-0/wal/wals/a76d075cb870424094e1ee3129135a70
I20251215 21:02:56.393674 5371 ts_tablet_manager.cc:1434] T 072037ebcc654a85a5115664fed3cbbf P 38e72b9cde82428ea8acd83a5194f75d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20251215 21:02:56.393862 5393 ts_tablet_manager.cc:1950] T a76d075cb870424094e1ee3129135a70 P 38e72b9cde82428ea8acd83a5194f75d: Deleting consensus metadata
I20251215 21:02:56.394702 4839 catalog_manager.cc:4985] TS 38e72b9cde82428ea8acd83a5194f75d (127.28.113.1:45679): tablet a76d075cb870424094e1ee3129135a70 (table test-table [id=9c8290cb2de94ef381e6fb70fd5af287]) successfully deleted
W20251215 21:02:56.394959 4839 catalog_manager.cc:4952] TS 38e72b9cde82428ea8acd83a5194f75d (127.28.113.1:45679): delete failed for tablet a76d075cb870424094e1ee3129135a70 because the tablet was not found. No further retry: Not found: Tablet not found: a76d075cb870424094e1ee3129135a70
I20251215 21:02:56.396649 5372 tablet_bootstrap.cc:492] T 072037ebcc654a85a5115664fed3cbbf P fe1797652ab94dff90667ef356a6c2ae: No bootstrap required, opened a new log
I20251215 21:02:56.396741 5372 ts_tablet_manager.cc:1403] T 072037ebcc654a85a5115664fed3cbbf P fe1797652ab94dff90667ef356a6c2ae: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.000s
I20251215 21:02:56.397147 5372 raft_consensus.cc:359] T 072037ebcc654a85a5115664fed3cbbf P fe1797652ab94dff90667ef356a6c2ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38e72b9cde82428ea8acd83a5194f75d" member_type: VOTER last_known_addr { host: "127.28.113.1" port: 45679 } } peers { permanent_uuid: "f11866f255744d2fb31b4a6c2bcf6109" member_type: VOTER last_known_addr { host: "127.28.113.2" port: 44919 } } peers { permanent_uuid: "fe1797652ab94dff90667ef356a6c2ae" member_type: VOTER last_known_addr { host: "127.28.113.3" port: 46345 } }
I20251215 21:02:56.397240 5372 raft_consensus.cc:385] T 072037ebcc654a85a5115664fed3cbbf P fe1797652ab94dff90667ef356a6c2ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251215 21:02:56.397269 5372 raft_consensus.cc:740] T 072037ebcc654a85a5115664fed3cbbf P fe1797652ab94dff90667ef356a6c2ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe1797652ab94dff90667ef356a6c2ae, State: Initialized, Role: FOLLOWER
I20251215 21:02:56.397382 5372 consensus_queue.cc:260] T 072037ebcc654a85a5115664fed3cbbf P fe1797652ab94dff90667ef356a6c2ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38e72b9cde82428ea8acd83a5194f75d" member_type: VOTER last_known_addr { host: "127.28.113.1" port: 45679 } } peers { permanent_uuid: "f11866f255744d2fb31b4a6c2bcf6109" member_type: VOTER last_known_addr { host: "127.28.113.2" port: 44919 } } peers { permanent_uuid: "fe1797652ab94dff90667ef356a6c2ae" member_type: VOTER last_known_addr { host: "127.28.113.3" port: 46345 } }
I20251215 21:02:56.397590 5372 ts_tablet_manager.cc:1434] T 072037ebcc654a85a5115664fed3cbbf P fe1797652ab94dff90667ef356a6c2ae: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20251215 21:02:56.402228 5377 raft_consensus.cc:493] T 072037ebcc654a85a5115664fed3cbbf P 38e72b9cde82428ea8acd83a5194f75d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 21:02:56.402350 5377 raft_consensus.cc:515] T 072037ebcc654a85a5115664fed3cbbf P 38e72b9cde82428ea8acd83a5194f75d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38e72b9cde82428ea8acd83a5194f75d" member_type: VOTER last_known_addr { host: "127.28.113.1" port: 45679 } } peers { permanent_uuid: "f11866f255744d2fb31b4a6c2bcf6109" member_type: VOTER last_known_addr { host: "127.28.113.2" port: 44919 } } peers { permanent_uuid: "fe1797652ab94dff90667ef356a6c2ae" member_type: VOTER last_known_addr { host: "127.28.113.3" port: 46345 } }
I20251215 21:02:56.402768 5377 leader_election.cc:290] T 072037ebcc654a85a5115664fed3cbbf P 38e72b9cde82428ea8acd83a5194f75d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f11866f255744d2fb31b4a6c2bcf6109 (127.28.113.2:44919), fe1797652ab94dff90667ef356a6c2ae (127.28.113.3:46345)
I20251215 21:02:56.403177 5154 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "072037ebcc654a85a5115664fed3cbbf" candidate_uuid: "38e72b9cde82428ea8acd83a5194f75d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f11866f255744d2fb31b4a6c2bcf6109" is_pre_election: true
I20251215 21:02:56.403215 5297 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "072037ebcc654a85a5115664fed3cbbf" candidate_uuid: "38e72b9cde82428ea8acd83a5194f75d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe1797652ab94dff90667ef356a6c2ae" is_pre_election: true
I20251215 21:02:56.403316 5297 raft_consensus.cc:2468] T 072037ebcc654a85a5115664fed3cbbf P fe1797652ab94dff90667ef356a6c2ae [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38e72b9cde82428ea8acd83a5194f75d in term 0.
I20251215 21:02:56.403316 5154 raft_consensus.cc:2468] T 072037ebcc654a85a5115664fed3cbbf P f11866f255744d2fb31b4a6c2bcf6109 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38e72b9cde82428ea8acd83a5194f75d in term 0.
I20251215 21:02:56.403589 4946 leader_election.cc:304] T 072037ebcc654a85a5115664fed3cbbf P 38e72b9cde82428ea8acd83a5194f75d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38e72b9cde82428ea8acd83a5194f75d, fe1797652ab94dff90667ef356a6c2ae; no voters:
I20251215 21:02:56.403736 5377 raft_consensus.cc:2804] T 072037ebcc654a85a5115664fed3cbbf P 38e72b9cde82428ea8acd83a5194f75d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251215 21:02:56.403800 5377 raft_consensus.cc:493] T 072037ebcc654a85a5115664fed3cbbf P 38e72b9cde82428ea8acd83a5194f75d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 21:02:56.403838 5377 raft_consensus.cc:3060] T 072037ebcc654a85a5115664fed3cbbf P 38e72b9cde82428ea8acd83a5194f75d [term 0 FOLLOWER]: Advancing to term 1
I20251215 21:02:56.404697 5377 raft_consensus.cc:515] T 072037ebcc654a85a5115664fed3cbbf P 38e72b9cde82428ea8acd83a5194f75d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38e72b9cde82428ea8acd83a5194f75d" member_type: VOTER last_known_addr { host: "127.28.113.1" port: 45679 } } peers { permanent_uuid: "f11866f255744d2fb31b4a6c2bcf6109" member_type: VOTER last_known_addr { host: "127.28.113.2" port: 44919 } } peers { permanent_uuid: "fe1797652ab94dff90667ef356a6c2ae" member_type: VOTER last_known_addr { host: "127.28.113.3" port: 46345 } }
I20251215 21:02:56.405045 5377 leader_election.cc:290] T 072037ebcc654a85a5115664fed3cbbf P 38e72b9cde82428ea8acd83a5194f75d [CANDIDATE]: Term 1 election: Requested vote from peers f11866f255744d2fb31b4a6c2bcf6109 (127.28.113.2:44919), fe1797652ab94dff90667ef356a6c2ae (127.28.113.3:46345)
I20251215 21:02:56.405431 5297 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "072037ebcc654a85a5115664fed3cbbf" candidate_uuid: "38e72b9cde82428ea8acd83a5194f75d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe1797652ab94dff90667ef356a6c2ae"
I20251215 21:02:56.405550 5297 raft_consensus.cc:3060] T 072037ebcc654a85a5115664fed3cbbf P fe1797652ab94dff90667ef356a6c2ae [term 0 FOLLOWER]: Advancing to term 1
I20251215 21:02:56.405536 5154 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "072037ebcc654a85a5115664fed3cbbf" candidate_uuid: "38e72b9cde82428ea8acd83a5194f75d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f11866f255744d2fb31b4a6c2bcf6109"
I20251215 21:02:56.405648 5154 raft_consensus.cc:3060] T 072037ebcc654a85a5115664fed3cbbf P f11866f255744d2fb31b4a6c2bcf6109 [term 0 FOLLOWER]: Advancing to term 1
I20251215 21:02:56.406468 5154 raft_consensus.cc:2468] T 072037ebcc654a85a5115664fed3cbbf P f11866f255744d2fb31b4a6c2bcf6109 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38e72b9cde82428ea8acd83a5194f75d in term 1.
I20251215 21:02:56.406582 5297 raft_consensus.cc:2468] T 072037ebcc654a85a5115664fed3cbbf P fe1797652ab94dff90667ef356a6c2ae [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38e72b9cde82428ea8acd83a5194f75d in term 1.
I20251215 21:02:56.406760 4946 leader_election.cc:304] T 072037ebcc654a85a5115664fed3cbbf P 38e72b9cde82428ea8acd83a5194f75d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38e72b9cde82428ea8acd83a5194f75d, f11866f255744d2fb31b4a6c2bcf6109; no voters:
I20251215 21:02:56.407059 5377 raft_consensus.cc:2804] T 072037ebcc654a85a5115664fed3cbbf P 38e72b9cde82428ea8acd83a5194f75d [term 1 FOLLOWER]: Leader election won for term 1
I20251215 21:02:56.407145 5377 raft_consensus.cc:697] T 072037ebcc654a85a5115664fed3cbbf P 38e72b9cde82428ea8acd83a5194f75d [term 1 LEADER]: Becoming Leader. State: Replica: 38e72b9cde82428ea8acd83a5194f75d, State: Running, Role: LEADER
I20251215 21:02:56.407274 5377 consensus_queue.cc:237] T 072037ebcc654a85a5115664fed3cbbf P 38e72b9cde82428ea8acd83a5194f75d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38e72b9cde82428ea8acd83a5194f75d" member_type: VOTER last_known_addr { host: "127.28.113.1" port: 45679 } } peers { permanent_uuid: "f11866f255744d2fb31b4a6c2bcf6109" member_type: VOTER last_known_addr { host: "127.28.113.2" port: 44919 } } peers { permanent_uuid: "fe1797652ab94dff90667ef356a6c2ae" member_type: VOTER last_known_addr { host: "127.28.113.3" port: 46345 } }
I20251215 21:02:56.408878 4852 catalog_manager.cc:5654] T 072037ebcc654a85a5115664fed3cbbf P 38e72b9cde82428ea8acd83a5194f75d reported cstate change: term changed from 0 to 1, leader changed from <none> to 38e72b9cde82428ea8acd83a5194f75d (127.28.113.1). New cstate: current_term: 1 leader_uuid: "38e72b9cde82428ea8acd83a5194f75d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38e72b9cde82428ea8acd83a5194f75d" member_type: VOTER last_known_addr { host: "127.28.113.1" port: 45679 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f11866f255744d2fb31b4a6c2bcf6109" member_type: VOTER last_known_addr { host: "127.28.113.2" port: 44919 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fe1797652ab94dff90667ef356a6c2ae" member_type: VOTER last_known_addr { host: "127.28.113.3" port: 46345 } health_report { overall_health: UNKNOWN } } }
I20251215 21:02:56.427050 5369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:02:56.424474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41354 (local address 127.28.113.1:45679)
1215 21:02:56.424582 (+ 108us) server_negotiation.cc:207] Beginning negotiation
1215 21:02:56.424586 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1215 21:02:56.424622 (+ 36us) server_negotiation.cc:408] Connection header received
1215 21:02:56.424756 (+ 134us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1215 21:02:56.424760 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1215 21:02:56.424809 (+ 49us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1215 21:02:56.424870 (+ 61us) server_negotiation.cc:227] Negotiated authn=TOKEN
1215 21:02:56.425191 (+ 321us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:02:56.425700 (+ 509us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1215 21:02:56.426370 (+ 670us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:02:56.426570 (+ 200us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:02:56.426622 (+ 52us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1215 21:02:56.426690 (+ 68us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1215 21:02:56.426749 (+ 59us) server_negotiation.cc:1036] Waiting for connection context
1215 21:02:56.426859 (+ 110us) server_negotiation.cc:300] Negotiation successful
1215 21:02:56.426908 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20251215 21:02:56.433276 5296 raft_consensus.cc:1275] T 072037ebcc654a85a5115664fed3cbbf P fe1797652ab94dff90667ef356a6c2ae [term 1 FOLLOWER]: Refusing update from remote peer 38e72b9cde82428ea8acd83a5194f75d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251215 21:02:56.433373 5154 raft_consensus.cc:1275] T 072037ebcc654a85a5115664fed3cbbf P f11866f255744d2fb31b4a6c2bcf6109 [term 1 FOLLOWER]: Refusing update from remote peer 38e72b9cde82428ea8acd83a5194f75d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251215 21:02:56.433892 5377 consensus_queue.cc:1048] T 072037ebcc654a85a5115664fed3cbbf P 38e72b9cde82428ea8acd83a5194f75d [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe1797652ab94dff90667ef356a6c2ae" member_type: VOTER last_known_addr { host: "127.28.113.3" port: 46345 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251215 21:02:56.434089 5424 consensus_queue.cc:1048] T 072037ebcc654a85a5115664fed3cbbf P 38e72b9cde82428ea8acd83a5194f75d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f11866f255744d2fb31b4a6c2bcf6109" member_type: VOTER last_known_addr { host: "127.28.113.2" port: 44919 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251215 21:02:56.469585 4854 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34176:
table { table_name: "test-table" } modify_external_catalogs: true
I20251215 21:02:56.469725 4854 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34176:
table { table_name: "test-table" } modify_external_catalogs: true
I20251215 21:02:56.471673 4854 catalog_manager.cc:5941] T 00000000000000000000000000000000 P 4fc227cc99184e17b315174b90a73b5c: Sending DeleteTablet for 3 replicas of tablet 072037ebcc654a85a5115664fed3cbbf
I20251215 21:02:56.472241 4992 tablet_service.cc:1552] Processing DeleteTablet for tablet 072037ebcc654a85a5115664fed3cbbf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-15 21:02:56 UTC) from {username='oryx', principal='oryx/127.28.113.62@KRBTEST.COM'} at 127.0.0.1:41348
I20251215 21:02:56.472370 5277 tablet_service.cc:1552] Processing DeleteTablet for tablet 072037ebcc654a85a5115664fed3cbbf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-15 21:02:56 UTC) from {username='oryx', principal='oryx/127.28.113.62@KRBTEST.COM'} at 127.0.0.1:50932
I20251215 21:02:56.472450 5393 tablet_replica.cc:333] T 072037ebcc654a85a5115664fed3cbbf P 38e72b9cde82428ea8acd83a5194f75d: stopping tablet replica
I20251215 21:02:56.472445 5135 tablet_service.cc:1552] Processing DeleteTablet for tablet 072037ebcc654a85a5115664fed3cbbf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-15 21:02:56 UTC) from {username='oryx', principal='oryx/127.28.113.62@KRBTEST.COM'} at 127.0.0.1:35118
I20251215 21:02:56.472527 5395 tablet_replica.cc:333] T 072037ebcc654a85a5115664fed3cbbf P fe1797652ab94dff90667ef356a6c2ae: stopping tablet replica
I20251215 21:02:56.472548 5393 raft_consensus.cc:2243] T 072037ebcc654a85a5115664fed3cbbf P 38e72b9cde82428ea8acd83a5194f75d [term 1 LEADER]: Raft consensus shutting down.
I20251215 21:02:56.472615 5395 raft_consensus.cc:2243] T 072037ebcc654a85a5115664fed3cbbf P fe1797652ab94dff90667ef356a6c2ae [term 1 FOLLOWER]: Raft consensus shutting down.
I20251215 21:02:56.472725 5393 raft_consensus.cc:2272] T 072037ebcc654a85a5115664fed3cbbf P 38e72b9cde82428ea8acd83a5194f75d [term 1 FOLLOWER]: Raft consensus is shut down!
I20251215 21:02:56.472733 5395 raft_consensus.cc:2272] T 072037ebcc654a85a5115664fed3cbbf P fe1797652ab94dff90667ef356a6c2ae [term 1 FOLLOWER]: Raft consensus is shut down!
I20251215 21:02:56.472906 5394 tablet_replica.cc:333] T 072037ebcc654a85a5115664fed3cbbf P f11866f255744d2fb31b4a6c2bcf6109: stopping tablet replica
I20251215 21:02:56.473011 5394 raft_consensus.cc:2243] T 072037ebcc654a85a5115664fed3cbbf P f11866f255744d2fb31b4a6c2bcf6109 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251215 21:02:56.473120 5394 raft_consensus.cc:2272] T 072037ebcc654a85a5115664fed3cbbf P f11866f255744d2fb31b4a6c2bcf6109 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251215 21:02:56.473546 5393 ts_tablet_manager.cc:1916] T 072037ebcc654a85a5115664fed3cbbf P 38e72b9cde82428ea8acd83a5194f75d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251215 21:02:56.473866 5395 ts_tablet_manager.cc:1916] T 072037ebcc654a85a5115664fed3cbbf P fe1797652ab94dff90667ef356a6c2ae: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251215 21:02:56.475863 5395 ts_tablet_manager.cc:1929] T 072037ebcc654a85a5115664fed3cbbf P fe1797652ab94dff90667ef356a6c2ae: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20251215 21:02:56.475863 5393 ts_tablet_manager.cc:1929] T 072037ebcc654a85a5115664fed3cbbf P 38e72b9cde82428ea8acd83a5194f75d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20251215 21:02:56.475929 5393 log.cc:1199] T 072037ebcc654a85a5115664fed3cbbf P 38e72b9cde82428ea8acd83a5194f75d: Deleting WAL directory at /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765832506145157-29124-0/minicluster-data/ts-0/wal/wals/072037ebcc654a85a5115664fed3cbbf
I20251215 21:02:56.475929 5395 log.cc:1199] T 072037ebcc654a85a5115664fed3cbbf P fe1797652ab94dff90667ef356a6c2ae: Deleting WAL directory at /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765832506145157-29124-0/minicluster-data/ts-2/wal/wals/072037ebcc654a85a5115664fed3cbbf
I20251215 21:02:56.476195 5393 ts_tablet_manager.cc:1950] T 072037ebcc654a85a5115664fed3cbbf P 38e72b9cde82428ea8acd83a5194f75d: Deleting consensus metadata
I20251215 21:02:56.476542 5395 ts_tablet_manager.cc:1950] T 072037ebcc654a85a5115664fed3cbbf P fe1797652ab94dff90667ef356a6c2ae: Deleting consensus metadata
I20251215 21:02:56.476858 4839 catalog_manager.cc:4985] TS 38e72b9cde82428ea8acd83a5194f75d (127.28.113.1:45679): tablet 072037ebcc654a85a5115664fed3cbbf (table test-table [id=e2512ff87ea54becb5d0fe3e92cb9c84]) successfully deleted
I20251215 21:02:56.477253 29124 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu with pid 4928
I20251215 21:02:56.479081 4840 catalog_manager.cc:4985] TS fe1797652ab94dff90667ef356a6c2ae (127.28.113.3:46345): tablet 072037ebcc654a85a5115664fed3cbbf (table test-table [id=e2512ff87ea54becb5d0fe3e92cb9c84]) successfully deleted
I20251215 21:02:56.484586 5394 ts_tablet_manager.cc:1916] T 072037ebcc654a85a5115664fed3cbbf P f11866f255744d2fb31b4a6c2bcf6109: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251215 21:02:56.485163 29124 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu with pid 5070
W20251215 21:02:56.490890 4840 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.28.113.2:44919 (error 108)
I20251215 21:02:56.491577 29124 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu with pid 5212
I20251215 21:02:56.492424 5455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:02:56.491534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.113.2:44919 (local address 127.0.0.1:35168)
1215 21:02:56.492257 (+ 723us) negotiation.cc:107] Waiting for socket to connect
1215 21:02:56.492321 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.113.2:44919: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":642,"thread_start_us":52,"threads_started":1}
W20251215 21:02:56.492946 4840 catalog_manager.cc:4712] TS f11866f255744d2fb31b4a6c2bcf6109 (127.28.113.2:44919): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 072037ebcc654a85a5115664fed3cbbf: Network error: Client connection negotiation failed: client connection to 127.28.113.2:44919: connect: Connection refused (error 111)
I20251215 21:02:56.496619 29124 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu with pid 4822
2025-12-15T21:02:56Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (3656 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1765832506145157-29124-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Dec 15 21:02:56 dist-test-slave-bjsf krb5kdc[5458](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Dec 15 21:02:56 dist-test-slave-bjsf krb5kdc[5458](info): set up 2 sockets
Dec 15 21:02:56 dist-test-slave-bjsf krb5kdc[5458](info): commencing operation
krb5kdc: starting...
W20251215 21:02:58.324385 29124 mini_kdc.cc:121] Time spent starting KDC: real 1.807s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1765832506145157-29124-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1765832506145157-29124-0/krb5kdc/test-user.keytab.
Dec 15 21:02:58 dist-test-slave-bjsf krb5kdc[5458](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765832578, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-12-15T21:02:58Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-12-15T21:02:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.113.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.113.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.113.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1765832506145157-29124-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.28.113.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1765832506145157-29124-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.28.113.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.28.113.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.28.113.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1765832506145157-29124-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.28.113.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1765832506145157-29124-0/krb5kdc/kudu.keytab.
I20251215 21:02:58.446409 29124 external_mini_cluster.cc:1366] Running /tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu
/tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1765832506145157-29124-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1765832506145157-29124-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1765832506145157-29124-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1765832506145157-29124-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.113.62:38263
--webserver_interface=127.28.113.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1765832506145157-29124-0/krb5kdc/kudu.keytab
--principal=kudu/127.28.113.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.28.113.20:36507
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.113.62:38263
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1765832506145157-29124-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1765832506145157-29124-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1765832506145157-29124-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251215 21:02:58.550856 5474 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251215 21:02:58.551105 5474 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251215 21:02:58.551183 5474 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251215 21:02:58.554539 5474 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251215 21:02:58.554620 5474 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251215 21:02:58.554657 5474 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251215 21:02:58.554682 5474 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251215 21:02:58.554709 5474 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251215 21:02:58.559089 5474 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.113.20:36507
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1765832506145157-29124-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1765832506145157-29124-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.113.62:38263
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1765832506145157-29124-0/krb5kdc/kudu.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.113.62:38263
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1765832506145157-29124-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.113.62
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5474
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1765832506145157-29124-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision dc741732bb6a40213e11a13869fad06a49a209a3
build type DEBUG
built by None at 15 Dec 2025 20:43:14 UTC on 24a791456cd2
build id 9530
I20251215 21:02:58.560245 5474 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251215 21:02:58.561062 5474 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251215 21:02:58.566396 5479 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251215 21:02:58.566452 5480 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251215 21:02:58.566443 5482 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251215 21:02:58.566707 5474 server_base.cc:1047] running on GCE node
I20251215 21:02:58.567248 5474 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251215 21:02:58.568229 5474 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251215 21:02:58.569334 5474 hybrid_clock.cc:648] HybridClock initialized: now 1765832578569310 us; error 37 us; skew 500 ppm
W20251215 21:02:58.572413 5474 builtin_ntp.cc:688] coult not shutdown socket: Network error: shutdown error: Transport endpoint is not connected (error 107)
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2025-12-15T21:02:58Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (2076 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765832506145157-29124-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Dec 15 21:02:58 dist-test-slave-bjsf krb5kdc[5487](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Dec 15 21:02:58 dist-test-slave-bjsf krb5kdc[5487](info): set up 2 sockets
Dec 15 21:02:58 dist-test-slave-bjsf krb5kdc[5487](info): commencing operation
krb5kdc: starting...
W20251215 21:03:00.400345 29124 mini_kdc.cc:121] Time spent starting KDC: real 1.806s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765832506145157-29124-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765832506145157-29124-0/krb5kdc/test-user.keytab.
Dec 15 21:03:00 dist-test-slave-bjsf krb5kdc[5487](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765832580, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-12-15T21:03:00Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-12-15T21:03:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.113.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.113.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.113.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765832506145157-29124-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.28.113.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765832506145157-29124-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.28.113.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.28.113.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.28.113.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765832506145157-29124-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.28.113.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765832506145157-29124-0/krb5kdc/kudu.keytab.
I20251215 21:03:00.522825 29124 external_mini_cluster.cc:1366] Running /tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu
/tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765832506145157-29124-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765832506145157-29124-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765832506145157-29124-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765832506145157-29124-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.113.62:36095
--webserver_interface=127.28.113.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765832506145157-29124-0/krb5kdc/kudu.keytab
--principal=kudu/127.28.113.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.28.113.20:35785
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.113.62:36095
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765832506145157-29124-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765832506145157-29124-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765832506145157-29124-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251215 21:03:00.625605 5503 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251215 21:03:00.625803 5503 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251215 21:03:00.625854 5503 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251215 21:03:00.628939 5503 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251215 21:03:00.628989 5503 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251215 21:03:00.629010 5503 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251215 21:03:00.629038 5503 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251215 21:03:00.629055 5503 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251215 21:03:00.633078 5503 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.113.20:35785
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765832506145157-29124-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765832506145157-29124-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.113.62:36095
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765832506145157-29124-0/krb5kdc/kudu.keytab
--principal=kudu/127.28.113.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.113.62:36095
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765832506145157-29124-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.113.62
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5503
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765832506145157-29124-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision dc741732bb6a40213e11a13869fad06a49a209a3
build type DEBUG
built by None at 15 Dec 2025 20:43:14 UTC on 24a791456cd2
build id 9530
I20251215 21:03:00.633960 5503 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251215 21:03:00.634698 5503 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251215 21:03:00.639657 5509 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251215 21:03:00.639667 5508 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251215 21:03:00.639668 5511 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251215 21:03:00.639755 5503 server_base.cc:1047] running on GCE node
I20251215 21:03:00.640344 5503 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251215 21:03:00.641142 5503 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251215 21:03:00.642287 5503 hybrid_clock.cc:648] HybridClock initialized: now 1765832580642272 us; error 30 us; skew 500 ppm
Dec 15 21:03:00 dist-test-slave-bjsf krb5kdc[5487](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765832580, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.113.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251215 21:03:00.644588 5503 init.cc:377] Logged in from keytab as kudu/127.28.113.62@KRBTEST.COM (short username kudu)
I20251215 21:03:00.645501 5503 webserver.cc:492] Webserver started at http://127.28.113.62:40493/ using document root <none> and password file <none>
I20251215 21:03:00.645984 5503 fs_manager.cc:362] Metadata directory not provided
I20251215 21:03:00.646029 5503 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251215 21:03:00.646189 5503 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251215 21:03:00.647727 5503 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765832506145157-29124-0/minicluster-data/master-0/data/instance:
uuid: "f94e89b05e2c4d8abb105f5f9402385f"
format_stamp: "Formatted at 2025-12-15 21:03:00 on dist-test-slave-bjsf"
server_key: "6b74cfc63d992c33a3cc4d632c9b292f"
server_key_iv: "b0680c0663ef49632961d9728430ce6e"
server_key_version: "encryptionkey@0"
I20251215 21:03:00.648180 5503 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765832506145157-29124-0/minicluster-data/master-0/wal/instance:
uuid: "f94e89b05e2c4d8abb105f5f9402385f"
format_stamp: "Formatted at 2025-12-15 21:03:00 on dist-test-slave-bjsf"
server_key: "6b74cfc63d992c33a3cc4d632c9b292f"
server_key_iv: "b0680c0663ef49632961d9728430ce6e"
server_key_version: "encryptionkey@0"
I20251215 21:03:00.651113 5503 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.001s sys 0.002s
I20251215 21:03:00.653152 5518 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251215 21:03:00.654033 5503 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20251215 21:03:00.654129 5503 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765832506145157-29124-0/minicluster-data/master-0/data,/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765832506145157-29124-0/minicluster-data/master-0/wal
uuid: "f94e89b05e2c4d8abb105f5f9402385f"
format_stamp: "Formatted at 2025-12-15 21:03:00 on dist-test-slave-bjsf"
server_key: "6b74cfc63d992c33a3cc4d632c9b292f"
server_key_iv: "b0680c0663ef49632961d9728430ce6e"
server_key_version: "encryptionkey@0"
I20251215 21:03:00.654210 5503 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765832506145157-29124-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765832506145157-29124-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765832506145157-29124-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251215 21:03:00.674173 5503 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251215 21:03:00.679536 5503 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251215 21:03:00.679685 5503 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251215 21:03:00.686100 5570 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.28.113.62:36095 every 8 connection(s)
I20251215 21:03:00.686101 5503 rpc_server.cc:307] RPC server started. Bound to: 127.28.113.62:36095
I20251215 21:03:00.687057 5503 server_base.cc:1179] Dumped server information to /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765832506145157-29124-0/minicluster-data/master-0/data/info.pb
I20251215 21:03:00.689653 5571 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 21:03:00.694713 5571 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f94e89b05e2c4d8abb105f5f9402385f: Bootstrap starting.
I20251215 21:03:00.696981 5571 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f94e89b05e2c4d8abb105f5f9402385f: Neither blocks nor log segments found. Creating new log.
I20251215 21:03:00.697314 29124 external_mini_cluster.cc:1428] Started /tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu as pid 5503
I20251215 21:03:00.697444 29124 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765832506145157-29124-0/minicluster-data/master-0/wal/instance
I20251215 21:03:00.697687 29124 external_mini_cluster.cc:1468] Setting key 415ee5ec17b3061989e6674906b10305
I20251215 21:03:00.697805 5571 log.cc:826] T 00000000000000000000000000000000 P f94e89b05e2c4d8abb105f5f9402385f: Log is configured to *not* fsync() on all Append() calls
I20251215 21:03:00.699585 5571 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f94e89b05e2c4d8abb105f5f9402385f: No bootstrap required, opened a new log
I20251215 21:03:00.702481 5571 raft_consensus.cc:359] T 00000000000000000000000000000000 P f94e89b05e2c4d8abb105f5f9402385f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f94e89b05e2c4d8abb105f5f9402385f" member_type: VOTER last_known_addr { host: "127.28.113.62" port: 36095 } }
I20251215 21:03:00.702693 5571 raft_consensus.cc:385] T 00000000000000000000000000000000 P f94e89b05e2c4d8abb105f5f9402385f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251215 21:03:00.702745 5571 raft_consensus.cc:740] T 00000000000000000000000000000000 P f94e89b05e2c4d8abb105f5f9402385f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f94e89b05e2c4d8abb105f5f9402385f, State: Initialized, Role: FOLLOWER
I20251215 21:03:00.703166 5571 consensus_queue.cc:260] T 00000000000000000000000000000000 P f94e89b05e2c4d8abb105f5f9402385f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f94e89b05e2c4d8abb105f5f9402385f" member_type: VOTER last_known_addr { host: "127.28.113.62" port: 36095 } }
I20251215 21:03:00.703299 5571 raft_consensus.cc:399] T 00000000000000000000000000000000 P f94e89b05e2c4d8abb105f5f9402385f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251215 21:03:00.703359 5571 raft_consensus.cc:493] T 00000000000000000000000000000000 P f94e89b05e2c4d8abb105f5f9402385f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251215 21:03:00.703441 5571 raft_consensus.cc:3060] T 00000000000000000000000000000000 P f94e89b05e2c4d8abb105f5f9402385f [term 0 FOLLOWER]: Advancing to term 1
I20251215 21:03:00.704365 5571 raft_consensus.cc:515] T 00000000000000000000000000000000 P f94e89b05e2c4d8abb105f5f9402385f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f94e89b05e2c4d8abb105f5f9402385f" member_type: VOTER last_known_addr { host: "127.28.113.62" port: 36095 } }
I20251215 21:03:00.704663 5571 leader_election.cc:304] T 00000000000000000000000000000000 P f94e89b05e2c4d8abb105f5f9402385f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f94e89b05e2c4d8abb105f5f9402385f; no voters:
I20251215 21:03:00.704895 5571 leader_election.cc:290] T 00000000000000000000000000000000 P f94e89b05e2c4d8abb105f5f9402385f [CANDIDATE]: Term 1 election: Requested vote from peers
I20251215 21:03:00.705044 5576 raft_consensus.cc:2804] T 00000000000000000000000000000000 P f94e89b05e2c4d8abb105f5f9402385f [term 1 FOLLOWER]: Leader election won for term 1
I20251215 21:03:00.705224 5576 raft_consensus.cc:697] T 00000000000000000000000000000000 P f94e89b05e2c4d8abb105f5f9402385f [term 1 LEADER]: Becoming Leader. State: Replica: f94e89b05e2c4d8abb105f5f9402385f, State: Running, Role: LEADER
I20251215 21:03:00.705513 5576 consensus_queue.cc:237] T 00000000000000000000000000000000 P f94e89b05e2c4d8abb105f5f9402385f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f94e89b05e2c4d8abb105f5f9402385f" member_type: VOTER last_known_addr { host: "127.28.113.62" port: 36095 } }
Dec 15 21:03:00 dist-test-slave-bjsf krb5kdc[5487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765832580, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.113.62@KRBTEST.COM
I20251215 21:03:00.705834 5571 sys_catalog.cc:565] T 00000000000000000000000000000000 P f94e89b05e2c4d8abb105f5f9402385f [sys.catalog]: configured and running, proceeding with master startup.
I20251215 21:03:00.706885 5578 sys_catalog.cc:455] T 00000000000000000000000000000000 P f94e89b05e2c4d8abb105f5f9402385f [sys.catalog]: SysCatalogTable state changed. Reason: New leader f94e89b05e2c4d8abb105f5f9402385f. Latest consensus state: current_term: 1 leader_uuid: "f94e89b05e2c4d8abb105f5f9402385f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f94e89b05e2c4d8abb105f5f9402385f" member_type: VOTER last_known_addr { host: "127.28.113.62" port: 36095 } } }
I20251215 21:03:00.706997 5578 sys_catalog.cc:458] T 00000000000000000000000000000000 P f94e89b05e2c4d8abb105f5f9402385f [sys.catalog]: This master's current role is: LEADER
I20251215 21:03:00.707680 5585 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251215 21:03:00.707801 5577 sys_catalog.cc:455] T 00000000000000000000000000000000 P f94e89b05e2c4d8abb105f5f9402385f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f94e89b05e2c4d8abb105f5f9402385f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f94e89b05e2c4d8abb105f5f9402385f" member_type: VOTER last_known_addr { host: "127.28.113.62" port: 36095 } } }
I20251215 21:03:00.708801 5577 sys_catalog.cc:458] T 00000000000000000000000000000000 P f94e89b05e2c4d8abb105f5f9402385f [sys.catalog]: This master's current role is: LEADER
I20251215 21:03:00.711424 5575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:03:00.698594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56634 (local address 127.28.113.62:36095)
1215 21:03:00.698955 (+ 361us) server_negotiation.cc:207] Beginning negotiation
1215 21:03:00.698964 (+ 9us) server_negotiation.cc:400] Waiting for connection header
1215 21:03:00.699901 (+ 937us) server_negotiation.cc:408] Connection header received
1215 21:03:00.700575 (+ 674us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1215 21:03:00.700599 (+ 24us) server_negotiation.cc:462] Received NEGOTIATE request from client
1215 21:03:00.700933 (+ 334us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1215 21:03:00.701267 (+ 334us) server_negotiation.cc:227] Negotiated authn=SASL
1215 21:03:00.701931 (+ 664us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:03:00.703049 (+ 1118us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1215 21:03:00.703745 (+ 696us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:03:00.704040 (+ 295us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:03:00.706155 (+ 2115us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1215 21:03:00.706176 (+ 21us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1215 21:03:00.706188 (+ 12us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1215 21:03:00.706225 (+ 37us) server_negotiation.cc:923] calling sasl_server_start()
1215 21:03:00.709218 (+ 2993us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1215 21:03:00.709645 (+ 427us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1215 21:03:00.709649 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1215 21:03:00.709653 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
1215 21:03:00.709720 (+ 67us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1215 21:03:00.709934 (+ 214us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1215 21:03:00.709938 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1215 21:03:00.709940 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1215 21:03:00.710236 (+ 296us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1215 21:03:00.710424 (+ 188us) server_negotiation.cc:1036] Waiting for connection context
1215 21:03:00.710681 (+ 257us) server_negotiation.cc:300] Negotiation successful
1215 21:03:00.710907 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":232,"thread_start_us":110,"threads_started":1}
I20251215 21:03:00.711484 5585 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251215 21:03:00.716630 5585 catalog_manager.cc:1357] Generated new cluster ID: 6369679763bc4ea383094c0beda70033
I20251215 21:03:00.716691 5585 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251215 21:03:00.757736 5585 catalog_manager.cc:1380] Generated new certificate authority record
I20251215 21:03:00.759021 5585 catalog_manager.cc:1514] Loading token signing keys...
I20251215 21:03:00.770978 5585 catalog_manager.cc:6027] T 00000000000000000000000000000000 P f94e89b05e2c4d8abb105f5f9402385f: Generated new TSK 0
I20251215 21:03:00.771793 5585 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.113.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.113.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.113.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765832506145157-29124-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.28.113.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765832506145157-29124-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.28.113.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.28.113.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.28.113.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765832506145157-29124-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.28.113.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765832506145157-29124-0/krb5kdc/kudu.keytab.
I20251215 21:03:00.816291 29124 external_mini_cluster.cc:1366] Running /tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu
/tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765832506145157-29124-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765832506145157-29124-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765832506145157-29124-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765832506145157-29124-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.113.1:0
--local_ip_for_outbound_sockets=127.28.113.1
--webserver_interface=127.28.113.1
--webserver_port=0
--tserver_master_addrs=127.28.113.62:36095
--keytab_file=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765832506145157-29124-0/krb5kdc/kudu.keytab
--principal=kudu/127.28.113.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.28.113.20:35785
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765832506145157-29124-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765832506145157-29124-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765832506145157-29124-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251215 21:03:00.920688 5599 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251215 21:03:00.920912 5599 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251215 21:03:00.920969 5599 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251215 21:03:00.924149 5599 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251215 21:03:00.924217 5599 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251215 21:03:00.924300 5599 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.113.1
I20251215 21:03:00.928365 5599 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.113.20:35785
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765832506145157-29124-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765832506145157-29124-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765832506145157-29124-0/krb5kdc/kudu.keytab
--principal=kudu/127.28.113.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.113.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765832506145157-29124-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.113.1
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.28.113.62:36095
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5599
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.113.1
--log_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765832506145157-29124-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision dc741732bb6a40213e11a13869fad06a49a209a3
build type DEBUG
built by None at 15 Dec 2025 20:43:14 UTC on 24a791456cd2
build id 9530
I20251215 21:03:00.929320 5599 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251215 21:03:00.930092 5599 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251215 21:03:00.936056 5604 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251215 21:03:00.936095 5605 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251215 21:03:00.936067 5607 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251215 21:03:00.936309 5599 server_base.cc:1047] running on GCE node
I20251215 21:03:00.936646 5599 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251215 21:03:00.937140 5599 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251215 21:03:00.938305 5599 hybrid_clock.cc:648] HybridClock initialized: now 1765832580938282 us; error 35 us; skew 500 ppm
Dec 15 21:03:00 dist-test-slave-bjsf krb5kdc[5487](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765832580, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.113.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251215 21:03:00.940757 5599 init.cc:377] Logged in from keytab as kudu/127.28.113.1@KRBTEST.COM (short username kudu)
I20251215 21:03:00.941738 5599 webserver.cc:492] Webserver started at http://127.28.113.1:39617/ using document root <none> and password file <none>
I20251215 21:03:00.942232 5599 fs_manager.cc:362] Metadata directory not provided
I20251215 21:03:00.942281 5599 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251215 21:03:00.942436 5599 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251215 21:03:00.944110 5599 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765832506145157-29124-0/minicluster-data/ts-0/data/instance:
uuid: "4c316270e54b48b9b41b54da62211911"
format_stamp: "Formatted at 2025-12-15 21:03:00 on dist-test-slave-bjsf"
server_key: "104e0063a2c374811263ba63e1e1f515"
server_key_iv: "df47b64488fe5ec72e9bc9679f6822e2"
server_key_version: "encryptionkey@0"
I20251215 21:03:00.944555 5599 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765832506145157-29124-0/minicluster-data/ts-0/wal/instance:
uuid: "4c316270e54b48b9b41b54da62211911"
format_stamp: "Formatted at 2025-12-15 21:03:00 on dist-test-slave-bjsf"
server_key: "104e0063a2c374811263ba63e1e1f515"
server_key_iv: "df47b64488fe5ec72e9bc9679f6822e2"
server_key_version: "encryptionkey@0"
I20251215 21:03:00.947710 5599 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20251215 21:03:00.949839 5614 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251215 21:03:00.950794 5599 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20251215 21:03:00.950891 5599 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765832506145157-29124-0/minicluster-data/ts-0/data,/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765832506145157-29124-0/minicluster-data/ts-0/wal
uuid: "4c316270e54b48b9b41b54da62211911"
format_stamp: "Formatted at 2025-12-15 21:03:00 on dist-test-slave-bjsf"
server_key: "104e0063a2c374811263ba63e1e1f515"
server_key_iv: "df47b64488fe5ec72e9bc9679f6822e2"
server_key_version: "encryptionkey@0"
I20251215 21:03:00.950989 5599 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765832506145157-29124-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765832506145157-29124-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765832506145157-29124-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251215 21:03:00.972712 5599 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251215 21:03:00.975180 5599 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251215 21:03:00.975339 5599 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251215 21:03:00.975862 5599 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251215 21:03:00.976747 5599 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251215 21:03:00.976796 5599 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251215 21:03:00.976843 5599 ts_tablet_manager.cc:616] Registered 0 tablets
I20251215 21:03:00.976876 5599 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251215 21:03:00.985941 5599 rpc_server.cc:307] RPC server started. Bound to: 127.28.113.1:36199
I20251215 21:03:00.985968 5727 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.28.113.1:36199 every 8 connection(s)
I20251215 21:03:00.986851 5599 server_base.cc:1179] Dumped server information to /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765832506145157-29124-0/minicluster-data/ts-0/data/info.pb
I20251215 21:03:00.991191 29124 external_mini_cluster.cc:1428] Started /tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu as pid 5599
I20251215 21:03:00.991334 29124 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765832506145157-29124-0/minicluster-data/ts-0/wal/instance
I20251215 21:03:00.991623 29124 external_mini_cluster.cc:1468] Setting key 3a642a4988e95eab38499049cbcbdf3f
Dec 15 21:03:00 dist-test-slave-bjsf krb5kdc[5487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765832580, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.113.1@KRBTEST.COM for kudu/127.28.113.62@KRBTEST.COM
I20251215 21:03:00.999249 5575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:03:00.988448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.113.1:56587 (local address 127.28.113.62:36095)
1215 21:03:00.988582 (+ 134us) server_negotiation.cc:207] Beginning negotiation
1215 21:03:00.988586 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1215 21:03:00.989194 (+ 608us) server_negotiation.cc:408] Connection header received
1215 21:03:00.989958 (+ 764us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1215 21:03:00.989961 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1215 21:03:00.990011 (+ 50us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1215 21:03:00.990079 (+ 68us) server_negotiation.cc:227] Negotiated authn=SASL
1215 21:03:00.991828 (+ 1749us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:03:00.992431 (+ 603us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1215 21:03:00.993180 (+ 749us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:03:00.993309 (+ 129us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:03:00.995727 (+ 2418us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1215 21:03:00.995747 (+ 20us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1215 21:03:00.995750 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1215 21:03:00.995776 (+ 26us) server_negotiation.cc:923] calling sasl_server_start()
1215 21:03:00.997385 (+ 1609us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1215 21:03:00.997872 (+ 487us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1215 21:03:00.997875 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1215 21:03:00.997877 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1215 21:03:00.997937 (+ 60us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1215 21:03:00.998255 (+ 318us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1215 21:03:00.998259 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1215 21:03:00.998261 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1215 21:03:00.998429 (+ 168us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1215 21:03:00.998529 (+ 100us) server_negotiation.cc:1036] Waiting for connection context
1215 21:03:00.998974 (+ 445us) server_negotiation.cc:300] Negotiation successful
1215 21:03:00.999090 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20251215 21:03:00.999718 5730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:03:00.988657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.113.62:36095 (local address 127.28.113.1:56587)
1215 21:03:00.989054 (+ 397us) negotiation.cc:107] Waiting for socket to connect
1215 21:03:00.989090 (+ 36us) client_negotiation.cc:174] Beginning negotiation
1215 21:03:00.989787 (+ 697us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1215 21:03:00.990216 (+ 429us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1215 21:03:00.990224 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
1215 21:03:00.990587 (+ 363us) client_negotiation.cc:189] Negotiated authn=SASL
1215 21:03:00.991681 (+ 1094us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:03:00.991700 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:03:00.992547 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1215 21:03:00.992551 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1215 21:03:00.993062 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:03:00.993073 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:03:00.993290 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:03:00.993859 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1215 21:03:00.993894 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
1215 21:03:00.995585 (+ 1691us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1215 21:03:00.997500 (+ 1915us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1215 21:03:00.997504 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1215 21:03:00.997513 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
1215 21:03:00.997735 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1215 21:03:00.998065 (+ 330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1215 21:03:00.998068 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1215 21:03:00.998069 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1215 21:03:00.998158 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1215 21:03:00.998523 (+ 365us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1215 21:03:00.998527 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1215 21:03:00.998715 (+ 188us) client_negotiation.cc:769] Sending connection context
1215 21:03:00.998900 (+ 185us) client_negotiation.cc:240] Negotiation successful
1215 21:03:00.999097 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":95,"threads_started":1}
WARNING: no policy specified for kudu/127.28.113.2@KRBTEST.COM; defaulting to no policy
I20251215 21:03:01.000842 5728 heartbeater.cc:344] Connected to a master server at 127.28.113.62:36095
I20251215 21:03:01.001037 5728 heartbeater.cc:461] Registering TS with master...
I20251215 21:03:01.001487 5728 heartbeater.cc:507] Master 127.28.113.62:36095 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.113.2@KRBTEST.COM" created.
I20251215 21:03:01.002918 5535 ts_manager.cc:194] Registered new tserver with Master: 4c316270e54b48b9b41b54da62211911 (127.28.113.1:36199)
I20251215 21:03:01.004112 5535 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.113.1@KRBTEST.COM'} at 127.28.113.1:56587
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.113.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765832506145157-29124-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.28.113.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765832506145157-29124-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.28.113.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.28.113.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.28.113.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765832506145157-29124-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.28.113.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765832506145157-29124-0/krb5kdc/kudu.keytab.
I20251215 21:03:01.034500 29124 external_mini_cluster.cc:1366] Running /tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu
/tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765832506145157-29124-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765832506145157-29124-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765832506145157-29124-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765832506145157-29124-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.113.2:0
--local_ip_for_outbound_sockets=127.28.113.2
--webserver_interface=127.28.113.2
--webserver_port=0
--tserver_master_addrs=127.28.113.62:36095
--keytab_file=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765832506145157-29124-0/krb5kdc/kudu.keytab
--principal=kudu/127.28.113.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.28.113.20:35785
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765832506145157-29124-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765832506145157-29124-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765832506145157-29124-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251215 21:03:01.134013 5735 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251215 21:03:01.134217 5735 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251215 21:03:01.134279 5735 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251215 21:03:01.137387 5735 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251215 21:03:01.137436 5735 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251215 21:03:01.137526 5735 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.113.2
I20251215 21:03:01.141489 5735 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.113.20:35785
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765832506145157-29124-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765832506145157-29124-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765832506145157-29124-0/krb5kdc/kudu.keytab
--principal=kudu/127.28.113.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.113.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765832506145157-29124-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.113.2
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.28.113.62:36095
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5735
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.113.2
--log_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765832506145157-29124-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision dc741732bb6a40213e11a13869fad06a49a209a3
build type DEBUG
built by None at 15 Dec 2025 20:43:14 UTC on 24a791456cd2
build id 9530
I20251215 21:03:01.142426 5735 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251215 21:03:01.143205 5735 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251215 21:03:01.149149 5741 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251215 21:03:01.149175 5740 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251215 21:03:01.149175 5743 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251215 21:03:01.149401 5735 server_base.cc:1047] running on GCE node
I20251215 21:03:01.149712 5735 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251215 21:03:01.150168 5735 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251215 21:03:01.151309 5735 hybrid_clock.cc:648] HybridClock initialized: now 1765832581151301 us; error 24 us; skew 500 ppm
Dec 15 21:03:01 dist-test-slave-bjsf krb5kdc[5487](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765832581, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.113.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251215 21:03:01.153687 5735 init.cc:377] Logged in from keytab as kudu/127.28.113.2@KRBTEST.COM (short username kudu)
I20251215 21:03:01.154562 5735 webserver.cc:492] Webserver started at http://127.28.113.2:38811/ using document root <none> and password file <none>
I20251215 21:03:01.155054 5735 fs_manager.cc:362] Metadata directory not provided
I20251215 21:03:01.155110 5735 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251215 21:03:01.155284 5735 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251215 21:03:01.156844 5735 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765832506145157-29124-0/minicluster-data/ts-1/data/instance:
uuid: "61d1d97b007c4f8d8046bc30765eaf0c"
format_stamp: "Formatted at 2025-12-15 21:03:01 on dist-test-slave-bjsf"
server_key: "96813cc8bcdb75163ffc7a34b0074816"
server_key_iv: "e5802d43ca0206e108b247e2f2a824b6"
server_key_version: "encryptionkey@0"
I20251215 21:03:01.157240 5735 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765832506145157-29124-0/minicluster-data/ts-1/wal/instance:
uuid: "61d1d97b007c4f8d8046bc30765eaf0c"
format_stamp: "Formatted at 2025-12-15 21:03:01 on dist-test-slave-bjsf"
server_key: "96813cc8bcdb75163ffc7a34b0074816"
server_key_iv: "e5802d43ca0206e108b247e2f2a824b6"
server_key_version: "encryptionkey@0"
I20251215 21:03:01.160271 5735 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20251215 21:03:01.162153 5750 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251215 21:03:01.163062 5735 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20251215 21:03:01.163154 5735 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765832506145157-29124-0/minicluster-data/ts-1/data,/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765832506145157-29124-0/minicluster-data/ts-1/wal
uuid: "61d1d97b007c4f8d8046bc30765eaf0c"
format_stamp: "Formatted at 2025-12-15 21:03:01 on dist-test-slave-bjsf"
server_key: "96813cc8bcdb75163ffc7a34b0074816"
server_key_iv: "e5802d43ca0206e108b247e2f2a824b6"
server_key_version: "encryptionkey@0"
I20251215 21:03:01.163232 5735 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765832506145157-29124-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765832506145157-29124-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765832506145157-29124-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251215 21:03:01.185185 5735 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251215 21:03:01.187420 5735 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251215 21:03:01.187566 5735 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251215 21:03:01.188055 5735 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251215 21:03:01.188830 5735 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251215 21:03:01.188874 5735 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251215 21:03:01.188912 5735 ts_tablet_manager.cc:616] Registered 0 tablets
I20251215 21:03:01.188941 5735 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251215 21:03:01.197072 5735 rpc_server.cc:307] RPC server started. Bound to: 127.28.113.2:40925
I20251215 21:03:01.197115 5863 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.28.113.2:40925 every 8 connection(s)
I20251215 21:03:01.197880 5735 server_base.cc:1179] Dumped server information to /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765832506145157-29124-0/minicluster-data/ts-1/data/info.pb
I20251215 21:03:01.198496 29124 external_mini_cluster.cc:1428] Started /tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu as pid 5735
I20251215 21:03:01.198606 29124 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765832506145157-29124-0/minicluster-data/ts-1/wal/instance
I20251215 21:03:01.198853 29124 external_mini_cluster.cc:1468] Setting key bcab16e296f15f3c15d6501e9a2d623c
Dec 15 21:03:01 dist-test-slave-bjsf krb5kdc[5487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765832581, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.113.2@KRBTEST.COM for kudu/127.28.113.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.28.113.3@KRBTEST.COM; defaulting to no policy
I20251215 21:03:01.209574 5575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:03:01.199438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.113.2:49937 (local address 127.28.113.62:36095)
1215 21:03:01.199566 (+ 128us) server_negotiation.cc:207] Beginning negotiation
1215 21:03:01.199571 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1215 21:03:01.200259 (+ 688us) server_negotiation.cc:408] Connection header received
1215 21:03:01.201079 (+ 820us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1215 21:03:01.201083 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1215 21:03:01.201132 (+ 49us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1215 21:03:01.201201 (+ 69us) server_negotiation.cc:227] Negotiated authn=SASL
1215 21:03:01.202678 (+ 1477us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:03:01.203424 (+ 746us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1215 21:03:01.204089 (+ 665us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:03:01.204270 (+ 181us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:03:01.206235 (+ 1965us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1215 21:03:01.206253 (+ 18us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1215 21:03:01.206257 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1215 21:03:01.206283 (+ 26us) server_negotiation.cc:923] calling sasl_server_start()
1215 21:03:01.207881 (+ 1598us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1215 21:03:01.208408 (+ 527us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1215 21:03:01.208412 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1215 21:03:01.208414 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1215 21:03:01.208463 (+ 49us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1215 21:03:01.208690 (+ 227us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1215 21:03:01.208692 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1215 21:03:01.208693 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1215 21:03:01.208821 (+ 128us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1215 21:03:01.208887 (+ 66us) server_negotiation.cc:1036] Waiting for connection context
1215 21:03:01.209367 (+ 480us) server_negotiation.cc:300] Negotiation successful
1215 21:03:01.209457 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20251215 21:03:01.210153 5866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:03:01.199712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.113.62:36095 (local address 127.28.113.2:49937)
1215 21:03:01.200133 (+ 421us) negotiation.cc:107] Waiting for socket to connect
1215 21:03:01.200179 (+ 46us) client_negotiation.cc:174] Beginning negotiation
1215 21:03:01.200907 (+ 728us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1215 21:03:01.201348 (+ 441us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1215 21:03:01.201355 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
1215 21:03:01.201683 (+ 328us) client_negotiation.cc:189] Negotiated authn=SASL
1215 21:03:01.202527 (+ 844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:03:01.202540 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:03:01.203545 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1215 21:03:01.203548 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1215 21:03:01.203949 (+ 401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:03:01.203992 (+ 43us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:03:01.204168 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:03:01.204737 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1215 21:03:01.204759 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
1215 21:03:01.206092 (+ 1333us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1215 21:03:01.208015 (+ 1923us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1215 21:03:01.208021 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1215 21:03:01.208030 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
1215 21:03:01.208290 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1215 21:03:01.208544 (+ 254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1215 21:03:01.208546 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1215 21:03:01.208548 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1215 21:03:01.208623 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1215 21:03:01.208892 (+ 269us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1215 21:03:01.208897 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1215 21:03:01.209142 (+ 245us) client_negotiation.cc:769] Sending connection context
1215 21:03:01.209327 (+ 185us) client_negotiation.cc:240] Negotiation successful
1215 21:03:01.209514 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":104,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.113.3@KRBTEST.COM" created.
I20251215 21:03:01.212574 5864 heartbeater.cc:344] Connected to a master server at 127.28.113.62:36095
I20251215 21:03:01.212782 5864 heartbeater.cc:461] Registering TS with master...
I20251215 21:03:01.213241 5864 heartbeater.cc:507] Master 127.28.113.62:36095 requested a full tablet report, sending...
I20251215 21:03:01.214182 5535 ts_manager.cc:194] Registered new tserver with Master: 61d1d97b007c4f8d8046bc30765eaf0c (127.28.113.2:40925)
I20251215 21:03:01.214695 5535 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.113.2@KRBTEST.COM'} at 127.28.113.2:49937
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.113.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765832506145157-29124-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.28.113.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765832506145157-29124-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.28.113.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.28.113.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.28.113.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765832506145157-29124-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.28.113.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765832506145157-29124-0/krb5kdc/kudu.keytab.
I20251215 21:03:01.240123 29124 external_mini_cluster.cc:1366] Running /tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu
/tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765832506145157-29124-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765832506145157-29124-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765832506145157-29124-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765832506145157-29124-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.113.3:0
--local_ip_for_outbound_sockets=127.28.113.3
--webserver_interface=127.28.113.3
--webserver_port=0
--tserver_master_addrs=127.28.113.62:36095
--keytab_file=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765832506145157-29124-0/krb5kdc/kudu.keytab
--principal=kudu/127.28.113.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.28.113.20:35785
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765832506145157-29124-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765832506145157-29124-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765832506145157-29124-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251215 21:03:01.338014 5871 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251215 21:03:01.338207 5871 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251215 21:03:01.338260 5871 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251215 21:03:01.341305 5871 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251215 21:03:01.341353 5871 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251215 21:03:01.341423 5871 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.113.3
I20251215 21:03:01.345399 5871 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.113.20:35785
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765832506145157-29124-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765832506145157-29124-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765832506145157-29124-0/krb5kdc/kudu.keytab
--principal=kudu/127.28.113.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.113.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765832506145157-29124-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.113.3
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.28.113.62:36095
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5871
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.113.3
--log_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765832506145157-29124-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision dc741732bb6a40213e11a13869fad06a49a209a3
build type DEBUG
built by None at 15 Dec 2025 20:43:14 UTC on 24a791456cd2
build id 9530
I20251215 21:03:01.346299 5871 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251215 21:03:01.347014 5871 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251215 21:03:01.353065 5876 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251215 21:03:01.353051 5879 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251215 21:03:01.353108 5871 server_base.cc:1047] running on GCE node
W20251215 21:03:01.353048 5877 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251215 21:03:01.353571 5871 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251215 21:03:01.354049 5871 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251215 21:03:01.355192 5871 hybrid_clock.cc:648] HybridClock initialized: now 1765832581355182 us; error 28 us; skew 500 ppm
Dec 15 21:03:01 dist-test-slave-bjsf krb5kdc[5487](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765832581, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.113.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251215 21:03:01.357522 5871 init.cc:377] Logged in from keytab as kudu/127.28.113.3@KRBTEST.COM (short username kudu)
I20251215 21:03:01.358458 5871 webserver.cc:492] Webserver started at http://127.28.113.3:35073/ using document root <none> and password file <none>
I20251215 21:03:01.358937 5871 fs_manager.cc:362] Metadata directory not provided
I20251215 21:03:01.358984 5871 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251215 21:03:01.359140 5871 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251215 21:03:01.360728 5871 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765832506145157-29124-0/minicluster-data/ts-2/data/instance:
uuid: "af9e6a36437a4646808aa957442cb807"
format_stamp: "Formatted at 2025-12-15 21:03:01 on dist-test-slave-bjsf"
server_key: "520ee2b037c512b41a41d3551c1530d1"
server_key_iv: "77e04ea860c4ed725475c5ba22ddc2c5"
server_key_version: "encryptionkey@0"
I20251215 21:03:01.361140 5871 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765832506145157-29124-0/minicluster-data/ts-2/wal/instance:
uuid: "af9e6a36437a4646808aa957442cb807"
format_stamp: "Formatted at 2025-12-15 21:03:01 on dist-test-slave-bjsf"
server_key: "520ee2b037c512b41a41d3551c1530d1"
server_key_iv: "77e04ea860c4ed725475c5ba22ddc2c5"
server_key_version: "encryptionkey@0"
I20251215 21:03:01.364164 5871 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20251215 21:03:01.366124 5886 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251215 21:03:01.367054 5871 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20251215 21:03:01.367137 5871 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765832506145157-29124-0/minicluster-data/ts-2/data,/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765832506145157-29124-0/minicluster-data/ts-2/wal
uuid: "af9e6a36437a4646808aa957442cb807"
format_stamp: "Formatted at 2025-12-15 21:03:01 on dist-test-slave-bjsf"
server_key: "520ee2b037c512b41a41d3551c1530d1"
server_key_iv: "77e04ea860c4ed725475c5ba22ddc2c5"
server_key_version: "encryptionkey@0"
I20251215 21:03:01.367205 5871 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765832506145157-29124-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765832506145157-29124-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765832506145157-29124-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251215 21:03:01.387624 5871 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251215 21:03:01.390084 5871 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251215 21:03:01.390218 5871 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251215 21:03:01.390659 5871 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251215 21:03:01.391422 5871 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251215 21:03:01.391464 5871 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251215 21:03:01.391500 5871 ts_tablet_manager.cc:616] Registered 0 tablets
I20251215 21:03:01.391520 5871 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251215 21:03:01.399669 5871 rpc_server.cc:307] RPC server started. Bound to: 127.28.113.3:41555
I20251215 21:03:01.399706 5999 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.28.113.3:41555 every 8 connection(s)
I20251215 21:03:01.400542 5871 server_base.cc:1179] Dumped server information to /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765832506145157-29124-0/minicluster-data/ts-2/data/info.pb
I20251215 21:03:01.404264 29124 external_mini_cluster.cc:1428] Started /tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu as pid 5871
I20251215 21:03:01.404371 29124 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765832506145157-29124-0/minicluster-data/ts-2/wal/instance
I20251215 21:03:01.404590 29124 external_mini_cluster.cc:1468] Setting key 7824c89a1def389e306bf97f363f1afb
Dec 15 21:03:01 dist-test-slave-bjsf krb5kdc[5487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765832581, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.113.3@KRBTEST.COM for kudu/127.28.113.62@KRBTEST.COM
I20251215 21:03:01.411912 5575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:03:01.402096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.113.3:59297 (local address 127.28.113.62:36095)
1215 21:03:01.402235 (+ 139us) server_negotiation.cc:207] Beginning negotiation
1215 21:03:01.402240 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1215 21:03:01.402873 (+ 633us) server_negotiation.cc:408] Connection header received
1215 21:03:01.403676 (+ 803us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1215 21:03:01.403679 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1215 21:03:01.403738 (+ 59us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1215 21:03:01.403811 (+ 73us) server_negotiation.cc:227] Negotiated authn=SASL
1215 21:03:01.405570 (+ 1759us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:03:01.406058 (+ 488us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1215 21:03:01.406720 (+ 662us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:03:01.406895 (+ 175us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:03:01.409053 (+ 2158us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1215 21:03:01.409076 (+ 23us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1215 21:03:01.409079 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1215 21:03:01.409103 (+ 24us) server_negotiation.cc:923] calling sasl_server_start()
1215 21:03:01.410214 (+ 1111us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1215 21:03:01.410648 (+ 434us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1215 21:03:01.410651 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1215 21:03:01.410653 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1215 21:03:01.410694 (+ 41us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1215 21:03:01.410948 (+ 254us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1215 21:03:01.410952 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1215 21:03:01.410954 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1215 21:03:01.411102 (+ 148us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1215 21:03:01.411185 (+ 83us) server_negotiation.cc:1036] Waiting for connection context
1215 21:03:01.411643 (+ 458us) server_negotiation.cc:300] Negotiation successful
1215 21:03:01.411756 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20251215 21:03:01.412530 6002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:03:01.402326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.113.62:36095 (local address 127.28.113.3:59297)
1215 21:03:01.402737 (+ 411us) negotiation.cc:107] Waiting for socket to connect
1215 21:03:01.402770 (+ 33us) client_negotiation.cc:174] Beginning negotiation
1215 21:03:01.403485 (+ 715us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1215 21:03:01.404035 (+ 550us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1215 21:03:01.404043 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
1215 21:03:01.404471 (+ 428us) client_negotiation.cc:189] Negotiated authn=SASL
1215 21:03:01.405436 (+ 965us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:03:01.405450 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:03:01.406181 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1215 21:03:01.406184 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1215 21:03:01.406586 (+ 402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:03:01.406599 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:03:01.406848 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:03:01.407437 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1215 21:03:01.407460 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
1215 21:03:01.408906 (+ 1446us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1215 21:03:01.410326 (+ 1420us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1215 21:03:01.410332 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1215 21:03:01.410343 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
1215 21:03:01.410556 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1215 21:03:01.410772 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1215 21:03:01.410775 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1215 21:03:01.410776 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1215 21:03:01.410864 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1215 21:03:01.411202 (+ 338us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1215 21:03:01.411209 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1215 21:03:01.411417 (+ 208us) client_negotiation.cc:769] Sending connection context
1215 21:03:01.411592 (+ 175us) client_negotiation.cc:240] Negotiation successful
1215 21:03:01.411850 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":98,"threads_started":1}
I20251215 21:03:01.413448 6000 heartbeater.cc:344] Connected to a master server at 127.28.113.62:36095
I20251215 21:03:01.413662 6000 heartbeater.cc:461] Registering TS with master...
I20251215 21:03:01.414117 6000 heartbeater.cc:507] Master 127.28.113.62:36095 requested a full tablet report, sending...
I20251215 21:03:01.415005 5535 ts_manager.cc:194] Registered new tserver with Master: af9e6a36437a4646808aa957442cb807 (127.28.113.3:41555)
I20251215 21:03:01.415517 5535 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.113.3@KRBTEST.COM'} at 127.28.113.3:59297
I20251215 21:03:01.417958 29124 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.28.113.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.28.113.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.28.113.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765832506145157-29124-0/krb5kdc/oryx_127.28.113.1.keytab.
Entry for principal oryx/127.28.113.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765832506145157-29124-0/krb5kdc/oryx_127.28.113.1.keytab.
WARNING: no policy specified for oryx/127.28.113.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.28.113.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.28.113.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765832506145157-29124-0/krb5kdc/oryx_127.28.113.2.keytab.
Entry for principal oryx/127.28.113.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765832506145157-29124-0/krb5kdc/oryx_127.28.113.2.keytab.
WARNING: no policy specified for oryx/127.28.113.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.28.113.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.28.113.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765832506145157-29124-0/krb5kdc/oryx_127.28.113.3.keytab.
Entry for principal oryx/127.28.113.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765832506145157-29124-0/krb5kdc/oryx_127.28.113.3.keytab.
I20251215 21:03:01.476012 29124 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu with pid 5599
I20251215 21:03:01.480924 29124 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu with pid 5735
I20251215 21:03:01.485574 29124 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu with pid 5871
I20251215 21:03:01.489821 29124 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu with pid 5503
I20251215 21:03:01.494746 29124 external_mini_cluster.cc:1366] Running /tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu
/tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765832506145157-29124-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765832506145157-29124-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765832506145157-29124-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765832506145157-29124-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.113.62:36095
--webserver_interface=127.28.113.62
--webserver_port=40493
--keytab_file=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765832506145157-29124-0/krb5kdc/kudu.keytab
--principal=kudu/127.28.113.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.28.113.20:35785
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.113.62:36095
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765832506145157-29124-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765832506145157-29124-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765832506145157-29124-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251215 21:03:01.599418 6009 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251215 21:03:01.599625 6009 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251215 21:03:01.599699 6009 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251215 21:03:01.602860 6009 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251215 21:03:01.602918 6009 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251215 21:03:01.602948 6009 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251215 21:03:01.602972 6009 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251215 21:03:01.603003 6009 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251215 21:03:01.607035 6009 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.113.20:35785
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765832506145157-29124-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765832506145157-29124-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.113.62:36095
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765832506145157-29124-0/krb5kdc/kudu.keytab
--principal=kudu/127.28.113.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.113.62:36095
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765832506145157-29124-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.113.62
--webserver_port=40493
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6009
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765832506145157-29124-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision dc741732bb6a40213e11a13869fad06a49a209a3
build type DEBUG
built by None at 15 Dec 2025 20:43:14 UTC on 24a791456cd2
build id 9530
I20251215 21:03:01.608037 6009 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251215 21:03:01.608831 6009 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251215 21:03:01.613893 6014 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251215 21:03:01.613945 6015 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251215 21:03:01.613898 6017 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251215 21:03:01.614107 6009 server_base.cc:1047] running on GCE node
I20251215 21:03:01.614583 6009 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251215 21:03:01.615399 6009 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251215 21:03:01.616540 6009 hybrid_clock.cc:648] HybridClock initialized: now 1765832581616528 us; error 26 us; skew 500 ppm
Dec 15 21:03:01 dist-test-slave-bjsf krb5kdc[5487](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765832581, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.113.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251215 21:03:01.618877 6009 init.cc:377] Logged in from keytab as kudu/127.28.113.62@KRBTEST.COM (short username kudu)
I20251215 21:03:01.619819 6009 webserver.cc:492] Webserver started at http://127.28.113.62:40493/ using document root <none> and password file <none>
I20251215 21:03:01.620350 6009 fs_manager.cc:362] Metadata directory not provided
I20251215 21:03:01.620416 6009 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251215 21:03:01.623647 6009 fs_manager.cc:714] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20251215 21:03:01.625512 6024 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251215 21:03:01.626426 6009 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20251215 21:03:01.626528 6009 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765832506145157-29124-0/minicluster-data/master-0/data,/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765832506145157-29124-0/minicluster-data/master-0/wal
uuid: "f94e89b05e2c4d8abb105f5f9402385f"
format_stamp: "Formatted at 2025-12-15 21:03:00 on dist-test-slave-bjsf"
server_key: "6b74cfc63d992c33a3cc4d632c9b292f"
server_key_iv: "b0680c0663ef49632961d9728430ce6e"
server_key_version: "encryptionkey@0"
I20251215 21:03:01.626902 6009 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765832506145157-29124-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765832506145157-29124-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765832506145157-29124-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251215 21:03:01.642499 6009 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251215 21:03:01.645042 6009 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251215 21:03:01.645202 6009 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251215 21:03:01.651940 6076 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.28.113.62:36095 every 8 connection(s)
I20251215 21:03:01.651945 6009 rpc_server.cc:307] RPC server started. Bound to: 127.28.113.62:36095
I20251215 21:03:01.652956 6009 server_base.cc:1179] Dumped server information to /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765832506145157-29124-0/minicluster-data/master-0/data/info.pb
I20251215 21:03:01.657249 6077 sys_catalog.cc:263] Verifying existing consensus state
I20251215 21:03:01.658694 6077 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f94e89b05e2c4d8abb105f5f9402385f: Bootstrap starting.
I20251215 21:03:01.659850 29124 external_mini_cluster.cc:1428] Started /tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu as pid 6009
I20251215 21:03:01.660373 29124 external_mini_cluster.cc:1366] Running /tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu
/tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765832506145157-29124-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765832506145157-29124-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765832506145157-29124-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765832506145157-29124-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.113.1:36199
--local_ip_for_outbound_sockets=127.28.113.1
--tserver_master_addrs=127.28.113.62:36095
--webserver_port=39617
--webserver_interface=127.28.113.1
--keytab_file=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765832506145157-29124-0/krb5kdc/kudu.keytab
--principal=kudu/127.28.113.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.28.113.20:35785
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.28.113.1
--keytab_file=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765832506145157-29124-0/krb5kdc/oryx_127.28.113.1.keytab with env {KRB5CCNAME=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765832506145157-29124-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765832506145157-29124-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765832506145157-29124-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20251215 21:03:01.674695 6077 log.cc:826] T 00000000000000000000000000000000 P f94e89b05e2c4d8abb105f5f9402385f: Log is configured to *not* fsync() on all Append() calls
I20251215 21:03:01.680783 6077 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f94e89b05e2c4d8abb105f5f9402385f: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251215 21:03:01.681123 6077 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f94e89b05e2c4d8abb105f5f9402385f: Bootstrap complete.
I20251215 21:03:01.684533 6077 raft_consensus.cc:359] T 00000000000000000000000000000000 P f94e89b05e2c4d8abb105f5f9402385f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f94e89b05e2c4d8abb105f5f9402385f" member_type: VOTER last_known_addr { host: "127.28.113.62" port: 36095 } }
I20251215 21:03:01.684834 6077 raft_consensus.cc:740] T 00000000000000000000000000000000 P f94e89b05e2c4d8abb105f5f9402385f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: f94e89b05e2c4d8abb105f5f9402385f, State: Initialized, Role: FOLLOWER
I20251215 21:03:01.685283 6077 consensus_queue.cc:260] T 00000000000000000000000000000000 P f94e89b05e2c4d8abb105f5f9402385f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f94e89b05e2c4d8abb105f5f9402385f" member_type: VOTER last_known_addr { host: "127.28.113.62" port: 36095 } }
I20251215 21:03:01.685388 6077 raft_consensus.cc:399] T 00000000000000000000000000000000 P f94e89b05e2c4d8abb105f5f9402385f [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251215 21:03:01.685443 6077 raft_consensus.cc:493] T 00000000000000000000000000000000 P f94e89b05e2c4d8abb105f5f9402385f [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251215 21:03:01.685523 6077 raft_consensus.cc:3060] T 00000000000000000000000000000000 P f94e89b05e2c4d8abb105f5f9402385f [term 1 FOLLOWER]: Advancing to term 2
I20251215 21:03:01.686925 6077 raft_consensus.cc:515] T 00000000000000000000000000000000 P f94e89b05e2c4d8abb105f5f9402385f [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f94e89b05e2c4d8abb105f5f9402385f" member_type: VOTER last_known_addr { host: "127.28.113.62" port: 36095 } }
I20251215 21:03:01.687232 6077 leader_election.cc:304] T 00000000000000000000000000000000 P f94e89b05e2c4d8abb105f5f9402385f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f94e89b05e2c4d8abb105f5f9402385f; no voters:
I20251215 21:03:01.687479 6077 leader_election.cc:290] T 00000000000000000000000000000000 P f94e89b05e2c4d8abb105f5f9402385f [CANDIDATE]: Term 2 election: Requested vote from peers
I20251215 21:03:01.687608 6081 raft_consensus.cc:2804] T 00000000000000000000000000000000 P f94e89b05e2c4d8abb105f5f9402385f [term 2 FOLLOWER]: Leader election won for term 2
I20251215 21:03:01.687883 6081 raft_consensus.cc:697] T 00000000000000000000000000000000 P f94e89b05e2c4d8abb105f5f9402385f [term 2 LEADER]: Becoming Leader. State: Replica: f94e89b05e2c4d8abb105f5f9402385f, State: Running, Role: LEADER
I20251215 21:03:01.688238 6081 consensus_queue.cc:237] T 00000000000000000000000000000000 P f94e89b05e2c4d8abb105f5f9402385f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f94e89b05e2c4d8abb105f5f9402385f" member_type: VOTER last_known_addr { host: "127.28.113.62" port: 36095 } }
I20251215 21:03:01.688503 6077 sys_catalog.cc:565] T 00000000000000000000000000000000 P f94e89b05e2c4d8abb105f5f9402385f [sys.catalog]: configured and running, proceeding with master startup.
I20251215 21:03:01.689447 6083 sys_catalog.cc:455] T 00000000000000000000000000000000 P f94e89b05e2c4d8abb105f5f9402385f [sys.catalog]: SysCatalogTable state changed. Reason: New leader f94e89b05e2c4d8abb105f5f9402385f. Latest consensus state: current_term: 2 leader_uuid: "f94e89b05e2c4d8abb105f5f9402385f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f94e89b05e2c4d8abb105f5f9402385f" member_type: VOTER last_known_addr { host: "127.28.113.62" port: 36095 } } }
I20251215 21:03:01.689549 6083 sys_catalog.cc:458] T 00000000000000000000000000000000 P f94e89b05e2c4d8abb105f5f9402385f [sys.catalog]: This master's current role is: LEADER
I20251215 21:03:01.690119 6082 sys_catalog.cc:455] T 00000000000000000000000000000000 P f94e89b05e2c4d8abb105f5f9402385f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "f94e89b05e2c4d8abb105f5f9402385f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f94e89b05e2c4d8abb105f5f9402385f" member_type: VOTER last_known_addr { host: "127.28.113.62" port: 36095 } } }
I20251215 21:03:01.690191 6082 sys_catalog.cc:458] T 00000000000000000000000000000000 P f94e89b05e2c4d8abb105f5f9402385f [sys.catalog]: This master's current role is: LEADER
I20251215 21:03:01.692612 6098 catalog_manager.cc:1269] Loaded cluster ID: 6369679763bc4ea383094c0beda70033
I20251215 21:03:01.692662 6098 catalog_manager.cc:1562] T 00000000000000000000000000000000 P f94e89b05e2c4d8abb105f5f9402385f: loading cluster ID for follower catalog manager: success
I20251215 21:03:01.694386 6098 catalog_manager.cc:1584] T 00000000000000000000000000000000 P f94e89b05e2c4d8abb105f5f9402385f: acquiring CA information for follower catalog manager: success
I20251215 21:03:01.695465 6098 catalog_manager.cc:1612] T 00000000000000000000000000000000 P f94e89b05e2c4d8abb105f5f9402385f: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20251215 21:03:01.695561 6090 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251215 21:03:01.696295 6090 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251215 21:03:01.696663 6090 catalog_manager.cc:1269] Loaded cluster ID: 6369679763bc4ea383094c0beda70033
I20251215 21:03:01.696710 6090 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251215 21:03:01.697199 6090 catalog_manager.cc:1514] Loading token signing keys...
I20251215 21:03:01.697590 6090 catalog_manager.cc:6038] T 00000000000000000000000000000000 P f94e89b05e2c4d8abb105f5f9402385f: Loaded TSK: 0
I20251215 21:03:01.698138 6090 catalog_manager.cc:1524] Initializing in-progress tserver states...
W20251215 21:03:01.777899 6079 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251215 21:03:01.778110 6079 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251215 21:03:01.778185 6079 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251215 21:03:01.781334 6079 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251215 21:03:01.781395 6079 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251215 21:03:01.781490 6079 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.113.1
I20251215 21:03:01.785547 6079 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.113.20:35785
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765832506145157-29124-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765832506145157-29124-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765832506145157-29124-0/krb5kdc/oryx_127.28.113.1.keytab
--principal=oryx/127.28.113.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.113.1:36199
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765832506145157-29124-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.113.1
--webserver_port=39617
--webserver_require_spnego=true
--tserver_master_addrs=127.28.113.62:36095
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6079
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.113.1
--log_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765832506145157-29124-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision dc741732bb6a40213e11a13869fad06a49a209a3
build type DEBUG
built by None at 15 Dec 2025 20:43:14 UTC on 24a791456cd2
build id 9530
I20251215 21:03:01.786504 6079 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251215 21:03:01.787302 6079 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251215 21:03:01.793664 6104 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251215 21:03:01.793669 6107 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251215 21:03:01.793749 6105 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251215 21:03:01.793983 6079 server_base.cc:1047] running on GCE node
I20251215 21:03:01.794327 6079 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251215 21:03:01.794806 6079 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251215 21:03:01.795949 6079 hybrid_clock.cc:648] HybridClock initialized: now 1765832581795930 us; error 39 us; skew 500 ppm
Dec 15 21:03:01 dist-test-slave-bjsf krb5kdc[5487](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765832581, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.113.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251215 21:03:01.798273 6079 init.cc:377] Logged in from keytab as oryx/127.28.113.1@KRBTEST.COM (short username oryx)
I20251215 21:03:01.799247 6079 webserver.cc:492] Webserver started at http://127.28.113.1:39617/ using document root <none> and password file <none>
I20251215 21:03:01.799741 6079 fs_manager.cc:362] Metadata directory not provided
I20251215 21:03:01.799800 6079 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251215 21:03:01.803334 6079 fs_manager.cc:714] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20251215 21:03:01.805222 6114 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251215 21:03:01.806186 6079 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20251215 21:03:01.806274 6079 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765832506145157-29124-0/minicluster-data/ts-0/data,/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765832506145157-29124-0/minicluster-data/ts-0/wal
uuid: "4c316270e54b48b9b41b54da62211911"
format_stamp: "Formatted at 2025-12-15 21:03:00 on dist-test-slave-bjsf"
server_key: "104e0063a2c374811263ba63e1e1f515"
server_key_iv: "df47b64488fe5ec72e9bc9679f6822e2"
server_key_version: "encryptionkey@0"
I20251215 21:03:01.806666 6079 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765832506145157-29124-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765832506145157-29124-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765832506145157-29124-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251215 21:03:01.835152 6079 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251215 21:03:01.838045 6079 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251215 21:03:01.838204 6079 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251215 21:03:01.838714 6079 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251215 21:03:01.839560 6079 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251215 21:03:01.839607 6079 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251215 21:03:01.839658 6079 ts_tablet_manager.cc:616] Registered 0 tablets
I20251215 21:03:01.839684 6079 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251215 21:03:01.848738 6079 rpc_server.cc:307] RPC server started. Bound to: 127.28.113.1:36199
I20251215 21:03:01.848805 6227 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.28.113.1:36199 every 8 connection(s)
I20251215 21:03:01.849664 6079 server_base.cc:1179] Dumped server information to /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765832506145157-29124-0/minicluster-data/ts-0/data/info.pb
I20251215 21:03:01.856132 29124 external_mini_cluster.cc:1428] Started /tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu as pid 6079
I20251215 21:03:01.856611 29124 external_mini_cluster.cc:1366] Running /tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu
/tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765832506145157-29124-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765832506145157-29124-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765832506145157-29124-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765832506145157-29124-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.113.2:40925
--local_ip_for_outbound_sockets=127.28.113.2
--tserver_master_addrs=127.28.113.62:36095
--webserver_port=38811
--webserver_interface=127.28.113.2
--keytab_file=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765832506145157-29124-0/krb5kdc/kudu.keytab
--principal=kudu/127.28.113.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.28.113.20:35785
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.28.113.2
--keytab_file=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765832506145157-29124-0/krb5kdc/oryx_127.28.113.2.keytab with env {KRB5CCNAME=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765832506145157-29124-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765832506145157-29124-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765832506145157-29124-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Dec 15 21:03:01 dist-test-slave-bjsf krb5kdc[5487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.113.1@KRBTEST.COM for oryx/127.28.113.62@KRBTEST.COM, Server not found in Kerberos database
Dec 15 21:03:01 dist-test-slave-bjsf krb5kdc[5487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.113.1@KRBTEST.COM for oryx/127.28.113.62@KRBTEST.COM, Server not found in Kerberos database
I20251215 21:03:01.862251 6231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:03:01.851708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.113.62:36095 (local address 127.28.113.1:50995)
1215 21:03:01.852245 (+ 537us) negotiation.cc:107] Waiting for socket to connect
1215 21:03:01.852286 (+ 41us) client_negotiation.cc:174] Beginning negotiation
1215 21:03:01.853156 (+ 870us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1215 21:03:01.854116 (+ 960us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1215 21:03:01.854125 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
1215 21:03:01.854594 (+ 469us) client_negotiation.cc:189] Negotiated authn=SASL
1215 21:03:01.855722 (+ 1128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:03:01.855739 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:03:01.857299 (+ 1560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1215 21:03:01.857303 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1215 21:03:01.857912 (+ 609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:03:01.857921 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:03:01.858132 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:03:01.858862 (+ 730us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1215 21:03:01.858894 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
1215 21:03:01.861481 (+ 2587us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.113.62:36095: Server oryx/127.28.113.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":334,"thread_start_us":101,"threads_started":1}
W20251215 21:03:01.862357 6230 negotiation.cc:337] Failed RPC negotiation. Trace:
1215 21:03:01.851670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.113.1:50995 (local address 127.28.113.62:36095)
1215 21:03:01.852145 (+ 475us) server_negotiation.cc:207] Beginning negotiation
1215 21:03:01.852157 (+ 12us) server_negotiation.cc:400] Waiting for connection header
1215 21:03:01.852390 (+ 233us) server_negotiation.cc:408] Connection header received
1215 21:03:01.853511 (+ 1121us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1215 21:03:01.853521 (+ 10us) server_negotiation.cc:462] Received NEGOTIATE request from client
1215 21:03:01.853752 (+ 231us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1215 21:03:01.853994 (+ 242us) server_negotiation.cc:227] Negotiated authn=SASL
1215 21:03:01.855904 (+ 1910us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:03:01.857141 (+ 1237us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1215 21:03:01.858047 (+ 906us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:03:01.858315 (+ 268us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:03:01.861701 (+ 3386us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.113.1:50995: BlockingRecv error: recv got EOF from 127.28.113.1:50995 (error 108)
Metrics: {"server-negotiator.queue_time_us":317,"thread_start_us":110,"threads_started":1}
W20251215 21:03:01.863408 6228 heartbeater.cc:646] Failed to heartbeat to 127.28.113.62:36095 (0 consecutive failures): Not authorized: Failed to ping master at 127.28.113.62:36095: Client connection negotiation failed: client connection to 127.28.113.62:36095: Server oryx/127.28.113.62@KRBTEST.COM not found in Kerberos database
Dec 15 21:03:01 dist-test-slave-bjsf krb5kdc[5487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.113.1@KRBTEST.COM for oryx/127.28.113.62@KRBTEST.COM, Server not found in Kerberos database
Dec 15 21:03:01 dist-test-slave-bjsf krb5kdc[5487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.113.1@KRBTEST.COM for oryx/127.28.113.62@KRBTEST.COM, Server not found in Kerberos database
I20251215 21:03:01.869067 6231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:03:01.863847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.113.62:36095 (local address 127.28.113.1:42711)
1215 21:03:01.864185 (+ 338us) negotiation.cc:107] Waiting for socket to connect
1215 21:03:01.864198 (+ 13us) client_negotiation.cc:174] Beginning negotiation
1215 21:03:01.864280 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1215 21:03:01.864513 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1215 21:03:01.864517 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
1215 21:03:01.864671 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
1215 21:03:01.864907 (+ 236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:03:01.864914 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:03:01.865869 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1215 21:03:01.865873 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1215 21:03:01.866424 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:03:01.866434 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:03:01.866542 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:03:01.867056 (+ 514us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1215 21:03:01.867073 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
1215 21:03:01.868950 (+ 1877us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.113.62:36095: Server oryx/127.28.113.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":254}
W20251215 21:03:01.869199 6230 negotiation.cc:337] Failed RPC negotiation. Trace:
1215 21:03:01.864046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.113.1:42711 (local address 127.28.113.62:36095)
1215 21:03:01.864175 (+ 129us) server_negotiation.cc:207] Beginning negotiation
1215 21:03:01.864180 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1215 21:03:01.864241 (+ 61us) server_negotiation.cc:408] Connection header received
1215 21:03:01.864362 (+ 121us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1215 21:03:01.864365 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1215 21:03:01.864413 (+ 48us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1215 21:03:01.864494 (+ 81us) server_negotiation.cc:227] Negotiated authn=SASL
1215 21:03:01.865018 (+ 524us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:03:01.865746 (+ 728us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1215 21:03:01.866541 (+ 795us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:03:01.867078 (+ 537us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:03:01.869093 (+ 2015us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.113.1:42711: BlockingRecv error: recv got EOF from 127.28.113.1:42711 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Dec 15 21:03:01 dist-test-slave-bjsf krb5kdc[5487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.113.1@KRBTEST.COM for oryx/127.28.113.62@KRBTEST.COM, Server not found in Kerberos database
Dec 15 21:03:01 dist-test-slave-bjsf krb5kdc[5487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.113.1@KRBTEST.COM for oryx/127.28.113.62@KRBTEST.COM, Server not found in Kerberos database
I20251215 21:03:01.874553 6231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:03:01.869799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.113.62:36095 (local address 127.28.113.1:46891)
1215 21:03:01.870046 (+ 247us) negotiation.cc:107] Waiting for socket to connect
1215 21:03:01.870058 (+ 12us) client_negotiation.cc:174] Beginning negotiation
1215 21:03:01.870132 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1215 21:03:01.870360 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1215 21:03:01.870363 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1215 21:03:01.870507 (+ 144us) client_negotiation.cc:189] Negotiated authn=SASL
1215 21:03:01.870708 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:03:01.870714 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:03:01.871578 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1215 21:03:01.871581 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1215 21:03:01.872154 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:03:01.872164 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:03:01.872265 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:03:01.872706 (+ 441us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1215 21:03:01.872719 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
1215 21:03:01.874444 (+ 1725us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.113.62:36095: Server oryx/127.28.113.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":181}
W20251215 21:03:01.874681 6230 negotiation.cc:337] Failed RPC negotiation. Trace:
1215 21:03:01.869935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.113.1:46891 (local address 127.28.113.62:36095)
1215 21:03:01.870046 (+ 111us) server_negotiation.cc:207] Beginning negotiation
1215 21:03:01.870051 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1215 21:03:01.870096 (+ 45us) server_negotiation.cc:408] Connection header received
1215 21:03:01.870216 (+ 120us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1215 21:03:01.870219 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1215 21:03:01.870261 (+ 42us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1215 21:03:01.870340 (+ 79us) server_negotiation.cc:227] Negotiated authn=SASL
1215 21:03:01.870807 (+ 467us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:03:01.871488 (+ 681us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1215 21:03:01.872256 (+ 768us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:03:01.872741 (+ 485us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:03:01.874570 (+ 1829us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.113.1:46891: BlockingRecv error: recv got EOF from 127.28.113.1:46891 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
W20251215 21:03:01.874976 6228 heartbeater.cc:412] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20251215 21:03:01.961844 6232 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251215 21:03:01.962046 6232 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251215 21:03:01.962100 6232 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251215 21:03:01.965231 6232 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251215 21:03:01.965287 6232 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251215 21:03:01.965360 6232 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.113.2
I20251215 21:03:01.969290 6232 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.113.20:35785
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765832506145157-29124-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765832506145157-29124-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765832506145157-29124-0/krb5kdc/oryx_127.28.113.2.keytab
--principal=oryx/127.28.113.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.113.2:40925
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765832506145157-29124-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.113.2
--webserver_port=38811
--webserver_require_spnego=true
--tserver_master_addrs=127.28.113.62:36095
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6232
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.113.2
--log_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765832506145157-29124-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision dc741732bb6a40213e11a13869fad06a49a209a3
build type DEBUG
built by None at 15 Dec 2025 20:43:14 UTC on 24a791456cd2
build id 9530
I20251215 21:03:01.970227 6232 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251215 21:03:01.970971 6232 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251215 21:03:01.976809 6240 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251215 21:03:01.976795 6238 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251215 21:03:01.976795 6237 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251215 21:03:01.977058 6232 server_base.cc:1047] running on GCE node
I20251215 21:03:01.977371 6232 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251215 21:03:01.977835 6232 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251215 21:03:01.978973 6232 hybrid_clock.cc:648] HybridClock initialized: now 1765832581978953 us; error 35 us; skew 500 ppm
Dec 15 21:03:01 dist-test-slave-bjsf krb5kdc[5487](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765832581, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.113.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251215 21:03:01.981175 6232 init.cc:377] Logged in from keytab as oryx/127.28.113.2@KRBTEST.COM (short username oryx)
I20251215 21:03:01.982059 6232 webserver.cc:492] Webserver started at http://127.28.113.2:38811/ using document root <none> and password file <none>
I20251215 21:03:01.982538 6232 fs_manager.cc:362] Metadata directory not provided
I20251215 21:03:01.982599 6232 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251215 21:03:01.985813 6232 fs_manager.cc:714] Time spent opening directory manager: real 0.002s user 0.002s sys 0.001s
I20251215 21:03:01.987439 6247 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251215 21:03:01.988384 6232 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20251215 21:03:01.988477 6232 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765832506145157-29124-0/minicluster-data/ts-1/data,/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765832506145157-29124-0/minicluster-data/ts-1/wal
uuid: "61d1d97b007c4f8d8046bc30765eaf0c"
format_stamp: "Formatted at 2025-12-15 21:03:01 on dist-test-slave-bjsf"
server_key: "96813cc8bcdb75163ffc7a34b0074816"
server_key_iv: "e5802d43ca0206e108b247e2f2a824b6"
server_key_version: "encryptionkey@0"
I20251215 21:03:01.988848 6232 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765832506145157-29124-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765832506145157-29124-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765832506145157-29124-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251215 21:03:02.002499 6232 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251215 21:03:02.005455 6232 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251215 21:03:02.005622 6232 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251215 21:03:02.006146 6232 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251215 21:03:02.006968 6232 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251215 21:03:02.007012 6232 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251215 21:03:02.007063 6232 ts_tablet_manager.cc:616] Registered 0 tablets
I20251215 21:03:02.007090 6232 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251215 21:03:02.015467 6232 rpc_server.cc:307] RPC server started. Bound to: 127.28.113.2:40925
I20251215 21:03:02.015492 6360 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.28.113.2:40925 every 8 connection(s)
I20251215 21:03:02.016419 6232 server_base.cc:1179] Dumped server information to /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765832506145157-29124-0/minicluster-data/ts-1/data/info.pb
I20251215 21:03:02.021986 29124 external_mini_cluster.cc:1428] Started /tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu as pid 6232
I20251215 21:03:02.022409 29124 external_mini_cluster.cc:1366] Running /tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu
/tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765832506145157-29124-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765832506145157-29124-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765832506145157-29124-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765832506145157-29124-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.113.3:41555
--local_ip_for_outbound_sockets=127.28.113.3
--tserver_master_addrs=127.28.113.62:36095
--webserver_port=35073
--webserver_interface=127.28.113.3
--keytab_file=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765832506145157-29124-0/krb5kdc/kudu.keytab
--principal=kudu/127.28.113.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.28.113.20:35785
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.28.113.3
--keytab_file=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765832506145157-29124-0/krb5kdc/oryx_127.28.113.3.keytab with env {KRB5CCNAME=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765832506145157-29124-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765832506145157-29124-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765832506145157-29124-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Dec 15 21:03:02 dist-test-slave-bjsf krb5kdc[5487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.113.2@KRBTEST.COM for oryx/127.28.113.62@KRBTEST.COM, Server not found in Kerberos database
Dec 15 21:03:02 dist-test-slave-bjsf krb5kdc[5487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.113.2@KRBTEST.COM for oryx/127.28.113.62@KRBTEST.COM, Server not found in Kerberos database
W20251215 21:03:02.026114 6230 negotiation.cc:337] Failed RPC negotiation. Trace:
1215 21:03:02.018026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.113.2:51703 (local address 127.28.113.62:36095)
1215 21:03:02.018174 (+ 148us) server_negotiation.cc:207] Beginning negotiation
1215 21:03:02.018179 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1215 21:03:02.018844 (+ 665us) server_negotiation.cc:408] Connection header received
1215 21:03:02.019611 (+ 767us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1215 21:03:02.019614 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1215 21:03:02.019665 (+ 51us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1215 21:03:02.019737 (+ 72us) server_negotiation.cc:227] Negotiated authn=SASL
1215 21:03:02.021328 (+ 1591us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:03:02.021918 (+ 590us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1215 21:03:02.022581 (+ 663us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:03:02.022733 (+ 152us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:03:02.025972 (+ 3239us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.113.2:51703: BlockingRecv error: recv got EOF from 127.28.113.2:51703 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
I20251215 21:03:02.026445 6363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:03:02.018321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.113.62:36095 (local address 127.28.113.2:51703)
1215 21:03:02.018713 (+ 392us) negotiation.cc:107] Waiting for socket to connect
1215 21:03:02.018746 (+ 33us) client_negotiation.cc:174] Beginning negotiation
1215 21:03:02.019439 (+ 693us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1215 21:03:02.019869 (+ 430us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1215 21:03:02.019876 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
1215 21:03:02.020301 (+ 425us) client_negotiation.cc:189] Negotiated authn=SASL
1215 21:03:02.021191 (+ 890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:03:02.021204 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:03:02.022019 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1215 21:03:02.022022 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1215 21:03:02.022445 (+ 423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:03:02.022453 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:03:02.022917 (+ 464us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:03:02.023476 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1215 21:03:02.023507 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
1215 21:03:02.025790 (+ 2283us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.113.62:36095: Server oryx/127.28.113.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":98,"threads_started":1}
W20251215 21:03:02.026829 6361 heartbeater.cc:646] Failed to heartbeat to 127.28.113.62:36095 (0 consecutive failures): Not authorized: Failed to ping master at 127.28.113.62:36095: Client connection negotiation failed: client connection to 127.28.113.62:36095: Server oryx/127.28.113.62@KRBTEST.COM not found in Kerberos database
Dec 15 21:03:02 dist-test-slave-bjsf krb5kdc[5487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.113.2@KRBTEST.COM for oryx/127.28.113.62@KRBTEST.COM, Server not found in Kerberos database
Dec 15 21:03:02 dist-test-slave-bjsf krb5kdc[5487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.113.2@KRBTEST.COM for oryx/127.28.113.62@KRBTEST.COM, Server not found in Kerberos database
I20251215 21:03:02.031883 6363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:03:02.027220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.113.62:36095 (local address 127.28.113.2:37535)
1215 21:03:02.027456 (+ 236us) negotiation.cc:107] Waiting for socket to connect
1215 21:03:02.027467 (+ 11us) client_negotiation.cc:174] Beginning negotiation
1215 21:03:02.027543 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1215 21:03:02.027776 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1215 21:03:02.027778 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1215 21:03:02.027901 (+ 123us) client_negotiation.cc:189] Negotiated authn=SASL
1215 21:03:02.028132 (+ 231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:03:02.028138 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:03:02.029090 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1215 21:03:02.029093 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1215 21:03:02.029454 (+ 361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:03:02.029460 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:03:02.029543 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:03:02.030023 (+ 480us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1215 21:03:02.030036 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
1215 21:03:02.031753 (+ 1717us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.113.62:36095: Server oryx/127.28.113.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":174}
W20251215 21:03:02.031983 6230 negotiation.cc:337] Failed RPC negotiation. Trace:
1215 21:03:02.027346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.113.2:37535 (local address 127.28.113.62:36095)
1215 21:03:02.027472 (+ 126us) server_negotiation.cc:207] Beginning negotiation
1215 21:03:02.027477 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1215 21:03:02.027509 (+ 32us) server_negotiation.cc:408] Connection header received
1215 21:03:02.027623 (+ 114us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1215 21:03:02.027626 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1215 21:03:02.027679 (+ 53us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1215 21:03:02.027766 (+ 87us) server_negotiation.cc:227] Negotiated authn=SASL
1215 21:03:02.028235 (+ 469us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:03:02.028978 (+ 743us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1215 21:03:02.029552 (+ 574us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:03:02.029743 (+ 191us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:03:02.031866 (+ 2123us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.113.2:37535: BlockingRecv error: recv got EOF from 127.28.113.2:37535 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Dec 15 21:03:02 dist-test-slave-bjsf krb5kdc[5487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.113.2@KRBTEST.COM for oryx/127.28.113.62@KRBTEST.COM, Server not found in Kerberos database
Dec 15 21:03:02 dist-test-slave-bjsf krb5kdc[5487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.113.2@KRBTEST.COM for oryx/127.28.113.62@KRBTEST.COM, Server not found in Kerberos database
I20251215 21:03:02.036571 6363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:03:02.032504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.113.62:36095 (local address 127.28.113.2:51655)
1215 21:03:02.032715 (+ 211us) negotiation.cc:107] Waiting for socket to connect
1215 21:03:02.032728 (+ 13us) client_negotiation.cc:174] Beginning negotiation
1215 21:03:02.032807 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1215 21:03:02.033014 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1215 21:03:02.033017 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1215 21:03:02.033155 (+ 138us) client_negotiation.cc:189] Negotiated authn=SASL
1215 21:03:02.033296 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:03:02.033302 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:03:02.033952 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1215 21:03:02.033955 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1215 21:03:02.034298 (+ 343us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:03:02.034305 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:03:02.034397 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:03:02.034909 (+ 512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1215 21:03:02.034921 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
1215 21:03:02.036459 (+ 1538us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.113.62:36095: Server oryx/127.28.113.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":149}
W20251215 21:03:02.036669 6230 negotiation.cc:337] Failed RPC negotiation. Trace:
1215 21:03:02.032605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.113.2:51655 (local address 127.28.113.62:36095)
1215 21:03:02.032715 (+ 110us) server_negotiation.cc:207] Beginning negotiation
1215 21:03:02.032719 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1215 21:03:02.032767 (+ 48us) server_negotiation.cc:408] Connection header received
1215 21:03:02.032886 (+ 119us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1215 21:03:02.032889 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1215 21:03:02.032929 (+ 40us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1215 21:03:02.032990 (+ 61us) server_negotiation.cc:227] Negotiated authn=SASL
1215 21:03:02.033397 (+ 407us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:03:02.033857 (+ 460us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1215 21:03:02.034397 (+ 540us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:03:02.034569 (+ 172us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:03:02.036571 (+ 2002us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.113.2:51655: BlockingRecv error: recv got EOF from 127.28.113.2:51655 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
W20251215 21:03:02.036885 6361 heartbeater.cc:412] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20251215 21:03:02.126111 6364 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251215 21:03:02.126324 6364 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251215 21:03:02.126386 6364 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251215 21:03:02.129536 6364 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251215 21:03:02.129597 6364 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251215 21:03:02.129683 6364 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.113.3
I20251215 21:03:02.133723 6364 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.113.20:35785
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765832506145157-29124-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765832506145157-29124-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765832506145157-29124-0/krb5kdc/oryx_127.28.113.3.keytab
--principal=oryx/127.28.113.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.113.3:41555
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765832506145157-29124-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.113.3
--webserver_port=35073
--webserver_require_spnego=true
--tserver_master_addrs=127.28.113.62:36095
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6364
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.113.3
--log_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765832506145157-29124-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision dc741732bb6a40213e11a13869fad06a49a209a3
build type DEBUG
built by None at 15 Dec 2025 20:43:14 UTC on 24a791456cd2
build id 9530
I20251215 21:03:02.134701 6364 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251215 21:03:02.135491 6364 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251215 21:03:02.141403 6372 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251215 21:03:02.141484 6370 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251215 21:03:02.141484 6369 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251215 21:03:02.141762 6364 server_base.cc:1047] running on GCE node
I20251215 21:03:02.142091 6364 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251215 21:03:02.142567 6364 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251215 21:03:02.143705 6364 hybrid_clock.cc:648] HybridClock initialized: now 1765832582143693 us; error 33 us; skew 500 ppm
Dec 15 21:03:02 dist-test-slave-bjsf krb5kdc[5487](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765832582, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.113.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251215 21:03:02.146088 6364 init.cc:377] Logged in from keytab as oryx/127.28.113.3@KRBTEST.COM (short username oryx)
I20251215 21:03:02.147084 6364 webserver.cc:492] Webserver started at http://127.28.113.3:35073/ using document root <none> and password file <none>
I20251215 21:03:02.147590 6364 fs_manager.cc:362] Metadata directory not provided
I20251215 21:03:02.147651 6364 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251215 21:03:02.150990 6364 fs_manager.cc:714] Time spent opening directory manager: real 0.002s user 0.000s sys 0.002s
I20251215 21:03:02.152714 6379 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251215 21:03:02.153623 6364 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20251215 21:03:02.153718 6364 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765832506145157-29124-0/minicluster-data/ts-2/data,/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765832506145157-29124-0/minicluster-data/ts-2/wal
uuid: "af9e6a36437a4646808aa957442cb807"
format_stamp: "Formatted at 2025-12-15 21:03:01 on dist-test-slave-bjsf"
server_key: "520ee2b037c512b41a41d3551c1530d1"
server_key_iv: "77e04ea860c4ed725475c5ba22ddc2c5"
server_key_version: "encryptionkey@0"
I20251215 21:03:02.154109 6364 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765832506145157-29124-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765832506145157-29124-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765832506145157-29124-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251215 21:03:02.174458 6364 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251215 21:03:02.177301 6364 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251215 21:03:02.177470 6364 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251215 21:03:02.177953 6364 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251215 21:03:02.178740 6364 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251215 21:03:02.178786 6364 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251215 21:03:02.178830 6364 ts_tablet_manager.cc:616] Registered 0 tablets
I20251215 21:03:02.178850 6364 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251215 21:03:02.187047 6364 rpc_server.cc:307] RPC server started. Bound to: 127.28.113.3:41555
I20251215 21:03:02.187079 6492 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.28.113.3:41555 every 8 connection(s)
I20251215 21:03:02.187996 6364 server_base.cc:1179] Dumped server information to /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765832506145157-29124-0/minicluster-data/ts-2/data/info.pb
Dec 15 21:03:02 dist-test-slave-bjsf krb5kdc[5487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.113.3@KRBTEST.COM for oryx/127.28.113.62@KRBTEST.COM, Server not found in Kerberos database
Dec 15 21:03:02 dist-test-slave-bjsf krb5kdc[5487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.113.3@KRBTEST.COM for oryx/127.28.113.62@KRBTEST.COM, Server not found in Kerberos database
W20251215 21:03:02.196918 6230 negotiation.cc:337] Failed RPC negotiation. Trace:
1215 21:03:02.189566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.113.3:49353 (local address 127.28.113.62:36095)
1215 21:03:02.189749 (+ 183us) server_negotiation.cc:207] Beginning negotiation
1215 21:03:02.189752 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1215 21:03:02.190459 (+ 707us) server_negotiation.cc:408] Connection header received
1215 21:03:02.191335 (+ 876us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1215 21:03:02.191339 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1215 21:03:02.191394 (+ 55us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1215 21:03:02.191461 (+ 67us) server_negotiation.cc:227] Negotiated authn=SASL
1215 21:03:02.193031 (+ 1570us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:03:02.193537 (+ 506us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1215 21:03:02.194136 (+ 599us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:03:02.194301 (+ 165us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:03:02.196824 (+ 2523us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.113.3:49353: BlockingRecv error: recv got EOF from 127.28.113.3:49353 (error 108)
Metrics: {"server-negotiator.queue_time_us":94}
I20251215 21:03:02.197283 6495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:03:02.189838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.113.62:36095 (local address 127.28.113.3:49353)
1215 21:03:02.190330 (+ 492us) negotiation.cc:107] Waiting for socket to connect
1215 21:03:02.190373 (+ 43us) client_negotiation.cc:174] Beginning negotiation
1215 21:03:02.191161 (+ 788us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1215 21:03:02.191613 (+ 452us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1215 21:03:02.191621 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
1215 21:03:02.192008 (+ 387us) client_negotiation.cc:189] Negotiated authn=SASL
1215 21:03:02.192891 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:03:02.192913 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:03:02.193640 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1215 21:03:02.193643 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1215 21:03:02.194036 (+ 393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:03:02.194043 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:03:02.194231 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:03:02.194763 (+ 532us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1215 21:03:02.194784 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
1215 21:03:02.196711 (+ 1927us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.113.62:36095: Server oryx/127.28.113.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":114,"threads_started":1}
W20251215 21:03:02.197604 6493 heartbeater.cc:646] Failed to heartbeat to 127.28.113.62:36095 (0 consecutive failures): Not authorized: Failed to ping master at 127.28.113.62:36095: Client connection negotiation failed: client connection to 127.28.113.62:36095: Server oryx/127.28.113.62@KRBTEST.COM not found in Kerberos database
I20251215 21:03:02.197873 29124 external_mini_cluster.cc:1428] Started /tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu as pid 6364
Dec 15 21:03:02 dist-test-slave-bjsf krb5kdc[5487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.113.3@KRBTEST.COM for oryx/127.28.113.62@KRBTEST.COM, Server not found in Kerberos database
Dec 15 21:03:02 dist-test-slave-bjsf krb5kdc[5487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.113.3@KRBTEST.COM for oryx/127.28.113.62@KRBTEST.COM, Server not found in Kerberos database
I20251215 21:03:02.202139 6495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:03:02.197949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.113.62:36095 (local address 127.28.113.3:60221)
1215 21:03:02.198112 (+ 163us) negotiation.cc:107] Waiting for socket to connect
1215 21:03:02.198125 (+ 13us) client_negotiation.cc:174] Beginning negotiation
1215 21:03:02.198190 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1215 21:03:02.198491 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1215 21:03:02.198494 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1215 21:03:02.198642 (+ 148us) client_negotiation.cc:189] Negotiated authn=SASL
1215 21:03:02.198826 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:03:02.198833 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:03:02.199460 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1215 21:03:02.199464 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1215 21:03:02.199831 (+ 367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:03:02.199839 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:03:02.199932 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:03:02.200438 (+ 506us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1215 21:03:02.200453 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
1215 21:03:02.202036 (+ 1583us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.113.62:36095: Server oryx/127.28.113.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":92}
W20251215 21:03:02.202231 6230 negotiation.cc:337] Failed RPC negotiation. Trace:
1215 21:03:02.198207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.113.3:60221 (local address 127.28.113.62:36095)
1215 21:03:02.198309 (+ 102us) server_negotiation.cc:207] Beginning negotiation
1215 21:03:02.198312 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1215 21:03:02.198323 (+ 11us) server_negotiation.cc:408] Connection header received
1215 21:03:02.198355 (+ 32us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1215 21:03:02.198357 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1215 21:03:02.198398 (+ 41us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1215 21:03:02.198464 (+ 66us) server_negotiation.cc:227] Negotiated authn=SASL
1215 21:03:02.198931 (+ 467us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:03:02.199365 (+ 434us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1215 21:03:02.199931 (+ 566us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:03:02.200136 (+ 205us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:03:02.202142 (+ 2006us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.113.3:60221: BlockingRecv error: recv got EOF from 127.28.113.3:60221 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
Dec 15 21:03:02 dist-test-slave-bjsf krb5kdc[5487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.113.3@KRBTEST.COM for oryx/127.28.113.62@KRBTEST.COM, Server not found in Kerberos database
I20251215 21:03:02.206051 6496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:03:02.198440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56650 (local address 127.28.113.62:36095)
1215 21:03:02.198643 (+ 203us) server_negotiation.cc:207] Beginning negotiation
1215 21:03:02.198647 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1215 21:03:02.198783 (+ 136us) server_negotiation.cc:408] Connection header received
1215 21:03:02.198890 (+ 107us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1215 21:03:02.198894 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1215 21:03:02.198930 (+ 36us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1215 21:03:02.198997 (+ 67us) server_negotiation.cc:227] Negotiated authn=SASL
1215 21:03:02.199741 (+ 744us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:03:02.200343 (+ 602us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1215 21:03:02.201171 (+ 828us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:03:02.201322 (+ 151us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:03:02.202317 (+ 995us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1215 21:03:02.202334 (+ 17us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1215 21:03:02.202347 (+ 13us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1215 21:03:02.202391 (+ 44us) server_negotiation.cc:923] calling sasl_server_start()
1215 21:03:02.204165 (+ 1774us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1215 21:03:02.204521 (+ 356us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1215 21:03:02.204528 (+ 7us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1215 21:03:02.204534 (+ 6us) server_negotiation.cc:968] calling sasl_server_step()
1215 21:03:02.204634 (+ 100us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1215 21:03:02.204897 (+ 263us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1215 21:03:02.204901 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1215 21:03:02.204903 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1215 21:03:02.205227 (+ 324us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1215 21:03:02.205417 (+ 190us) server_negotiation.cc:1036] Waiting for connection context
1215 21:03:02.205704 (+ 287us) server_negotiation.cc:300] Negotiation successful
1215 21:03:02.205937 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":69,"threads_started":1}
Dec 15 21:03:02 dist-test-slave-bjsf krb5kdc[5487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.113.3@KRBTEST.COM for oryx/127.28.113.62@KRBTEST.COM, Server not found in Kerberos database
I20251215 21:03:02.206599 6495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:03:02.202633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.113.62:36095 (local address 127.28.113.3:32939)
1215 21:03:02.202731 (+ 98us) negotiation.cc:107] Waiting for socket to connect
1215 21:03:02.202744 (+ 13us) client_negotiation.cc:174] Beginning negotiation
1215 21:03:02.202811 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1215 21:03:02.203103 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1215 21:03:02.203106 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1215 21:03:02.203223 (+ 117us) client_negotiation.cc:189] Negotiated authn=SASL
1215 21:03:02.203417 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:03:02.203426 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:03:02.204090 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1215 21:03:02.204094 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1215 21:03:02.204538 (+ 444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:03:02.204547 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:03:02.204652 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:03:02.205078 (+ 426us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1215 21:03:02.205090 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
1215 21:03:02.206519 (+ 1429us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.113.62:36095: Server oryx/127.28.113.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":24}
W20251215 21:03:02.206722 6230 negotiation.cc:337] Failed RPC negotiation. Trace:
1215 21:03:02.202701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.113.3:32939 (local address 127.28.113.62:36095)
1215 21:03:02.202928 (+ 227us) server_negotiation.cc:207] Beginning negotiation
1215 21:03:02.202931 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1215 21:03:02.202942 (+ 11us) server_negotiation.cc:408] Connection header received
1215 21:03:02.202966 (+ 24us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1215 21:03:02.202968 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1215 21:03:02.202998 (+ 30us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1215 21:03:02.203054 (+ 56us) server_negotiation.cc:227] Negotiated authn=SASL
1215 21:03:02.203535 (+ 481us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:03:02.203988 (+ 453us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1215 21:03:02.204640 (+ 652us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:03:02.204795 (+ 155us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:03:02.206634 (+ 1839us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.113.3:32939: BlockingRecv error: recv got EOF from 127.28.113.3:32939 (error 108)
Metrics: {"server-negotiator.queue_time_us":158}
W20251215 21:03:02.206952 6493 heartbeater.cc:412] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Dec 15 21:03:02 dist-test-slave-bjsf krb5kdc[5487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.113.1@KRBTEST.COM for oryx/127.28.113.62@KRBTEST.COM, Server not found in Kerberos database
Dec 15 21:03:02 dist-test-slave-bjsf krb5kdc[5487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.113.1@KRBTEST.COM for oryx/127.28.113.62@KRBTEST.COM, Server not found in Kerberos database
I20251215 21:03:02.879976 6498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:03:02.875513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.113.62:36095 (local address 127.28.113.1:54783)
1215 21:03:02.875754 (+ 241us) negotiation.cc:107] Waiting for socket to connect
1215 21:03:02.875767 (+ 13us) client_negotiation.cc:174] Beginning negotiation
1215 21:03:02.875858 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1215 21:03:02.876106 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1215 21:03:02.876109 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1215 21:03:02.876286 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
1215 21:03:02.876502 (+ 216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:03:02.876508 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:03:02.877152 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1215 21:03:02.877154 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1215 21:03:02.877548 (+ 394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:03:02.877554 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:03:02.877634 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:03:02.878170 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1215 21:03:02.878182 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
1215 21:03:02.879796 (+ 1614us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.113.62:36095: Server oryx/127.28.113.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":69,"threads_started":1}
W20251215 21:03:02.880081 6499 negotiation.cc:337] Failed RPC negotiation. Trace:
1215 21:03:02.875563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.113.1:54783 (local address 127.28.113.62:36095)
1215 21:03:02.875795 (+ 232us) server_negotiation.cc:207] Beginning negotiation
1215 21:03:02.875798 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1215 21:03:02.875811 (+ 13us) server_negotiation.cc:408] Connection header received
1215 21:03:02.875929 (+ 118us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1215 21:03:02.875931 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1215 21:03:02.875991 (+ 60us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1215 21:03:02.876059 (+ 68us) server_negotiation.cc:227] Negotiated authn=SASL
1215 21:03:02.876587 (+ 528us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:03:02.877080 (+ 493us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1215 21:03:02.877622 (+ 542us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:03:02.877742 (+ 120us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:03:02.879924 (+ 2182us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.113.1:54783: BlockingRecv error: recv got EOF from 127.28.113.1:54783 (error 108)
Metrics: {"server-negotiator.queue_time_us":166,"thread_start_us":80,"threads_started":1}
Dec 15 21:03:03 dist-test-slave-bjsf krb5kdc[5487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.113.2@KRBTEST.COM for oryx/127.28.113.62@KRBTEST.COM, Server not found in Kerberos database
Dec 15 21:03:03 dist-test-slave-bjsf krb5kdc[5487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.113.2@KRBTEST.COM for oryx/127.28.113.62@KRBTEST.COM, Server not found in Kerberos database
I20251215 21:03:03.041908 6500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:03:03.037442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.113.62:36095 (local address 127.28.113.2:40935)
1215 21:03:03.037703 (+ 261us) negotiation.cc:107] Waiting for socket to connect
1215 21:03:03.037715 (+ 12us) client_negotiation.cc:174] Beginning negotiation
1215 21:03:03.037820 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1215 21:03:03.038067 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1215 21:03:03.038070 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1215 21:03:03.038263 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
1215 21:03:03.038450 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:03:03.038456 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:03:03.039149 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1215 21:03:03.039153 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1215 21:03:03.039535 (+ 382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:03:03.039541 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:03:03.039638 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:03:03.040199 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1215 21:03:03.040212 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
1215 21:03:03.041783 (+ 1571us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.113.62:36095: Server oryx/127.28.113.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":85,"threads_started":1}
W20251215 21:03:03.041999 6499 negotiation.cc:337] Failed RPC negotiation. Trace:
1215 21:03:03.037512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.113.2:40935 (local address 127.28.113.62:36095)
1215 21:03:03.037644 (+ 132us) server_negotiation.cc:207] Beginning negotiation
1215 21:03:03.037649 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1215 21:03:03.037776 (+ 127us) server_negotiation.cc:408] Connection header received
1215 21:03:03.037901 (+ 125us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1215 21:03:03.037905 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1215 21:03:03.037953 (+ 48us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1215 21:03:03.038015 (+ 62us) server_negotiation.cc:227] Negotiated authn=SASL
1215 21:03:03.038561 (+ 546us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:03:03.039052 (+ 491us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1215 21:03:03.039637 (+ 585us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:03:03.039812 (+ 175us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:03:03.041903 (+ 2091us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.113.2:40935: BlockingRecv error: recv got EOF from 127.28.113.2:40935 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
Dec 15 21:03:03 dist-test-slave-bjsf krb5kdc[5487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.113.3@KRBTEST.COM for oryx/127.28.113.62@KRBTEST.COM, Server not found in Kerberos database
Dec 15 21:03:03 dist-test-slave-bjsf krb5kdc[5487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.113.3@KRBTEST.COM for oryx/127.28.113.62@KRBTEST.COM, Server not found in Kerberos database
I20251215 21:03:03.211889 6501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:03:03.207500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.113.62:36095 (local address 127.28.113.3:59469)
1215 21:03:03.207753 (+ 253us) negotiation.cc:107] Waiting for socket to connect
1215 21:03:03.207765 (+ 12us) client_negotiation.cc:174] Beginning negotiation
1215 21:03:03.207867 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1215 21:03:03.208135 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1215 21:03:03.208138 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1215 21:03:03.208323 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
1215 21:03:03.208507 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:03:03.208514 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:03:03.209175 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1215 21:03:03.209179 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1215 21:03:03.209554 (+ 375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:03:03.209561 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:03:03.209654 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:03:03.210181 (+ 527us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1215 21:03:03.210193 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
1215 21:03:03.211756 (+ 1563us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.113.62:36095: Server oryx/127.28.113.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":85,"threads_started":1}
W20251215 21:03:03.211997 6499 negotiation.cc:337] Failed RPC negotiation. Trace:
1215 21:03:03.207576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.113.3:59469 (local address 127.28.113.62:36095)
1215 21:03:03.207709 (+ 133us) server_negotiation.cc:207] Beginning negotiation
1215 21:03:03.207714 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1215 21:03:03.207824 (+ 110us) server_negotiation.cc:408] Connection header received
1215 21:03:03.207949 (+ 125us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1215 21:03:03.207953 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1215 21:03:03.208022 (+ 69us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1215 21:03:03.208086 (+ 64us) server_negotiation.cc:227] Negotiated authn=SASL
1215 21:03:03.208613 (+ 527us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:03:03.209082 (+ 469us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1215 21:03:03.209652 (+ 570us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:03:03.209821 (+ 169us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:03:03.211876 (+ 2055us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.113.3:59469: BlockingRecv error: recv got EOF from 127.28.113.3:59469 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Dec 15 21:03:03 dist-test-slave-bjsf krb5kdc[5487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.113.1@KRBTEST.COM for oryx/127.28.113.62@KRBTEST.COM, Server not found in Kerberos database
Dec 15 21:03:03 dist-test-slave-bjsf krb5kdc[5487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.113.1@KRBTEST.COM for oryx/127.28.113.62@KRBTEST.COM, Server not found in Kerberos database
I20251215 21:03:03.885313 6502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:03:03.880762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.113.62:36095 (local address 127.28.113.1:45677)
1215 21:03:03.881023 (+ 261us) negotiation.cc:107] Waiting for socket to connect
1215 21:03:03.881036 (+ 13us) client_negotiation.cc:174] Beginning negotiation
1215 21:03:03.881132 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1215 21:03:03.881351 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1215 21:03:03.881353 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1215 21:03:03.881534 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
1215 21:03:03.881735 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:03:03.881740 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:03:03.882420 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1215 21:03:03.882423 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1215 21:03:03.882836 (+ 413us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:03:03.882844 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:03:03.882931 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:03:03.883451 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1215 21:03:03.883466 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
1215 21:03:03.885193 (+ 1727us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.113.62:36095: Server oryx/127.28.113.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":77,"threads_started":1}
W20251215 21:03:03.885442 6503 negotiation.cc:337] Failed RPC negotiation. Trace:
1215 21:03:03.880835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.113.1:45677 (local address 127.28.113.62:36095)
1215 21:03:03.881066 (+ 231us) server_negotiation.cc:207] Beginning negotiation
1215 21:03:03.881070 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1215 21:03:03.881082 (+ 12us) server_negotiation.cc:408] Connection header received
1215 21:03:03.881207 (+ 125us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1215 21:03:03.881209 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1215 21:03:03.881254 (+ 45us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1215 21:03:03.881325 (+ 71us) server_negotiation.cc:227] Negotiated authn=SASL
1215 21:03:03.881818 (+ 493us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:03:03.882325 (+ 507us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1215 21:03:03.882918 (+ 593us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:03:03.883041 (+ 123us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:03:03.885326 (+ 2285us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.113.1:45677: BlockingRecv error: recv got EOF from 127.28.113.1:45677 (error 108)
Metrics: {"server-negotiator.queue_time_us":164,"thread_start_us":82,"threads_started":1}
Dec 15 21:03:04 dist-test-slave-bjsf krb5kdc[5487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.113.2@KRBTEST.COM for oryx/127.28.113.62@KRBTEST.COM, Server not found in Kerberos database
Dec 15 21:03:04 dist-test-slave-bjsf krb5kdc[5487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.113.2@KRBTEST.COM for oryx/127.28.113.62@KRBTEST.COM, Server not found in Kerberos database
I20251215 21:03:04.046844 6504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:03:04.042580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.113.62:36095 (local address 127.28.113.2:52475)
1215 21:03:04.042839 (+ 259us) negotiation.cc:107] Waiting for socket to connect
1215 21:03:04.042852 (+ 13us) client_negotiation.cc:174] Beginning negotiation
1215 21:03:04.042960 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1215 21:03:04.043162 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1215 21:03:04.043165 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1215 21:03:04.043329 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
1215 21:03:04.043517 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:03:04.043522 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:03:04.044195 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1215 21:03:04.044197 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1215 21:03:04.044565 (+ 368us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:03:04.044572 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:03:04.044658 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:03:04.045143 (+ 485us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1215 21:03:04.045157 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
1215 21:03:04.046732 (+ 1575us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.113.62:36095: Server oryx/127.28.113.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":84,"threads_started":1}
W20251215 21:03:04.047006 6503 negotiation.cc:337] Failed RPC negotiation. Trace:
1215 21:03:04.042643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.113.2:52475 (local address 127.28.113.62:36095)
1215 21:03:04.042805 (+ 162us) server_negotiation.cc:207] Beginning negotiation
1215 21:03:04.042810 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1215 21:03:04.042915 (+ 105us) server_negotiation.cc:408] Connection header received
1215 21:03:04.043031 (+ 116us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1215 21:03:04.043033 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1215 21:03:04.043086 (+ 53us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1215 21:03:04.043150 (+ 64us) server_negotiation.cc:227] Negotiated authn=SASL
1215 21:03:04.043599 (+ 449us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:03:04.044116 (+ 517us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1215 21:03:04.044645 (+ 529us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:03:04.044764 (+ 119us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:03:04.046895 (+ 2131us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.113.2:52475: BlockingRecv error: recv got EOF from 127.28.113.2:52475 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Dec 15 21:03:04 dist-test-slave-bjsf krb5kdc[5487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.113.3@KRBTEST.COM for oryx/127.28.113.62@KRBTEST.COM, Server not found in Kerberos database
Dec 15 21:03:04 dist-test-slave-bjsf krb5kdc[5487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.113.3@KRBTEST.COM for oryx/127.28.113.62@KRBTEST.COM, Server not found in Kerberos database
I20251215 21:03:04.217263 6505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:03:04.212624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.113.62:36095 (local address 127.28.113.3:36823)
1215 21:03:04.212884 (+ 260us) negotiation.cc:107] Waiting for socket to connect
1215 21:03:04.212898 (+ 14us) client_negotiation.cc:174] Beginning negotiation
1215 21:03:04.213008 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1215 21:03:04.213231 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1215 21:03:04.213234 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1215 21:03:04.213414 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
1215 21:03:04.213608 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:03:04.213614 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:03:04.214296 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1215 21:03:04.214299 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1215 21:03:04.214686 (+ 387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:03:04.214692 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:03:04.214786 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:03:04.215373 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1215 21:03:04.215390 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
1215 21:03:04.217149 (+ 1759us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.113.62:36095: Server oryx/127.28.113.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":93,"threads_started":1}
W20251215 21:03:04.217373 6503 negotiation.cc:337] Failed RPC negotiation. Trace:
1215 21:03:04.212742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.113.3:36823 (local address 127.28.113.62:36095)
1215 21:03:04.212885 (+ 143us) server_negotiation.cc:207] Beginning negotiation
1215 21:03:04.212889 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1215 21:03:04.212962 (+ 73us) server_negotiation.cc:408] Connection header received
1215 21:03:04.213082 (+ 120us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1215 21:03:04.213085 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1215 21:03:04.213134 (+ 49us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1215 21:03:04.213202 (+ 68us) server_negotiation.cc:227] Negotiated authn=SASL
1215 21:03:04.213712 (+ 510us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:03:04.214200 (+ 488us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1215 21:03:04.214784 (+ 584us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:03:04.214980 (+ 196us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:03:04.217268 (+ 2288us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.113.3:36823: BlockingRecv error: recv got EOF from 127.28.113.3:36823 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Dec 15 21:03:04 dist-test-slave-bjsf krb5kdc[5487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.113.1@KRBTEST.COM for oryx/127.28.113.62@KRBTEST.COM, Server not found in Kerberos database
Dec 15 21:03:04 dist-test-slave-bjsf krb5kdc[5487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.113.1@KRBTEST.COM for oryx/127.28.113.62@KRBTEST.COM, Server not found in Kerberos database
I20251215 21:03:04.890304 6506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:03:04.885981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.113.62:36095 (local address 127.28.113.1:36771)
1215 21:03:04.886262 (+ 281us) negotiation.cc:107] Waiting for socket to connect
1215 21:03:04.886273 (+ 11us) client_negotiation.cc:174] Beginning negotiation
1215 21:03:04.886366 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1215 21:03:04.886571 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1215 21:03:04.886573 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1215 21:03:04.886736 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
1215 21:03:04.886927 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:03:04.886932 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:03:04.887588 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1215 21:03:04.887590 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1215 21:03:04.887948 (+ 358us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:03:04.887953 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:03:04.888068 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:03:04.888562 (+ 494us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1215 21:03:04.888575 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
1215 21:03:04.890199 (+ 1624us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.113.62:36095: Server oryx/127.28.113.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":96,"threads_started":1}
W20251215 21:03:04.890445 6507 negotiation.cc:337] Failed RPC negotiation. Trace:
1215 21:03:04.886043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.113.1:36771 (local address 127.28.113.62:36095)
1215 21:03:04.886266 (+ 223us) server_negotiation.cc:207] Beginning negotiation
1215 21:03:04.886269 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1215 21:03:04.886331 (+ 62us) server_negotiation.cc:408] Connection header received
1215 21:03:04.886434 (+ 103us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1215 21:03:04.886437 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1215 21:03:04.886481 (+ 44us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1215 21:03:04.886552 (+ 71us) server_negotiation.cc:227] Negotiated authn=SASL
1215 21:03:04.887008 (+ 456us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:03:04.887509 (+ 501us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1215 21:03:04.888089 (+ 580us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:03:04.888229 (+ 140us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:03:04.890320 (+ 2091us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.113.1:36771: BlockingRecv error: recv got EOF from 127.28.113.1:36771 (error 108)
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":57,"threads_started":1}
Dec 15 21:03:05 dist-test-slave-bjsf krb5kdc[5487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.113.2@KRBTEST.COM for oryx/127.28.113.62@KRBTEST.COM, Server not found in Kerberos database
Dec 15 21:03:05 dist-test-slave-bjsf krb5kdc[5487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.113.2@KRBTEST.COM for oryx/127.28.113.62@KRBTEST.COM, Server not found in Kerberos database
I20251215 21:03:05.052489 6508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:03:05.047604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.113.62:36095 (local address 127.28.113.2:33291)
1215 21:03:05.047894 (+ 290us) negotiation.cc:107] Waiting for socket to connect
1215 21:03:05.047906 (+ 12us) client_negotiation.cc:174] Beginning negotiation
1215 21:03:05.048033 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1215 21:03:05.048265 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1215 21:03:05.048268 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1215 21:03:05.048458 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
1215 21:03:05.048644 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:03:05.048651 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:03:05.049364 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1215 21:03:05.049368 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1215 21:03:05.049732 (+ 364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:03:05.049738 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:03:05.049859 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:03:05.050407 (+ 548us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1215 21:03:05.050420 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
1215 21:03:05.052371 (+ 1951us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.113.62:36095: Server oryx/127.28.113.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":73,"threads_started":1}
W20251215 21:03:05.052589 6507 negotiation.cc:337] Failed RPC negotiation. Trace:
1215 21:03:05.047682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.113.2:33291 (local address 127.28.113.62:36095)
1215 21:03:05.047816 (+ 134us) server_negotiation.cc:207] Beginning negotiation
1215 21:03:05.047820 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1215 21:03:05.047974 (+ 154us) server_negotiation.cc:408] Connection header received
1215 21:03:05.048124 (+ 150us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1215 21:03:05.048127 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1215 21:03:05.048170 (+ 43us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1215 21:03:05.048234 (+ 64us) server_negotiation.cc:227] Negotiated authn=SASL
1215 21:03:05.048765 (+ 531us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:03:05.049262 (+ 497us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1215 21:03:05.049837 (+ 575us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:03:05.050014 (+ 177us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:03:05.052491 (+ 2477us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.113.2:33291: BlockingRecv error: recv got EOF from 127.28.113.2:33291 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Dec 15 21:03:05 dist-test-slave-bjsf krb5kdc[5487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.113.3@KRBTEST.COM for oryx/127.28.113.62@KRBTEST.COM, Server not found in Kerberos database
Dec 15 21:03:05 dist-test-slave-bjsf krb5kdc[5487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.113.3@KRBTEST.COM for oryx/127.28.113.62@KRBTEST.COM, Server not found in Kerberos database
I20251215 21:03:05.222610 6509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:03:05.217909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.113.62:36095 (local address 127.28.113.3:43327)
1215 21:03:05.218146 (+ 237us) negotiation.cc:107] Waiting for socket to connect
1215 21:03:05.218159 (+ 13us) client_negotiation.cc:174] Beginning negotiation
1215 21:03:05.218263 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1215 21:03:05.218478 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1215 21:03:05.218480 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1215 21:03:05.218641 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
1215 21:03:05.218851 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:03:05.218859 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:03:05.219674 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1215 21:03:05.219678 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1215 21:03:05.220093 (+ 415us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:03:05.220101 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:03:05.220206 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:03:05.220715 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1215 21:03:05.220728 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
1215 21:03:05.222496 (+ 1768us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.113.62:36095: Server oryx/127.28.113.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":67,"threads_started":1}
W20251215 21:03:05.222790 6507 negotiation.cc:337] Failed RPC negotiation. Trace:
1215 21:03:05.217960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.113.3:43327 (local address 127.28.113.62:36095)
1215 21:03:05.218097 (+ 137us) server_negotiation.cc:207] Beginning negotiation
1215 21:03:05.218100 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1215 21:03:05.218226 (+ 126us) server_negotiation.cc:408] Connection header received
1215 21:03:05.218335 (+ 109us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1215 21:03:05.218337 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1215 21:03:05.218387 (+ 50us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1215 21:03:05.218456 (+ 69us) server_negotiation.cc:227] Negotiated authn=SASL
1215 21:03:05.218952 (+ 496us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:03:05.219569 (+ 617us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1215 21:03:05.220181 (+ 612us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:03:05.220321 (+ 140us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:03:05.222644 (+ 2323us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.113.3:43327: BlockingRecv error: recv got EOF from 127.28.113.3:43327 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Dec 15 21:03:05 dist-test-slave-bjsf krb5kdc[5487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.113.1@KRBTEST.COM for oryx/127.28.113.62@KRBTEST.COM, Server not found in Kerberos database
Dec 15 21:03:05 dist-test-slave-bjsf krb5kdc[5487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.113.1@KRBTEST.COM for oryx/127.28.113.62@KRBTEST.COM, Server not found in Kerberos database
I20251215 21:03:05.899032 6510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:03:05.891058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.113.62:36095 (local address 127.28.113.1:41047)
1215 21:03:05.891313 (+ 255us) negotiation.cc:107] Waiting for socket to connect
1215 21:03:05.891325 (+ 12us) client_negotiation.cc:174] Beginning negotiation
1215 21:03:05.891418 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1215 21:03:05.891712 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1215 21:03:05.891714 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1215 21:03:05.891875 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
1215 21:03:05.892107 (+ 232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:03:05.892114 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:03:05.893071 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1215 21:03:05.893074 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1215 21:03:05.893611 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:03:05.893622 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:03:05.893734 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:03:05.897267 (+ 3533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1215 21:03:05.897280 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
1215 21:03:05.898890 (+ 1610us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.113.62:36095: Server oryx/127.28.113.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":75,"threads_started":1}
W20251215 21:03:05.899125 6511 negotiation.cc:337] Failed RPC negotiation. Trace:
1215 21:03:05.891147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.113.1:41047 (local address 127.28.113.62:36095)
1215 21:03:05.891373 (+ 226us) server_negotiation.cc:207] Beginning negotiation
1215 21:03:05.891377 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1215 21:03:05.891391 (+ 14us) server_negotiation.cc:408] Connection header received
1215 21:03:05.891534 (+ 143us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1215 21:03:05.891538 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1215 21:03:05.891598 (+ 60us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1215 21:03:05.891683 (+ 85us) server_negotiation.cc:227] Negotiated authn=SASL
1215 21:03:05.892219 (+ 536us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:03:05.892948 (+ 729us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1215 21:03:05.893785 (+ 837us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:03:05.893992 (+ 207us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:03:05.899020 (+ 5028us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.113.1:41047: BlockingRecv error: recv got EOF from 127.28.113.1:41047 (error 108)
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":67,"threads_started":1}
Dec 15 21:03:06 dist-test-slave-bjsf krb5kdc[5487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.113.2@KRBTEST.COM for oryx/127.28.113.62@KRBTEST.COM, Server not found in Kerberos database
Dec 15 21:03:06 dist-test-slave-bjsf krb5kdc[5487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.113.2@KRBTEST.COM for oryx/127.28.113.62@KRBTEST.COM, Server not found in Kerberos database
I20251215 21:03:06.057384 6512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:03:06.053150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.113.62:36095 (local address 127.28.113.2:46325)
1215 21:03:06.053401 (+ 251us) negotiation.cc:107] Waiting for socket to connect
1215 21:03:06.053413 (+ 12us) client_negotiation.cc:174] Beginning negotiation
1215 21:03:06.053507 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1215 21:03:06.053703 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1215 21:03:06.053705 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1215 21:03:06.053866 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
1215 21:03:06.054052 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:03:06.054057 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:03:06.054709 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1215 21:03:06.054711 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1215 21:03:06.055065 (+ 354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:03:06.055070 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:03:06.055144 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:03:06.055628 (+ 484us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1215 21:03:06.055641 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
1215 21:03:06.057274 (+ 1633us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.113.62:36095: Server oryx/127.28.113.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":65,"threads_started":1}
W20251215 21:03:06.057483 6511 negotiation.cc:337] Failed RPC negotiation. Trace:
1215 21:03:06.053219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.113.2:46325 (local address 127.28.113.62:36095)
1215 21:03:06.053354 (+ 135us) server_negotiation.cc:207] Beginning negotiation
1215 21:03:06.053359 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1215 21:03:06.053471 (+ 112us) server_negotiation.cc:408] Connection header received
1215 21:03:06.053574 (+ 103us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1215 21:03:06.053576 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1215 21:03:06.053620 (+ 44us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1215 21:03:06.053683 (+ 63us) server_negotiation.cc:227] Negotiated authn=SASL
1215 21:03:06.054152 (+ 469us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:03:06.054637 (+ 485us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1215 21:03:06.055149 (+ 512us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:03:06.055275 (+ 126us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:03:06.057386 (+ 2111us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.113.2:46325: BlockingRecv error: recv got EOF from 127.28.113.2:46325 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Dec 15 21:03:06 dist-test-slave-bjsf krb5kdc[5487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.113.3@KRBTEST.COM for oryx/127.28.113.62@KRBTEST.COM, Server not found in Kerberos database
Dec 15 21:03:06 dist-test-slave-bjsf krb5kdc[5487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.113.3@KRBTEST.COM for oryx/127.28.113.62@KRBTEST.COM, Server not found in Kerberos database
I20251215 21:03:06.227641 6513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:03:06.223249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.113.62:36095 (local address 127.28.113.3:35185)
1215 21:03:06.223522 (+ 273us) negotiation.cc:107] Waiting for socket to connect
1215 21:03:06.223534 (+ 12us) client_negotiation.cc:174] Beginning negotiation
1215 21:03:06.223629 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1215 21:03:06.223848 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1215 21:03:06.223850 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1215 21:03:06.224050 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
1215 21:03:06.224244 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:03:06.224251 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:03:06.224935 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1215 21:03:06.224937 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1215 21:03:06.225296 (+ 359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:03:06.225302 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:03:06.225378 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:03:06.225858 (+ 480us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1215 21:03:06.225871 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
1215 21:03:06.227497 (+ 1626us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.113.62:36095: Server oryx/127.28.113.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":77,"threads_started":1}
W20251215 21:03:06.227710 6511 negotiation.cc:337] Failed RPC negotiation. Trace:
1215 21:03:06.223330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.113.3:35185 (local address 127.28.113.62:36095)
1215 21:03:06.223489 (+ 159us) server_negotiation.cc:207] Beginning negotiation
1215 21:03:06.223493 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1215 21:03:06.223631 (+ 138us) server_negotiation.cc:408] Connection header received
1215 21:03:06.223698 (+ 67us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1215 21:03:06.223701 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1215 21:03:06.223756 (+ 55us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1215 21:03:06.223836 (+ 80us) server_negotiation.cc:227] Negotiated authn=SASL
1215 21:03:06.224359 (+ 523us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:03:06.224851 (+ 492us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1215 21:03:06.225392 (+ 541us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:03:06.225526 (+ 134us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:03:06.227617 (+ 2091us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.113.3:35185: BlockingRecv error: recv got EOF from 127.28.113.3:35185 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Dec 15 21:03:06 dist-test-slave-bjsf krb5kdc[5487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.113.1@KRBTEST.COM for oryx/127.28.113.62@KRBTEST.COM, Server not found in Kerberos database
Dec 15 21:03:06 dist-test-slave-bjsf krb5kdc[5487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.113.1@KRBTEST.COM for oryx/127.28.113.62@KRBTEST.COM, Server not found in Kerberos database
I20251215 21:03:06.904721 6514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:03:06.899802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.113.62:36095 (local address 127.28.113.1:36443)
1215 21:03:06.900123 (+ 321us) negotiation.cc:107] Waiting for socket to connect
1215 21:03:06.900137 (+ 14us) client_negotiation.cc:174] Beginning negotiation
1215 21:03:06.900243 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1215 21:03:06.900479 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1215 21:03:06.900482 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1215 21:03:06.900660 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
1215 21:03:06.900861 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:03:06.900868 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:03:06.901586 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1215 21:03:06.901589 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1215 21:03:06.901970 (+ 381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:03:06.901977 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:03:06.902077 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:03:06.902722 (+ 645us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1215 21:03:06.902735 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
1215 21:03:06.904580 (+ 1845us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.113.62:36095: Server oryx/127.28.113.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":101,"threads_started":1}
W20251215 21:03:06.904814 6515 negotiation.cc:337] Failed RPC negotiation. Trace:
1215 21:03:06.899834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.113.1:36443 (local address 127.28.113.62:36095)
1215 21:03:06.900119 (+ 285us) server_negotiation.cc:207] Beginning negotiation
1215 21:03:06.900124 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1215 21:03:06.900201 (+ 77us) server_negotiation.cc:408] Connection header received
1215 21:03:06.900336 (+ 135us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1215 21:03:06.900339 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1215 21:03:06.900381 (+ 42us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1215 21:03:06.900448 (+ 67us) server_negotiation.cc:227] Negotiated authn=SASL
1215 21:03:06.900972 (+ 524us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:03:06.901484 (+ 512us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1215 21:03:06.902073 (+ 589us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:03:06.902251 (+ 178us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:03:06.904708 (+ 2457us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.113.1:36443: BlockingRecv error: recv got EOF from 127.28.113.1:36443 (error 108)
Metrics: {"server-negotiator.queue_time_us":212,"thread_start_us":67,"threads_started":1}
Dec 15 21:03:07 dist-test-slave-bjsf krb5kdc[5487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.113.2@KRBTEST.COM for oryx/127.28.113.62@KRBTEST.COM, Server not found in Kerberos database
Dec 15 21:03:07 dist-test-slave-bjsf krb5kdc[5487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.113.2@KRBTEST.COM for oryx/127.28.113.62@KRBTEST.COM, Server not found in Kerberos database
I20251215 21:03:07.062354 6516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:03:07.058163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.113.62:36095 (local address 127.28.113.2:58085)
1215 21:03:07.058407 (+ 244us) negotiation.cc:107] Waiting for socket to connect
1215 21:03:07.058421 (+ 14us) client_negotiation.cc:174] Beginning negotiation
1215 21:03:07.058511 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1215 21:03:07.058708 (+ 197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1215 21:03:07.058711 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1215 21:03:07.058910 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
1215 21:03:07.059115 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:03:07.059122 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:03:07.059765 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1215 21:03:07.059768 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1215 21:03:07.060184 (+ 416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:03:07.060192 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:03:07.060277 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:03:07.060744 (+ 467us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1215 21:03:07.060755 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
1215 21:03:07.062248 (+ 1493us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.113.62:36095: Server oryx/127.28.113.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":80,"threads_started":1}
W20251215 21:03:07.062464 6515 negotiation.cc:337] Failed RPC negotiation. Trace:
1215 21:03:07.058298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.113.2:58085 (local address 127.28.113.62:36095)
1215 21:03:07.058425 (+ 127us) server_negotiation.cc:207] Beginning negotiation
1215 21:03:07.058429 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1215 21:03:07.058480 (+ 51us) server_negotiation.cc:408] Connection header received
1215 21:03:07.058578 (+ 98us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1215 21:03:07.058580 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1215 21:03:07.058623 (+ 43us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1215 21:03:07.058684 (+ 61us) server_negotiation.cc:227] Negotiated authn=SASL
1215 21:03:07.059205 (+ 521us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:03:07.059691 (+ 486us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1215 21:03:07.060266 (+ 575us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:03:07.060381 (+ 115us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:03:07.062367 (+ 1986us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.113.2:58085: BlockingRecv error: recv got EOF from 127.28.113.2:58085 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Dec 15 21:03:07 dist-test-slave-bjsf krb5kdc[5487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.113.3@KRBTEST.COM for oryx/127.28.113.62@KRBTEST.COM, Server not found in Kerberos database
Dec 15 21:03:07 dist-test-slave-bjsf krb5kdc[5487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.113.3@KRBTEST.COM for oryx/127.28.113.62@KRBTEST.COM, Server not found in Kerberos database
I20251215 21:03:07.232983 6517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:03:07.228409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.113.62:36095 (local address 127.28.113.3:33595)
1215 21:03:07.228672 (+ 263us) negotiation.cc:107] Waiting for socket to connect
1215 21:03:07.228684 (+ 12us) client_negotiation.cc:174] Beginning negotiation
1215 21:03:07.228783 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1215 21:03:07.229032 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1215 21:03:07.229035 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1215 21:03:07.229230 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
1215 21:03:07.229416 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:03:07.229421 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:03:07.230118 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1215 21:03:07.230121 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1215 21:03:07.230510 (+ 389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:03:07.230517 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:03:07.230614 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:03:07.231141 (+ 527us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1215 21:03:07.231155 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
1215 21:03:07.232830 (+ 1675us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.113.62:36095: Server oryx/127.28.113.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":93,"threads_started":1}
W20251215 21:03:07.233049 6515 negotiation.cc:337] Failed RPC negotiation. Trace:
1215 21:03:07.228464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.113.3:33595 (local address 127.28.113.62:36095)
1215 21:03:07.228600 (+ 136us) server_negotiation.cc:207] Beginning negotiation
1215 21:03:07.228605 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1215 21:03:07.228744 (+ 139us) server_negotiation.cc:408] Connection header received
1215 21:03:07.228884 (+ 140us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1215 21:03:07.228887 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1215 21:03:07.228939 (+ 52us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1215 21:03:07.229003 (+ 64us) server_negotiation.cc:227] Negotiated authn=SASL
1215 21:03:07.229523 (+ 520us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:03:07.230013 (+ 490us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1215 21:03:07.230613 (+ 600us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:03:07.230776 (+ 163us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:03:07.232956 (+ 2180us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.113.3:33595: BlockingRecv error: recv got EOF from 127.28.113.3:33595 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Dec 15 21:03:07 dist-test-slave-bjsf krb5kdc[5487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.113.1@KRBTEST.COM for oryx/127.28.113.62@KRBTEST.COM, Server not found in Kerberos database
Dec 15 21:03:07 dist-test-slave-bjsf krb5kdc[5487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.113.1@KRBTEST.COM for oryx/127.28.113.62@KRBTEST.COM, Server not found in Kerberos database
I20251215 21:03:07.910140 6518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:03:07.905627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.113.62:36095 (local address 127.28.113.1:38573)
1215 21:03:07.905916 (+ 289us) negotiation.cc:107] Waiting for socket to connect
1215 21:03:07.905928 (+ 12us) client_negotiation.cc:174] Beginning negotiation
1215 21:03:07.906031 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1215 21:03:07.906293 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1215 21:03:07.906296 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1215 21:03:07.906483 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
1215 21:03:07.906679 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:03:07.906684 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:03:07.907357 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1215 21:03:07.907359 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1215 21:03:07.907727 (+ 368us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:03:07.907732 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:03:07.907813 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:03:07.908343 (+ 530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1215 21:03:07.908355 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
1215 21:03:07.910019 (+ 1664us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.113.62:36095: Server oryx/127.28.113.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":98,"threads_started":1}
W20251215 21:03:07.910261 6519 negotiation.cc:337] Failed RPC negotiation. Trace:
1215 21:03:07.905685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.113.1:38573 (local address 127.28.113.62:36095)
1215 21:03:07.905908 (+ 223us) server_negotiation.cc:207] Beginning negotiation
1215 21:03:07.905911 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1215 21:03:07.905994 (+ 83us) server_negotiation.cc:408] Connection header received
1215 21:03:07.906103 (+ 109us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1215 21:03:07.906106 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1215 21:03:07.906152 (+ 46us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1215 21:03:07.906222 (+ 70us) server_negotiation.cc:227] Negotiated authn=SASL
1215 21:03:07.906773 (+ 551us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:03:07.907281 (+ 508us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1215 21:03:07.907800 (+ 519us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:03:07.907919 (+ 119us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:03:07.910147 (+ 2228us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.113.1:38573: BlockingRecv error: recv got EOF from 127.28.113.1:38573 (error 108)
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":55,"threads_started":1}
Dec 15 21:03:08 dist-test-slave-bjsf krb5kdc[5487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.113.2@KRBTEST.COM for oryx/127.28.113.62@KRBTEST.COM, Server not found in Kerberos database
Dec 15 21:03:08 dist-test-slave-bjsf krb5kdc[5487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.113.2@KRBTEST.COM for oryx/127.28.113.62@KRBTEST.COM, Server not found in Kerberos database
I20251215 21:03:08.067463 6520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:03:08.063014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.113.62:36095 (local address 127.28.113.2:59823)
1215 21:03:08.063269 (+ 255us) negotiation.cc:107] Waiting for socket to connect
1215 21:03:08.063282 (+ 13us) client_negotiation.cc:174] Beginning negotiation
1215 21:03:08.063376 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1215 21:03:08.063592 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1215 21:03:08.063594 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1215 21:03:08.063760 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
1215 21:03:08.063991 (+ 231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:03:08.064000 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:03:08.064770 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1215 21:03:08.064773 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1215 21:03:08.065161 (+ 388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:03:08.065167 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:03:08.065257 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:03:08.065751 (+ 494us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1215 21:03:08.065764 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
1215 21:03:08.067350 (+ 1586us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.113.62:36095: Server oryx/127.28.113.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":83,"threads_started":1}
W20251215 21:03:08.067608 6519 negotiation.cc:337] Failed RPC negotiation. Trace:
1215 21:03:08.063104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.113.2:59823 (local address 127.28.113.62:36095)
1215 21:03:08.063227 (+ 123us) server_negotiation.cc:207] Beginning negotiation
1215 21:03:08.063230 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1215 21:03:08.063340 (+ 110us) server_negotiation.cc:408] Connection header received
1215 21:03:08.063447 (+ 107us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1215 21:03:08.063450 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1215 21:03:08.063495 (+ 45us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1215 21:03:08.063564 (+ 69us) server_negotiation.cc:227] Negotiated authn=SASL
1215 21:03:08.064129 (+ 565us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:03:08.064674 (+ 545us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1215 21:03:08.065279 (+ 605us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:03:08.065422 (+ 143us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:03:08.067487 (+ 2065us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.113.2:59823: BlockingRecv error: recv got EOF from 127.28.113.2:59823 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Dec 15 21:03:08 dist-test-slave-bjsf krb5kdc[5487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.113.3@KRBTEST.COM for oryx/127.28.113.62@KRBTEST.COM, Server not found in Kerberos database
Dec 15 21:03:08 dist-test-slave-bjsf krb5kdc[5487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.113.3@KRBTEST.COM for oryx/127.28.113.62@KRBTEST.COM, Server not found in Kerberos database
I20251215 21:03:08.238382 6521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:03:08.233823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.113.62:36095 (local address 127.28.113.3:43943)
1215 21:03:08.234111 (+ 288us) negotiation.cc:107] Waiting for socket to connect
1215 21:03:08.234126 (+ 15us) client_negotiation.cc:174] Beginning negotiation
1215 21:03:08.234242 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1215 21:03:08.234464 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1215 21:03:08.234467 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1215 21:03:08.234659 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
1215 21:03:08.234843 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:03:08.234849 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:03:08.235542 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1215 21:03:08.235546 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1215 21:03:08.235918 (+ 372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:03:08.235925 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:03:08.236049 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:03:08.236593 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1215 21:03:08.236605 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
1215 21:03:08.238232 (+ 1627us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.113.62:36095: Server oryx/127.28.113.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":93,"threads_started":1}
W20251215 21:03:08.238463 6519 negotiation.cc:337] Failed RPC negotiation. Trace:
1215 21:03:08.233933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.113.3:43943 (local address 127.28.113.62:36095)
1215 21:03:08.234081 (+ 148us) server_negotiation.cc:207] Beginning negotiation
1215 21:03:08.234086 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1215 21:03:08.234188 (+ 102us) server_negotiation.cc:408] Connection header received
1215 21:03:08.234323 (+ 135us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1215 21:03:08.234327 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1215 21:03:08.234373 (+ 46us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1215 21:03:08.234435 (+ 62us) server_negotiation.cc:227] Negotiated authn=SASL
1215 21:03:08.234964 (+ 529us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:03:08.235443 (+ 479us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1215 21:03:08.236041 (+ 598us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:03:08.236207 (+ 166us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:03:08.238367 (+ 2160us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.113.3:43943: BlockingRecv error: recv got EOF from 127.28.113.3:43943 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Dec 15 21:03:08 dist-test-slave-bjsf krb5kdc[5487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.113.1@KRBTEST.COM for oryx/127.28.113.62@KRBTEST.COM, Server not found in Kerberos database
Dec 15 21:03:08 dist-test-slave-bjsf krb5kdc[5487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.113.1@KRBTEST.COM for oryx/127.28.113.62@KRBTEST.COM, Server not found in Kerberos database
I20251215 21:03:08.915282 6522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:03:08.910850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.113.62:36095 (local address 127.28.113.1:53117)
1215 21:03:08.911138 (+ 288us) negotiation.cc:107] Waiting for socket to connect
1215 21:03:08.911150 (+ 12us) client_negotiation.cc:174] Beginning negotiation
1215 21:03:08.911254 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1215 21:03:08.911470 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1215 21:03:08.911472 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1215 21:03:08.911639 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
1215 21:03:08.911831 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:03:08.911836 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:03:08.912588 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1215 21:03:08.912590 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1215 21:03:08.912956 (+ 366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:03:08.912962 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:03:08.913045 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:03:08.913583 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1215 21:03:08.913597 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
1215 21:03:08.915157 (+ 1560us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.113.62:36095: Server oryx/127.28.113.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":76,"threads_started":1}
W20251215 21:03:08.915408 6523 negotiation.cc:337] Failed RPC negotiation. Trace:
1215 21:03:08.910919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.113.1:53117 (local address 127.28.113.62:36095)
1215 21:03:08.911139 (+ 220us) server_negotiation.cc:207] Beginning negotiation
1215 21:03:08.911143 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1215 21:03:08.911217 (+ 74us) server_negotiation.cc:408] Connection header received
1215 21:03:08.911325 (+ 108us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1215 21:03:08.911328 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1215 21:03:08.911374 (+ 46us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1215 21:03:08.911452 (+ 78us) server_negotiation.cc:227] Negotiated authn=SASL
1215 21:03:08.911977 (+ 525us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:03:08.912506 (+ 529us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1215 21:03:08.913033 (+ 527us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:03:08.913156 (+ 123us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:03:08.915294 (+ 2138us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.113.1:53117: BlockingRecv error: recv got EOF from 127.28.113.1:53117 (error 108)
Metrics: {"server-negotiator.queue_time_us":152,"thread_start_us":68,"threads_started":1}
Dec 15 21:03:09 dist-test-slave-bjsf krb5kdc[5487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.113.2@KRBTEST.COM for oryx/127.28.113.62@KRBTEST.COM, Server not found in Kerberos database
Dec 15 21:03:09 dist-test-slave-bjsf krb5kdc[5487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.113.2@KRBTEST.COM for oryx/127.28.113.62@KRBTEST.COM, Server not found in Kerberos database
I20251215 21:03:09.072638 6524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:03:09.068095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.113.62:36095 (local address 127.28.113.2:53495)
1215 21:03:09.068365 (+ 270us) negotiation.cc:107] Waiting for socket to connect
1215 21:03:09.068377 (+ 12us) client_negotiation.cc:174] Beginning negotiation
1215 21:03:09.068477 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1215 21:03:09.068724 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1215 21:03:09.068727 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1215 21:03:09.068936 (+ 209us) client_negotiation.cc:189] Negotiated authn=SASL
1215 21:03:09.069156 (+ 220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:03:09.069163 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:03:09.069864 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1215 21:03:09.069868 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1215 21:03:09.070266 (+ 398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:03:09.070273 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:03:09.070368 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:03:09.070894 (+ 526us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1215 21:03:09.070907 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
1215 21:03:09.072528 (+ 1621us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.113.62:36095: Server oryx/127.28.113.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":64,"threads_started":1}
W20251215 21:03:09.072744 6523 negotiation.cc:337] Failed RPC negotiation. Trace:
1215 21:03:09.068190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.113.2:53495 (local address 127.28.113.62:36095)
1215 21:03:09.068341 (+ 151us) server_negotiation.cc:207] Beginning negotiation
1215 21:03:09.068345 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1215 21:03:09.068437 (+ 92us) server_negotiation.cc:408] Connection header received
1215 21:03:09.068545 (+ 108us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1215 21:03:09.068547 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1215 21:03:09.068590 (+ 43us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1215 21:03:09.068652 (+ 62us) server_negotiation.cc:227] Negotiated authn=SASL
1215 21:03:09.069254 (+ 602us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:03:09.069736 (+ 482us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1215 21:03:09.070354 (+ 618us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:03:09.070469 (+ 115us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:03:09.072646 (+ 2177us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.113.2:53495: BlockingRecv error: recv got EOF from 127.28.113.2:53495 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Dec 15 21:03:09 dist-test-slave-bjsf krb5kdc[5487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.113.3@KRBTEST.COM for oryx/127.28.113.62@KRBTEST.COM, Server not found in Kerberos database
Dec 15 21:03:09 dist-test-slave-bjsf krb5kdc[5487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.113.3@KRBTEST.COM for oryx/127.28.113.62@KRBTEST.COM, Server not found in Kerberos database
I20251215 21:03:09.243613 6525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:03:09.239282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.113.62:36095 (local address 127.28.113.3:34115)
1215 21:03:09.239552 (+ 270us) negotiation.cc:107] Waiting for socket to connect
1215 21:03:09.239565 (+ 13us) client_negotiation.cc:174] Beginning negotiation
1215 21:03:09.239664 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1215 21:03:09.239858 (+ 194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1215 21:03:09.239860 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1215 21:03:09.240059 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
1215 21:03:09.240254 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:03:09.240260 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:03:09.240869 (+ 609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1215 21:03:09.240872 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1215 21:03:09.241246 (+ 374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:03:09.241252 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:03:09.241334 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:03:09.241819 (+ 485us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1215 21:03:09.241830 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
1215 21:03:09.243469 (+ 1639us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.113.62:36095: Server oryx/127.28.113.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":78,"threads_started":1}
W20251215 21:03:09.243723 6523 negotiation.cc:337] Failed RPC negotiation. Trace:
1215 21:03:09.239356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.113.3:34115 (local address 127.28.113.62:36095)
1215 21:03:09.239474 (+ 118us) server_negotiation.cc:207] Beginning negotiation
1215 21:03:09.239477 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1215 21:03:09.239622 (+ 145us) server_negotiation.cc:408] Connection header received
1215 21:03:09.239728 (+ 106us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1215 21:03:09.239730 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1215 21:03:09.239774 (+ 44us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1215 21:03:09.239834 (+ 60us) server_negotiation.cc:227] Negotiated authn=SASL
1215 21:03:09.240334 (+ 500us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:03:09.240798 (+ 464us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1215 21:03:09.241321 (+ 523us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:03:09.241438 (+ 117us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:03:09.243628 (+ 2190us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.113.3:34115: BlockingRecv error: recv got EOF from 127.28.113.3:34115 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
Dec 15 21:03:09 dist-test-slave-bjsf krb5kdc[5487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.113.1@KRBTEST.COM for oryx/127.28.113.62@KRBTEST.COM, Server not found in Kerberos database
Dec 15 21:03:09 dist-test-slave-bjsf krb5kdc[5487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.113.1@KRBTEST.COM for oryx/127.28.113.62@KRBTEST.COM, Server not found in Kerberos database
I20251215 21:03:09.920894 6526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:03:09.916138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.113.62:36095 (local address 127.28.113.1:54703)
1215 21:03:09.916427 (+ 289us) negotiation.cc:107] Waiting for socket to connect
1215 21:03:09.916442 (+ 15us) client_negotiation.cc:174] Beginning negotiation
1215 21:03:09.916547 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1215 21:03:09.916793 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1215 21:03:09.916795 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1215 21:03:09.916994 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
1215 21:03:09.917221 (+ 227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:03:09.917228 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:03:09.917916 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1215 21:03:09.917918 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1215 21:03:09.918316 (+ 398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:03:09.918323 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:03:09.918419 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:03:09.918978 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1215 21:03:09.918993 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
1215 21:03:09.920738 (+ 1745us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.113.62:36095: Server oryx/127.28.113.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":82,"threads_started":1}
W20251215 21:03:09.920996 6527 negotiation.cc:337] Failed RPC negotiation. Trace:
1215 21:03:09.916252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.113.1:54703 (local address 127.28.113.62:36095)
1215 21:03:09.916469 (+ 217us) server_negotiation.cc:207] Beginning negotiation
1215 21:03:09.916473 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1215 21:03:09.916488 (+ 15us) server_negotiation.cc:408] Connection header received
1215 21:03:09.916641 (+ 153us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1215 21:03:09.916644 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1215 21:03:09.916689 (+ 45us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1215 21:03:09.916756 (+ 67us) server_negotiation.cc:227] Negotiated authn=SASL
1215 21:03:09.917334 (+ 578us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:03:09.917833 (+ 499us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1215 21:03:09.918405 (+ 572us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:03:09.918529 (+ 124us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:03:09.920877 (+ 2348us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.113.1:54703: BlockingRecv error: recv got EOF from 127.28.113.1:54703 (error 108)
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":63,"threads_started":1}
Dec 15 21:03:10 dist-test-slave-bjsf krb5kdc[5487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.113.2@KRBTEST.COM for oryx/127.28.113.62@KRBTEST.COM, Server not found in Kerberos database
Dec 15 21:03:10 dist-test-slave-bjsf krb5kdc[5487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.113.2@KRBTEST.COM for oryx/127.28.113.62@KRBTEST.COM, Server not found in Kerberos database
I20251215 21:03:10.077926 6528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:03:10.073343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.113.62:36095 (local address 127.28.113.2:48711)
1215 21:03:10.073667 (+ 324us) negotiation.cc:107] Waiting for socket to connect
1215 21:03:10.073682 (+ 15us) client_negotiation.cc:174] Beginning negotiation
1215 21:03:10.073790 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1215 21:03:10.074028 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1215 21:03:10.074030 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1215 21:03:10.074205 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
1215 21:03:10.074396 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:03:10.074401 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:03:10.075255 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1215 21:03:10.075257 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1215 21:03:10.075618 (+ 361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:03:10.075623 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:03:10.075701 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:03:10.076247 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1215 21:03:10.076259 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
1215 21:03:10.077814 (+ 1555us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.113.62:36095: Server oryx/127.28.113.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":77,"threads_started":1}
W20251215 21:03:10.078042 6527 negotiation.cc:337] Failed RPC negotiation. Trace:
1215 21:03:10.073409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.113.2:48711 (local address 127.28.113.62:36095)
1215 21:03:10.073571 (+ 162us) server_negotiation.cc:207] Beginning negotiation
1215 21:03:10.073575 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1215 21:03:10.073753 (+ 178us) server_negotiation.cc:408] Connection header received
1215 21:03:10.073883 (+ 130us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1215 21:03:10.073886 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1215 21:03:10.073940 (+ 54us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1215 21:03:10.074006 (+ 66us) server_negotiation.cc:227] Negotiated authn=SASL
1215 21:03:10.074489 (+ 483us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:03:10.075178 (+ 689us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1215 21:03:10.075693 (+ 515us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:03:10.075820 (+ 127us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:03:10.077942 (+ 2122us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.113.2:48711: BlockingRecv error: recv got EOF from 127.28.113.2:48711 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
Dec 15 21:03:10 dist-test-slave-bjsf krb5kdc[5487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.113.3@KRBTEST.COM for oryx/127.28.113.62@KRBTEST.COM, Server not found in Kerberos database
Dec 15 21:03:10 dist-test-slave-bjsf krb5kdc[5487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.113.3@KRBTEST.COM for oryx/127.28.113.62@KRBTEST.COM, Server not found in Kerberos database
I20251215 21:03:10.249365 6529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:03:10.244617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.113.62:36095 (local address 127.28.113.3:52705)
1215 21:03:10.244885 (+ 268us) negotiation.cc:107] Waiting for socket to connect
1215 21:03:10.244899 (+ 14us) client_negotiation.cc:174] Beginning negotiation
1215 21:03:10.245000 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1215 21:03:10.245270 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1215 21:03:10.245273 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1215 21:03:10.245454 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
1215 21:03:10.245705 (+ 251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:03:10.245716 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:03:10.246511 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1215 21:03:10.246515 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1215 21:03:10.246895 (+ 380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:03:10.246902 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:03:10.246991 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:03:10.247504 (+ 513us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1215 21:03:10.247516 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
1215 21:03:10.249217 (+ 1701us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.113.62:36095: Server oryx/127.28.113.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":80,"threads_started":1}
W20251215 21:03:10.249454 6527 negotiation.cc:337] Failed RPC negotiation. Trace:
1215 21:03:10.244728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.113.3:52705 (local address 127.28.113.62:36095)
1215 21:03:10.244915 (+ 187us) server_negotiation.cc:207] Beginning negotiation
1215 21:03:10.244918 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1215 21:03:10.244952 (+ 34us) server_negotiation.cc:408] Connection header received
1215 21:03:10.245087 (+ 135us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1215 21:03:10.245090 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1215 21:03:10.245155 (+ 65us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1215 21:03:10.245232 (+ 77us) server_negotiation.cc:227] Negotiated authn=SASL
1215 21:03:10.245819 (+ 587us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:03:10.246411 (+ 592us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1215 21:03:10.246988 (+ 577us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:03:10.247121 (+ 133us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:03:10.249356 (+ 2235us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.113.3:52705: BlockingRecv error: recv got EOF from 127.28.113.3:52705 (error 108)
Metrics: {"server-negotiator.queue_time_us":95}
Dec 15 21:03:10 dist-test-slave-bjsf krb5kdc[5487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.113.1@KRBTEST.COM for oryx/127.28.113.62@KRBTEST.COM, Server not found in Kerberos database
Dec 15 21:03:10 dist-test-slave-bjsf krb5kdc[5487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.113.1@KRBTEST.COM for oryx/127.28.113.62@KRBTEST.COM, Server not found in Kerberos database
I20251215 21:03:10.929994 6530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:03:10.921710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.113.62:36095 (local address 127.28.113.1:56537)
1215 21:03:10.922020 (+ 310us) negotiation.cc:107] Waiting for socket to connect
1215 21:03:10.922034 (+ 14us) client_negotiation.cc:174] Beginning negotiation
1215 21:03:10.922141 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1215 21:03:10.922377 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1215 21:03:10.922379 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1215 21:03:10.922548 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
1215 21:03:10.922757 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:03:10.922763 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:03:10.923453 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1215 21:03:10.923455 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1215 21:03:10.923819 (+ 364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:03:10.923825 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:03:10.923912 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:03:10.928180 (+ 4268us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1215 21:03:10.928196 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
1215 21:03:10.929869 (+ 1673us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.113.62:36095: Server oryx/127.28.113.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":92,"threads_started":1}
W20251215 21:03:10.930297 6531 negotiation.cc:337] Failed RPC negotiation. Trace:
1215 21:03:10.921834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.113.1:56537 (local address 127.28.113.62:36095)
1215 21:03:10.922082 (+ 248us) server_negotiation.cc:207] Beginning negotiation
1215 21:03:10.922086 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1215 21:03:10.922101 (+ 15us) server_negotiation.cc:408] Connection header received
1215 21:03:10.922224 (+ 123us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1215 21:03:10.922226 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1215 21:03:10.922273 (+ 47us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1215 21:03:10.922349 (+ 76us) server_negotiation.cc:227] Negotiated authn=SASL
1215 21:03:10.922848 (+ 499us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:03:10.923373 (+ 525us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1215 21:03:10.923933 (+ 560us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:03:10.924135 (+ 202us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:03:10.930077 (+ 5942us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.113.1:56537: BlockingRecv error: recv got EOF from 127.28.113.1:56537 (error 108)
Metrics: {"server-negotiator.queue_time_us":179,"thread_start_us":63,"threads_started":1}
Dec 15 21:03:11 dist-test-slave-bjsf krb5kdc[5487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.113.2@KRBTEST.COM for oryx/127.28.113.62@KRBTEST.COM, Server not found in Kerberos database
Dec 15 21:03:11 dist-test-slave-bjsf krb5kdc[5487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.113.2@KRBTEST.COM for oryx/127.28.113.62@KRBTEST.COM, Server not found in Kerberos database
I20251215 21:03:11.083186 6532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:03:11.078675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.113.62:36095 (local address 127.28.113.2:49157)
1215 21:03:11.078965 (+ 290us) negotiation.cc:107] Waiting for socket to connect
1215 21:03:11.078978 (+ 13us) client_negotiation.cc:174] Beginning negotiation
1215 21:03:11.079077 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1215 21:03:11.079296 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1215 21:03:11.079298 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1215 21:03:11.079475 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
1215 21:03:11.079671 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:03:11.079677 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:03:11.080420 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1215 21:03:11.080424 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1215 21:03:11.080806 (+ 382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:03:11.080813 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:03:11.080916 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:03:11.081447 (+ 531us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1215 21:03:11.081461 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
1215 21:03:11.083065 (+ 1604us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.113.62:36095: Server oryx/127.28.113.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":84,"threads_started":1}
W20251215 21:03:11.083307 6531 negotiation.cc:337] Failed RPC negotiation. Trace:
1215 21:03:11.078762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.113.2:49157 (local address 127.28.113.62:36095)
1215 21:03:11.078885 (+ 123us) server_negotiation.cc:207] Beginning negotiation
1215 21:03:11.078888 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1215 21:03:11.079034 (+ 146us) server_negotiation.cc:408] Connection header received
1215 21:03:11.079157 (+ 123us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1215 21:03:11.079159 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1215 21:03:11.079211 (+ 52us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1215 21:03:11.079272 (+ 61us) server_negotiation.cc:227] Negotiated authn=SASL
1215 21:03:11.079757 (+ 485us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:03:11.080296 (+ 539us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1215 21:03:11.080895 (+ 599us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:03:11.081014 (+ 119us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:03:11.083205 (+ 2191us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.113.2:49157: BlockingRecv error: recv got EOF from 127.28.113.2:49157 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Dec 15 21:03:11 dist-test-slave-bjsf krb5kdc[5487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.113.3@KRBTEST.COM for oryx/127.28.113.62@KRBTEST.COM, Server not found in Kerberos database
Dec 15 21:03:11 dist-test-slave-bjsf krb5kdc[5487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.113.3@KRBTEST.COM for oryx/127.28.113.62@KRBTEST.COM, Server not found in Kerberos database
I20251215 21:03:11.254560 6533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:03:11.250236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.113.62:36095 (local address 127.28.113.3:37751)
1215 21:03:11.250487 (+ 251us) negotiation.cc:107] Waiting for socket to connect
1215 21:03:11.250502 (+ 15us) client_negotiation.cc:174] Beginning negotiation
1215 21:03:11.250603 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1215 21:03:11.250801 (+ 198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1215 21:03:11.250803 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1215 21:03:11.251000 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
1215 21:03:11.251194 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:03:11.251199 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:03:11.251821 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1215 21:03:11.251824 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1215 21:03:11.252245 (+ 421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:03:11.252253 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:03:11.252337 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:03:11.252826 (+ 489us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1215 21:03:11.252838 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
1215 21:03:11.254442 (+ 1604us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.113.62:36095: Server oryx/127.28.113.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":70,"threads_started":1}
W20251215 21:03:11.254676 6531 negotiation.cc:337] Failed RPC negotiation. Trace:
1215 21:03:11.250314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.113.3:37751 (local address 127.28.113.62:36095)
1215 21:03:11.250434 (+ 120us) server_negotiation.cc:207] Beginning negotiation
1215 21:03:11.250437 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1215 21:03:11.250562 (+ 125us) server_negotiation.cc:408] Connection header received
1215 21:03:11.250670 (+ 108us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1215 21:03:11.250672 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1215 21:03:11.250716 (+ 44us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1215 21:03:11.250778 (+ 62us) server_negotiation.cc:227] Negotiated authn=SASL
1215 21:03:11.251278 (+ 500us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:03:11.251746 (+ 468us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1215 21:03:11.252330 (+ 584us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:03:11.252451 (+ 121us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:03:11.254575 (+ 2124us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.113.3:37751: BlockingRecv error: recv got EOF from 127.28.113.3:37751 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Dec 15 21:03:11 dist-test-slave-bjsf krb5kdc[5487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.113.1@KRBTEST.COM for oryx/127.28.113.62@KRBTEST.COM, Server not found in Kerberos database
Dec 15 21:03:11 dist-test-slave-bjsf krb5kdc[5487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.113.1@KRBTEST.COM for oryx/127.28.113.62@KRBTEST.COM, Server not found in Kerberos database
I20251215 21:03:11.935292 6534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:03:11.930836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.113.62:36095 (local address 127.28.113.1:58335)
1215 21:03:11.931177 (+ 341us) negotiation.cc:107] Waiting for socket to connect
1215 21:03:11.931189 (+ 12us) client_negotiation.cc:174] Beginning negotiation
1215 21:03:11.931290 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1215 21:03:11.931501 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1215 21:03:11.931503 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1215 21:03:11.931675 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
1215 21:03:11.931868 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:03:11.931874 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:03:11.932603 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1215 21:03:11.932606 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1215 21:03:11.932985 (+ 379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:03:11.932992 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:03:11.933080 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:03:11.933577 (+ 497us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1215 21:03:11.933589 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
1215 21:03:11.935168 (+ 1579us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.113.62:36095: Server oryx/127.28.113.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":97,"threads_started":1}
W20251215 21:03:11.935449 6535 negotiation.cc:337] Failed RPC negotiation. Trace:
1215 21:03:11.930874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.113.1:58335 (local address 127.28.113.62:36095)
1215 21:03:11.931108 (+ 234us) server_negotiation.cc:207] Beginning negotiation
1215 21:03:11.931111 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1215 21:03:11.931257 (+ 146us) server_negotiation.cc:408] Connection header received
1215 21:03:11.931357 (+ 100us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1215 21:03:11.931359 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1215 21:03:11.931404 (+ 45us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1215 21:03:11.931478 (+ 74us) server_negotiation.cc:227] Negotiated authn=SASL
1215 21:03:11.931971 (+ 493us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:03:11.932508 (+ 537us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1215 21:03:11.933066 (+ 558us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:03:11.933199 (+ 133us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:03:11.935317 (+ 2118us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.113.1:58335: BlockingRecv error: recv got EOF from 127.28.113.1:58335 (error 108)
Metrics: {"server-negotiator.queue_time_us":157,"thread_start_us":69,"threads_started":1}
Dec 15 21:03:12 dist-test-slave-bjsf krb5kdc[5487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.113.2@KRBTEST.COM for oryx/127.28.113.62@KRBTEST.COM, Server not found in Kerberos database
Dec 15 21:03:12 dist-test-slave-bjsf krb5kdc[5487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.113.2@KRBTEST.COM for oryx/127.28.113.62@KRBTEST.COM, Server not found in Kerberos database
I20251215 21:03:12.088291 6536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:03:12.083944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.113.62:36095 (local address 127.28.113.2:43973)
1215 21:03:12.084234 (+ 290us) negotiation.cc:107] Waiting for socket to connect
1215 21:03:12.084248 (+ 14us) client_negotiation.cc:174] Beginning negotiation
1215 21:03:12.084338 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1215 21:03:12.084541 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1215 21:03:12.084543 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1215 21:03:12.084712 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
1215 21:03:12.084903 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:03:12.084908 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:03:12.085586 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1215 21:03:12.085589 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1215 21:03:12.085961 (+ 372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:03:12.085967 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:03:12.086046 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:03:12.086540 (+ 494us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1215 21:03:12.086551 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
1215 21:03:12.088174 (+ 1623us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.113.62:36095: Server oryx/127.28.113.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":90,"threads_started":1}
W20251215 21:03:12.088444 6535 negotiation.cc:337] Failed RPC negotiation. Trace:
1215 21:03:12.084041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.113.2:43973 (local address 127.28.113.62:36095)
1215 21:03:12.084162 (+ 121us) server_negotiation.cc:207] Beginning negotiation
1215 21:03:12.084166 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1215 21:03:12.084305 (+ 139us) server_negotiation.cc:408] Connection header received
1215 21:03:12.084408 (+ 103us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1215 21:03:12.084410 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1215 21:03:12.084453 (+ 43us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1215 21:03:12.084518 (+ 65us) server_negotiation.cc:227] Negotiated authn=SASL
1215 21:03:12.085014 (+ 496us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:03:12.085510 (+ 496us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1215 21:03:12.086036 (+ 526us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:03:12.086158 (+ 122us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:03:12.088327 (+ 2169us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.113.2:43973: BlockingRecv error: recv got EOF from 127.28.113.2:43973 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Dec 15 21:03:12 dist-test-slave-bjsf krb5kdc[5487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.113.3@KRBTEST.COM for oryx/127.28.113.62@KRBTEST.COM, Server not found in Kerberos database
Dec 15 21:03:12 dist-test-slave-bjsf krb5kdc[5487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.113.3@KRBTEST.COM for oryx/127.28.113.62@KRBTEST.COM, Server not found in Kerberos database
I20251215 21:03:12.259634 6537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:03:12.255340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.113.62:36095 (local address 127.28.113.3:39317)
1215 21:03:12.255595 (+ 255us) negotiation.cc:107] Waiting for socket to connect
1215 21:03:12.255609 (+ 14us) client_negotiation.cc:174] Beginning negotiation
1215 21:03:12.255697 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1215 21:03:12.255907 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1215 21:03:12.255909 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1215 21:03:12.256098 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
1215 21:03:12.256304 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:03:12.256310 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:03:12.256943 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1215 21:03:12.256945 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1215 21:03:12.257321 (+ 376us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:03:12.257326 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:03:12.257404 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:03:12.257922 (+ 518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1215 21:03:12.257939 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
1215 21:03:12.259519 (+ 1580us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.113.62:36095: Server oryx/127.28.113.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":81,"threads_started":1}
W20251215 21:03:12.259773 6535 negotiation.cc:337] Failed RPC negotiation. Trace:
1215 21:03:12.255475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.113.3:39317 (local address 127.28.113.62:36095)
1215 21:03:12.255598 (+ 123us) server_negotiation.cc:207] Beginning negotiation
1215 21:03:12.255601 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1215 21:03:12.255666 (+ 65us) server_negotiation.cc:408] Connection header received
1215 21:03:12.255765 (+ 99us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1215 21:03:12.255768 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1215 21:03:12.255813 (+ 45us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1215 21:03:12.255881 (+ 68us) server_negotiation.cc:227] Negotiated authn=SASL
1215 21:03:12.256390 (+ 509us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:03:12.256866 (+ 476us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1215 21:03:12.257394 (+ 528us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:03:12.257511 (+ 117us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:03:12.259653 (+ 2142us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.113.3:39317: BlockingRecv error: recv got EOF from 127.28.113.3:39317 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Dec 15 21:03:12 dist-test-slave-bjsf krb5kdc[5487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.113.1@KRBTEST.COM for oryx/127.28.113.62@KRBTEST.COM, Server not found in Kerberos database
Dec 15 21:03:12 dist-test-slave-bjsf krb5kdc[5487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.113.1@KRBTEST.COM for oryx/127.28.113.62@KRBTEST.COM, Server not found in Kerberos database
I20251215 21:03:12.940332 6538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:03:12.936044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.113.62:36095 (local address 127.28.113.1:38815)
1215 21:03:12.936313 (+ 269us) negotiation.cc:107] Waiting for socket to connect
1215 21:03:12.936326 (+ 13us) client_negotiation.cc:174] Beginning negotiation
1215 21:03:12.936416 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1215 21:03:12.936640 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1215 21:03:12.936642 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1215 21:03:12.936812 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
1215 21:03:12.937009 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:03:12.937014 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:03:12.937675 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1215 21:03:12.937677 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1215 21:03:12.938037 (+ 360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:03:12.938043 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:03:12.938120 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:03:12.938629 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1215 21:03:12.938642 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
1215 21:03:12.940182 (+ 1540us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.113.62:36095: Server oryx/127.28.113.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":79,"threads_started":1}
W20251215 21:03:12.940441 6539 negotiation.cc:337] Failed RPC negotiation. Trace:
1215 21:03:12.936095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.113.1:38815 (local address 127.28.113.62:36095)
1215 21:03:12.936347 (+ 252us) server_negotiation.cc:207] Beginning negotiation
1215 21:03:12.936350 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1215 21:03:12.936367 (+ 17us) server_negotiation.cc:408] Connection header received
1215 21:03:12.936492 (+ 125us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1215 21:03:12.936494 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1215 21:03:12.936541 (+ 47us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1215 21:03:12.936616 (+ 75us) server_negotiation.cc:227] Negotiated authn=SASL
1215 21:03:12.937095 (+ 479us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:03:12.937595 (+ 500us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1215 21:03:12.938109 (+ 514us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:03:12.938232 (+ 123us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:03:12.940322 (+ 2090us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.113.1:38815: BlockingRecv error: recv got EOF from 127.28.113.1:38815 (error 108)
Metrics: {"server-negotiator.queue_time_us":178,"thread_start_us":77,"threads_started":1}
Dec 15 21:03:13 dist-test-slave-bjsf krb5kdc[5487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.113.2@KRBTEST.COM for oryx/127.28.113.62@KRBTEST.COM, Server not found in Kerberos database
Dec 15 21:03:13 dist-test-slave-bjsf krb5kdc[5487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.113.2@KRBTEST.COM for oryx/127.28.113.62@KRBTEST.COM, Server not found in Kerberos database
I20251215 21:03:13.093844 6540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:03:13.089149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.113.62:36095 (local address 127.28.113.2:45633)
1215 21:03:13.089412 (+ 263us) negotiation.cc:107] Waiting for socket to connect
1215 21:03:13.089425 (+ 13us) client_negotiation.cc:174] Beginning negotiation
1215 21:03:13.089533 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1215 21:03:13.089801 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1215 21:03:13.089805 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
1215 21:03:13.090079 (+ 274us) client_negotiation.cc:189] Negotiated authn=SASL
1215 21:03:13.090292 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:03:13.090300 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:03:13.090976 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1215 21:03:13.090979 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1215 21:03:13.091362 (+ 383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:03:13.091369 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:03:13.091461 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:03:13.091997 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1215 21:03:13.092016 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
1215 21:03:13.093726 (+ 1710us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.113.62:36095: Server oryx/127.28.113.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":75,"threads_started":1}
W20251215 21:03:13.093963 6539 negotiation.cc:337] Failed RPC negotiation. Trace:
1215 21:03:13.089220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.113.2:45633 (local address 127.28.113.62:36095)
1215 21:03:13.089339 (+ 119us) server_negotiation.cc:207] Beginning negotiation
1215 21:03:13.089343 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1215 21:03:13.089491 (+ 148us) server_negotiation.cc:408] Connection header received
1215 21:03:13.089615 (+ 124us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1215 21:03:13.089619 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1215 21:03:13.089670 (+ 51us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1215 21:03:13.089741 (+ 71us) server_negotiation.cc:227] Negotiated authn=SASL
1215 21:03:13.090389 (+ 648us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:03:13.090886 (+ 497us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1215 21:03:13.091448 (+ 562us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:03:13.091567 (+ 119us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:03:13.093857 (+ 2290us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.113.2:45633: BlockingRecv error: recv got EOF from 127.28.113.2:45633 (error 108)
Metrics: {"server-negotiator.queue_time_us":31}
Dec 15 21:03:13 dist-test-slave-bjsf krb5kdc[5487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.113.3@KRBTEST.COM for oryx/127.28.113.62@KRBTEST.COM, Server not found in Kerberos database
Dec 15 21:03:13 dist-test-slave-bjsf krb5kdc[5487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.113.3@KRBTEST.COM for oryx/127.28.113.62@KRBTEST.COM, Server not found in Kerberos database
I20251215 21:03:13.268993 6541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:03:13.260323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.113.62:36095 (local address 127.28.113.3:43541)
1215 21:03:13.260626 (+ 303us) negotiation.cc:107] Waiting for socket to connect
1215 21:03:13.260641 (+ 15us) client_negotiation.cc:174] Beginning negotiation
1215 21:03:13.260758 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1215 21:03:13.260986 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1215 21:03:13.260988 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1215 21:03:13.261181 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
1215 21:03:13.261382 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:03:13.261389 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:03:13.262272 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1215 21:03:13.262277 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1215 21:03:13.262762 (+ 485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:03:13.262772 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:03:13.262878 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:03:13.267133 (+ 4255us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1215 21:03:13.267148 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
1215 21:03:13.268836 (+ 1688us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.113.62:36095: Server oryx/127.28.113.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":84,"threads_started":1}
W20251215 21:03:13.269078 6539 negotiation.cc:337] Failed RPC negotiation. Trace:
1215 21:03:13.260407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.113.3:43541 (local address 127.28.113.62:36095)
1215 21:03:13.260550 (+ 143us) server_negotiation.cc:207] Beginning negotiation
1215 21:03:13.260554 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1215 21:03:13.260709 (+ 155us) server_negotiation.cc:408] Connection header received
1215 21:03:13.260835 (+ 126us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1215 21:03:13.260837 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1215 21:03:13.260888 (+ 51us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1215 21:03:13.260955 (+ 67us) server_negotiation.cc:227] Negotiated authn=SASL
1215 21:03:13.261519 (+ 564us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:03:13.262171 (+ 652us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1215 21:03:13.262865 (+ 694us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:03:13.263035 (+ 170us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:03:13.268980 (+ 5945us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.113.3:43541: BlockingRecv error: recv got EOF from 127.28.113.3:43541 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Dec 15 21:03:13 dist-test-slave-bjsf krb5kdc[5487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.113.1@KRBTEST.COM for oryx/127.28.113.62@KRBTEST.COM, Server not found in Kerberos database
Dec 15 21:03:13 dist-test-slave-bjsf krb5kdc[5487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.113.1@KRBTEST.COM for oryx/127.28.113.62@KRBTEST.COM, Server not found in Kerberos database
I20251215 21:03:13.945528 6542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:03:13.941029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.113.62:36095 (local address 127.28.113.1:33113)
1215 21:03:13.941348 (+ 319us) negotiation.cc:107] Waiting for socket to connect
1215 21:03:13.941361 (+ 13us) client_negotiation.cc:174] Beginning negotiation
1215 21:03:13.941464 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1215 21:03:13.941674 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1215 21:03:13.941676 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1215 21:03:13.941844 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
1215 21:03:13.942043 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:03:13.942048 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:03:13.942790 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1215 21:03:13.942792 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1215 21:03:13.943170 (+ 378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:03:13.943176 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:03:13.943256 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:03:13.943764 (+ 508us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1215 21:03:13.943776 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
1215 21:03:13.945410 (+ 1634us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.113.62:36095: Server oryx/127.28.113.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":95,"threads_started":1}
W20251215 21:03:13.945703 6543 negotiation.cc:337] Failed RPC negotiation. Trace:
1215 21:03:13.941137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.113.1:33113 (local address 127.28.113.62:36095)
1215 21:03:13.941369 (+ 232us) server_negotiation.cc:207] Beginning negotiation
1215 21:03:13.941372 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1215 21:03:13.941426 (+ 54us) server_negotiation.cc:408] Connection header received
1215 21:03:13.941533 (+ 107us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1215 21:03:13.941535 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1215 21:03:13.941580 (+ 45us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1215 21:03:13.941650 (+ 70us) server_negotiation.cc:227] Negotiated authn=SASL
1215 21:03:13.942170 (+ 520us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:03:13.942689 (+ 519us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1215 21:03:13.943244 (+ 555us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:03:13.943371 (+ 127us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:03:13.945566 (+ 2195us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.113.1:33113: BlockingRecv error: recv got EOF from 127.28.113.1:33113 (error 108)
Metrics: {"server-negotiator.queue_time_us":168,"thread_start_us":55,"threads_started":1}
Dec 15 21:03:14 dist-test-slave-bjsf krb5kdc[5487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.113.2@KRBTEST.COM for oryx/127.28.113.62@KRBTEST.COM, Server not found in Kerberos database
Dec 15 21:03:14 dist-test-slave-bjsf krb5kdc[5487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.113.2@KRBTEST.COM for oryx/127.28.113.62@KRBTEST.COM, Server not found in Kerberos database
I20251215 21:03:14.099056 6544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:03:14.094561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.113.62:36095 (local address 127.28.113.2:36745)
1215 21:03:14.094832 (+ 271us) negotiation.cc:107] Waiting for socket to connect
1215 21:03:14.094845 (+ 13us) client_negotiation.cc:174] Beginning negotiation
1215 21:03:14.094949 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1215 21:03:14.095166 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1215 21:03:14.095168 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1215 21:03:14.095339 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
1215 21:03:14.095534 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:03:14.095539 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:03:14.096262 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1215 21:03:14.096265 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1215 21:03:14.096646 (+ 381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:03:14.096652 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:03:14.096743 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:03:14.097252 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1215 21:03:14.097264 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
1215 21:03:14.098939 (+ 1675us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.113.62:36095: Server oryx/127.28.113.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":79,"threads_started":1}
W20251215 21:03:14.099218 6543 negotiation.cc:337] Failed RPC negotiation. Trace:
1215 21:03:14.094645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.113.2:36745 (local address 127.28.113.62:36095)
1215 21:03:14.094828 (+ 183us) server_negotiation.cc:207] Beginning negotiation
1215 21:03:14.094833 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1215 21:03:14.094913 (+ 80us) server_negotiation.cc:408] Connection header received
1215 21:03:14.095024 (+ 111us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1215 21:03:14.095026 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1215 21:03:14.095074 (+ 48us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1215 21:03:14.095150 (+ 76us) server_negotiation.cc:227] Negotiated authn=SASL
1215 21:03:14.095625 (+ 475us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:03:14.096177 (+ 552us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1215 21:03:14.096728 (+ 551us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:03:14.096855 (+ 127us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:03:14.099081 (+ 2226us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.113.2:36745: BlockingRecv error: recv got EOF from 127.28.113.2:36745 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Dec 15 21:03:14 dist-test-slave-bjsf krb5kdc[5487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.113.3@KRBTEST.COM for oryx/127.28.113.62@KRBTEST.COM, Server not found in Kerberos database
Dec 15 21:03:14 dist-test-slave-bjsf krb5kdc[5487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.113.3@KRBTEST.COM for oryx/127.28.113.62@KRBTEST.COM, Server not found in Kerberos database
I20251215 21:03:14.274461 6545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:03:14.269771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.113.62:36095 (local address 127.28.113.3:38977)
1215 21:03:14.270026 (+ 255us) negotiation.cc:107] Waiting for socket to connect
1215 21:03:14.270040 (+ 14us) client_negotiation.cc:174] Beginning negotiation
1215 21:03:14.270146 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1215 21:03:14.270374 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1215 21:03:14.270376 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1215 21:03:14.270548 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
1215 21:03:14.270742 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:03:14.270747 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:03:14.271439 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1215 21:03:14.271442 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1215 21:03:14.271811 (+ 369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:03:14.271817 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:03:14.271903 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:03:14.272454 (+ 551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1215 21:03:14.272467 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
1215 21:03:14.274317 (+ 1850us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.113.62:36095: Server oryx/127.28.113.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":70,"threads_started":1}
W20251215 21:03:14.274570 6543 negotiation.cc:337] Failed RPC negotiation. Trace:
1215 21:03:14.269846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.113.3:38977 (local address 127.28.113.62:36095)
1215 21:03:14.270039 (+ 193us) server_negotiation.cc:207] Beginning negotiation
1215 21:03:14.270043 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1215 21:03:14.270104 (+ 61us) server_negotiation.cc:408] Connection header received
1215 21:03:14.270233 (+ 129us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1215 21:03:14.270235 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1215 21:03:14.270285 (+ 50us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1215 21:03:14.270350 (+ 65us) server_negotiation.cc:227] Negotiated authn=SASL
1215 21:03:14.270847 (+ 497us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:03:14.271355 (+ 508us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1215 21:03:14.271905 (+ 550us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:03:14.272067 (+ 162us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:03:14.274457 (+ 2390us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.113.3:38977: BlockingRecv error: recv got EOF from 127.28.113.3:38977 (error 108)
Metrics: {"server-negotiator.queue_time_us":97}
Dec 15 21:03:14 dist-test-slave-bjsf krb5kdc[5487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.113.1@KRBTEST.COM for oryx/127.28.113.62@KRBTEST.COM, Server not found in Kerberos database
Dec 15 21:03:14 dist-test-slave-bjsf krb5kdc[5487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.113.1@KRBTEST.COM for oryx/127.28.113.62@KRBTEST.COM, Server not found in Kerberos database
I20251215 21:03:14.950721 6546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:03:14.946289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.113.62:36095 (local address 127.28.113.1:46921)
1215 21:03:14.946570 (+ 281us) negotiation.cc:107] Waiting for socket to connect
1215 21:03:14.946583 (+ 13us) client_negotiation.cc:174] Beginning negotiation
1215 21:03:14.946681 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1215 21:03:14.946912 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1215 21:03:14.946914 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1215 21:03:14.947102 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
1215 21:03:14.947323 (+ 221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:03:14.947331 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:03:14.948026 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1215 21:03:14.948028 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1215 21:03:14.948404 (+ 376us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:03:14.948410 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:03:14.948494 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:03:14.948995 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1215 21:03:14.949006 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
1215 21:03:14.950607 (+ 1601us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.113.62:36095: Server oryx/127.28.113.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":84,"threads_started":1}
W20251215 21:03:14.950861 6547 negotiation.cc:337] Failed RPC negotiation. Trace:
1215 21:03:14.946355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.113.1:46921 (local address 127.28.113.62:36095)
1215 21:03:14.946571 (+ 216us) server_negotiation.cc:207] Beginning negotiation
1215 21:03:14.946574 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1215 21:03:14.946642 (+ 68us) server_negotiation.cc:408] Connection header received
1215 21:03:14.946748 (+ 106us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1215 21:03:14.946750 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1215 21:03:14.946794 (+ 44us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1215 21:03:14.946865 (+ 71us) server_negotiation.cc:227] Negotiated authn=SASL
1215 21:03:14.947413 (+ 548us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:03:14.947915 (+ 502us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1215 21:03:14.948490 (+ 575us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:03:14.948618 (+ 128us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:03:14.950743 (+ 2125us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.113.1:46921: BlockingRecv error: recv got EOF from 127.28.113.1:46921 (error 108)
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":67,"threads_started":1}
Dec 15 21:03:15 dist-test-slave-bjsf krb5kdc[5487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.113.2@KRBTEST.COM for oryx/127.28.113.62@KRBTEST.COM, Server not found in Kerberos database
Dec 15 21:03:15 dist-test-slave-bjsf krb5kdc[5487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.113.2@KRBTEST.COM for oryx/127.28.113.62@KRBTEST.COM, Server not found in Kerberos database
I20251215 21:03:15.104192 6548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:03:15.099795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.113.62:36095 (local address 127.28.113.2:60737)
1215 21:03:15.100093 (+ 298us) negotiation.cc:107] Waiting for socket to connect
1215 21:03:15.100107 (+ 14us) client_negotiation.cc:174] Beginning negotiation
1215 21:03:15.100218 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1215 21:03:15.100428 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1215 21:03:15.100430 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1215 21:03:15.100602 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
1215 21:03:15.100791 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:03:15.100797 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:03:15.101459 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1215 21:03:15.101462 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1215 21:03:15.101825 (+ 363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:03:15.101831 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:03:15.101916 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:03:15.102400 (+ 484us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1215 21:03:15.102411 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
1215 21:03:15.104075 (+ 1664us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.113.62:36095: Server oryx/127.28.113.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":84,"threads_started":1}
W20251215 21:03:15.104329 6547 negotiation.cc:337] Failed RPC negotiation. Trace:
1215 21:03:15.099920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.113.2:60737 (local address 127.28.113.62:36095)
1215 21:03:15.100047 (+ 127us) server_negotiation.cc:207] Beginning negotiation
1215 21:03:15.100051 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1215 21:03:15.100177 (+ 126us) server_negotiation.cc:408] Connection header received
1215 21:03:15.100290 (+ 113us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1215 21:03:15.100292 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1215 21:03:15.100337 (+ 45us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1215 21:03:15.100404 (+ 67us) server_negotiation.cc:227] Negotiated authn=SASL
1215 21:03:15.100884 (+ 480us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:03:15.101375 (+ 491us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1215 21:03:15.101914 (+ 539us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:03:15.102046 (+ 132us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:03:15.104212 (+ 2166us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.113.2:60737: BlockingRecv error: recv got EOF from 127.28.113.2:60737 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Dec 15 21:03:15 dist-test-slave-bjsf krb5kdc[5487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.113.3@KRBTEST.COM for oryx/127.28.113.62@KRBTEST.COM, Server not found in Kerberos database
Dec 15 21:03:15 dist-test-slave-bjsf krb5kdc[5487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.113.3@KRBTEST.COM for oryx/127.28.113.62@KRBTEST.COM, Server not found in Kerberos database
I20251215 21:03:15.280526 6549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:03:15.275357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.113.62:36095 (local address 127.28.113.3:55309)
1215 21:03:15.275639 (+ 282us) negotiation.cc:107] Waiting for socket to connect
1215 21:03:15.275654 (+ 15us) client_negotiation.cc:174] Beginning negotiation
1215 21:03:15.275765 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1215 21:03:15.276016 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1215 21:03:15.276019 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1215 21:03:15.276226 (+ 207us) client_negotiation.cc:189] Negotiated authn=SASL
1215 21:03:15.276414 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:03:15.276421 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:03:15.277240 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1215 21:03:15.277242 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1215 21:03:15.277699 (+ 457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:03:15.277711 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:03:15.277812 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:03:15.278384 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1215 21:03:15.278400 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
1215 21:03:15.280380 (+ 1980us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.113.62:36095: Server oryx/127.28.113.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":99,"threads_started":1}
W20251215 21:03:15.280606 6547 negotiation.cc:337] Failed RPC negotiation. Trace:
1215 21:03:15.275466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.113.3:55309 (local address 127.28.113.62:36095)
1215 21:03:15.275625 (+ 159us) server_negotiation.cc:207] Beginning negotiation
1215 21:03:15.275629 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1215 21:03:15.275717 (+ 88us) server_negotiation.cc:408] Connection header received
1215 21:03:15.275852 (+ 135us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1215 21:03:15.275856 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1215 21:03:15.275903 (+ 47us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1215 21:03:15.276010 (+ 107us) server_negotiation.cc:227] Negotiated authn=SASL
1215 21:03:15.276533 (+ 523us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:03:15.277117 (+ 584us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1215 21:03:15.277808 (+ 691us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:03:15.277983 (+ 175us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:03:15.280508 (+ 2525us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.113.3:55309: BlockingRecv error: recv got EOF from 127.28.113.3:55309 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Dec 15 21:03:15 dist-test-slave-bjsf krb5kdc[5487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.113.1@KRBTEST.COM for oryx/127.28.113.62@KRBTEST.COM, Server not found in Kerberos database
Dec 15 21:03:15 dist-test-slave-bjsf krb5kdc[5487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.113.1@KRBTEST.COM for oryx/127.28.113.62@KRBTEST.COM, Server not found in Kerberos database
I20251215 21:03:15.958557 6550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:03:15.951531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.113.62:36095 (local address 127.28.113.1:45931)
1215 21:03:15.951809 (+ 278us) negotiation.cc:107] Waiting for socket to connect
1215 21:03:15.951822 (+ 13us) client_negotiation.cc:174] Beginning negotiation
1215 21:03:15.951914 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1215 21:03:15.952162 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1215 21:03:15.952164 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1215 21:03:15.952335 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
1215 21:03:15.952530 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:03:15.952536 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:03:15.953231 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1215 21:03:15.953233 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1215 21:03:15.953604 (+ 371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:03:15.953610 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:03:15.953689 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:03:15.956784 (+ 3095us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1215 21:03:15.956798 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
1215 21:03:15.958439 (+ 1641us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.113.62:36095: Server oryx/127.28.113.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":88,"threads_started":1}
W20251215 21:03:15.958701 6551 negotiation.cc:337] Failed RPC negotiation. Trace:
1215 21:03:15.951630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.113.1:45931 (local address 127.28.113.62:36095)
1215 21:03:15.951899 (+ 269us) server_negotiation.cc:207] Beginning negotiation
1215 21:03:15.951903 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1215 21:03:15.951917 (+ 14us) server_negotiation.cc:408] Connection header received
1215 21:03:15.952014 (+ 97us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1215 21:03:15.952017 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1215 21:03:15.952063 (+ 46us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1215 21:03:15.952136 (+ 73us) server_negotiation.cc:227] Negotiated authn=SASL
1215 21:03:15.952623 (+ 487us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:03:15.953141 (+ 518us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1215 21:03:15.953680 (+ 539us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:03:15.953805 (+ 125us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:03:15.958580 (+ 4775us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.113.1:45931: BlockingRecv error: recv got EOF from 127.28.113.1:45931 (error 108)
Metrics: {"server-negotiator.queue_time_us":199,"thread_start_us":83,"threads_started":1}
Dec 15 21:03:16 dist-test-slave-bjsf krb5kdc[5487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.113.2@KRBTEST.COM for oryx/127.28.113.62@KRBTEST.COM, Server not found in Kerberos database
Dec 15 21:03:16 dist-test-slave-bjsf krb5kdc[5487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.113.2@KRBTEST.COM for oryx/127.28.113.62@KRBTEST.COM, Server not found in Kerberos database
I20251215 21:03:16.109144 6552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:03:16.104840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.113.62:36095 (local address 127.28.113.2:47967)
1215 21:03:16.105095 (+ 255us) negotiation.cc:107] Waiting for socket to connect
1215 21:03:16.105109 (+ 14us) client_negotiation.cc:174] Beginning negotiation
1215 21:03:16.105242 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1215 21:03:16.105457 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1215 21:03:16.105460 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1215 21:03:16.105634 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
1215 21:03:16.105826 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:03:16.105831 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:03:16.106515 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1215 21:03:16.106517 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1215 21:03:16.106890 (+ 373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:03:16.106896 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:03:16.106973 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:03:16.107468 (+ 495us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1215 21:03:16.107481 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
1215 21:03:16.109032 (+ 1551us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.113.62:36095: Server oryx/127.28.113.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":74,"threads_started":1}
W20251215 21:03:16.109278 6551 negotiation.cc:337] Failed RPC negotiation. Trace:
1215 21:03:16.104894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.113.2:47967 (local address 127.28.113.62:36095)
1215 21:03:16.105029 (+ 135us) server_negotiation.cc:207] Beginning negotiation
1215 21:03:16.105033 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1215 21:03:16.105209 (+ 176us) server_negotiation.cc:408] Connection header received
1215 21:03:16.105319 (+ 110us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1215 21:03:16.105321 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1215 21:03:16.105369 (+ 48us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1215 21:03:16.105432 (+ 63us) server_negotiation.cc:227] Negotiated authn=SASL
1215 21:03:16.105920 (+ 488us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:03:16.106412 (+ 492us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1215 21:03:16.106974 (+ 562us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:03:16.107099 (+ 125us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:03:16.109162 (+ 2063us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.113.2:47967: BlockingRecv error: recv got EOF from 127.28.113.2:47967 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Dec 15 21:03:16 dist-test-slave-bjsf krb5kdc[5487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.113.3@KRBTEST.COM for oryx/127.28.113.62@KRBTEST.COM, Server not found in Kerberos database
Dec 15 21:03:16 dist-test-slave-bjsf krb5kdc[5487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.113.3@KRBTEST.COM for oryx/127.28.113.62@KRBTEST.COM, Server not found in Kerberos database
I20251215 21:03:16.286818 6553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:03:16.281565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.113.62:36095 (local address 127.28.113.3:34657)
1215 21:03:16.281836 (+ 271us) negotiation.cc:107] Waiting for socket to connect
1215 21:03:16.281850 (+ 14us) client_negotiation.cc:174] Beginning negotiation
1215 21:03:16.281972 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1215 21:03:16.282248 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1215 21:03:16.282252 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
1215 21:03:16.282472 (+ 220us) client_negotiation.cc:189] Negotiated authn=SASL
1215 21:03:16.282734 (+ 262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:03:16.282760 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:03:16.283721 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1215 21:03:16.283744 (+ 23us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1215 21:03:16.284186 (+ 442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:03:16.284196 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:03:16.284299 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:03:16.284917 (+ 618us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1215 21:03:16.284935 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
1215 21:03:16.286672 (+ 1737us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.113.62:36095: Server oryx/127.28.113.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":91,"threads_started":1}
W20251215 21:03:16.286896 6551 negotiation.cc:337] Failed RPC negotiation. Trace:
1215 21:03:16.281616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.113.3:34657 (local address 127.28.113.62:36095)
1215 21:03:16.281788 (+ 172us) server_negotiation.cc:207] Beginning negotiation
1215 21:03:16.281793 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1215 21:03:16.281929 (+ 136us) server_negotiation.cc:408] Connection header received
1215 21:03:16.282067 (+ 138us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1215 21:03:16.282070 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1215 21:03:16.282128 (+ 58us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1215 21:03:16.282198 (+ 70us) server_negotiation.cc:227] Negotiated authn=SASL
1215 21:03:16.282884 (+ 686us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:03:16.283615 (+ 731us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1215 21:03:16.284297 (+ 682us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:03:16.284471 (+ 174us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:03:16.286800 (+ 2329us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.113.3:34657: BlockingRecv error: recv got EOF from 127.28.113.3:34657 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Dec 15 21:03:16 dist-test-slave-bjsf krb5kdc[5487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.113.1@KRBTEST.COM for oryx/127.28.113.62@KRBTEST.COM, Server not found in Kerberos database
Dec 15 21:03:16 dist-test-slave-bjsf krb5kdc[5487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.113.1@KRBTEST.COM for oryx/127.28.113.62@KRBTEST.COM, Server not found in Kerberos database
I20251215 21:03:16.963616 6554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:03:16.959320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.113.62:36095 (local address 127.28.113.1:51799)
1215 21:03:16.959582 (+ 262us) negotiation.cc:107] Waiting for socket to connect
1215 21:03:16.959594 (+ 12us) client_negotiation.cc:174] Beginning negotiation
1215 21:03:16.959687 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1215 21:03:16.959900 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1215 21:03:16.959902 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1215 21:03:16.960101 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
1215 21:03:16.960358 (+ 257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:03:16.960366 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:03:16.961052 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1215 21:03:16.961054 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1215 21:03:16.961420 (+ 366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:03:16.961425 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:03:16.961501 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:03:16.961984 (+ 483us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1215 21:03:16.961995 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
1215 21:03:16.963506 (+ 1511us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.113.62:36095: Server oryx/127.28.113.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":84,"threads_started":1}
W20251215 21:03:16.963773 6555 negotiation.cc:337] Failed RPC negotiation. Trace:
1215 21:03:16.959390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.113.1:51799 (local address 127.28.113.62:36095)
1215 21:03:16.959603 (+ 213us) server_negotiation.cc:207] Beginning negotiation
1215 21:03:16.959606 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1215 21:03:16.959662 (+ 56us) server_negotiation.cc:408] Connection header received
1215 21:03:16.959757 (+ 95us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1215 21:03:16.959759 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1215 21:03:16.959806 (+ 47us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1215 21:03:16.959876 (+ 70us) server_negotiation.cc:227] Negotiated authn=SASL
1215 21:03:16.960459 (+ 583us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:03:16.960970 (+ 511us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1215 21:03:16.961490 (+ 520us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:03:16.961610 (+ 120us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:03:16.963646 (+ 2036us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.113.1:51799: BlockingRecv error: recv got EOF from 127.28.113.1:51799 (error 108)
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":57,"threads_started":1}
Dec 15 21:03:17 dist-test-slave-bjsf krb5kdc[5487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.113.2@KRBTEST.COM for oryx/127.28.113.62@KRBTEST.COM, Server not found in Kerberos database
Dec 15 21:03:17 dist-test-slave-bjsf krb5kdc[5487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.113.2@KRBTEST.COM for oryx/127.28.113.62@KRBTEST.COM, Server not found in Kerberos database
I20251215 21:03:17.114362 6556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:03:17.109879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.113.62:36095 (local address 127.28.113.2:58823)
1215 21:03:17.110147 (+ 268us) negotiation.cc:107] Waiting for socket to connect
1215 21:03:17.110160 (+ 13us) client_negotiation.cc:174] Beginning negotiation
1215 21:03:17.110260 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1215 21:03:17.110482 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1215 21:03:17.110486 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
1215 21:03:17.110681 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
1215 21:03:17.110868 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:03:17.110874 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:03:17.111599 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1215 21:03:17.111602 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1215 21:03:17.112001 (+ 399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:03:17.112008 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:03:17.112106 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:03:17.112614 (+ 508us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1215 21:03:17.112626 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
1215 21:03:17.114247 (+ 1621us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.113.62:36095: Server oryx/127.28.113.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":85,"threads_started":1}
W20251215 21:03:17.114475 6555 negotiation.cc:337] Failed RPC negotiation. Trace:
1215 21:03:17.109962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.113.2:58823 (local address 127.28.113.62:36095)
1215 21:03:17.110106 (+ 144us) server_negotiation.cc:207] Beginning negotiation
1215 21:03:17.110110 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1215 21:03:17.110221 (+ 111us) server_negotiation.cc:408] Connection header received
1215 21:03:17.110345 (+ 124us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1215 21:03:17.110348 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1215 21:03:17.110390 (+ 42us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1215 21:03:17.110451 (+ 61us) server_negotiation.cc:227] Negotiated authn=SASL
1215 21:03:17.111002 (+ 551us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:03:17.111498 (+ 496us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1215 21:03:17.112104 (+ 606us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:03:17.112283 (+ 179us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:03:17.114375 (+ 2092us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.113.2:58823: BlockingRecv error: recv got EOF from 127.28.113.2:58823 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
I20251215 21:03:17.209025 29124 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu with pid 6079
I20251215 21:03:17.213991 29124 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu with pid 6232
I20251215 21:03:17.218808 29124 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu with pid 6364
I20251215 21:03:17.223433 29124 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu with pid 6009
2025-12-15T21:03:17Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (18649 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2025-12-15T21:03:17Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-12-15T21:03:17Z Disabled control of system clock
I20251215 21:03:17.250506 29124 external_mini_cluster.cc:1366] Running /tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu
/tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1765832506145157-29124-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1765832506145157-29124-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1765832506145157-29124-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1765832506145157-29124-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.113.62:44373
--webserver_interface=127.28.113.62
--webserver_port=0
--builtin_ntp_servers=127.28.113.20:32999
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.113.62:44373
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20251215 21:03:17.355154 6560 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251215 21:03:17.355383 6560 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251215 21:03:17.355424 6560 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251215 21:03:17.358588 6560 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251215 21:03:17.358654 6560 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251215 21:03:17.358669 6560 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251215 21:03:17.358692 6560 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251215 21:03:17.358716 6560 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251215 21:03:17.362409 6560 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.113.20:32999
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1765832506145157-29124-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1765832506145157-29124-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.113.62:44373
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.113.62:44373
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1765832506145157-29124-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.113.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6560
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1765832506145157-29124-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision dc741732bb6a40213e11a13869fad06a49a209a3
build type DEBUG
built by None at 15 Dec 2025 20:43:14 UTC on 24a791456cd2
build id 9530
I20251215 21:03:17.363488 6560 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251215 21:03:17.364578 6560 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251215 21:03:17.370065 6565 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251215 21:03:17.370175 6560 server_base.cc:1047] running on GCE node
W20251215 21:03:17.370198 6568 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251215 21:03:17.370065 6566 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251215 21:03:17.370726 6560 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251215 21:03:17.371600 6560 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251215 21:03:17.372750 6560 hybrid_clock.cc:648] HybridClock initialized: now 1765832597372735 us; error 27 us; skew 500 ppm
I20251215 21:03:17.374473 6560 webserver.cc:492] Webserver started at http://127.28.113.62:39091/ using document root <none> and password file <none>
I20251215 21:03:17.375029 6560 fs_manager.cc:362] Metadata directory not provided
I20251215 21:03:17.375085 6560 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251215 21:03:17.375275 6560 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251215 21:03:17.376972 6560 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1765832506145157-29124-0/minicluster-data/master-0/data/instance:
uuid: "a1341342868a466a99242e6719fb93fc"
format_stamp: "Formatted at 2025-12-15 21:03:17 on dist-test-slave-bjsf"
server_key: "0560a0a281e590766f88a2f623eeb91c"
server_key_iv: "7735133f8075cdb152d80c605004dad0"
server_key_version: "encryptionkey@0"
I20251215 21:03:17.377444 6560 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1765832506145157-29124-0/minicluster-data/master-0/wal/instance:
uuid: "a1341342868a466a99242e6719fb93fc"
format_stamp: "Formatted at 2025-12-15 21:03:17 on dist-test-slave-bjsf"
server_key: "0560a0a281e590766f88a2f623eeb91c"
server_key_iv: "7735133f8075cdb152d80c605004dad0"
server_key_version: "encryptionkey@0"
I20251215 21:03:17.380685 6560 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20251215 21:03:17.382820 6574 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251215 21:03:17.383770 6560 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20251215 21:03:17.383869 6560 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1765832506145157-29124-0/minicluster-data/master-0/data,/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1765832506145157-29124-0/minicluster-data/master-0/wal
uuid: "a1341342868a466a99242e6719fb93fc"
format_stamp: "Formatted at 2025-12-15 21:03:17 on dist-test-slave-bjsf"
server_key: "0560a0a281e590766f88a2f623eeb91c"
server_key_iv: "7735133f8075cdb152d80c605004dad0"
server_key_version: "encryptionkey@0"
I20251215 21:03:17.383996 6560 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1765832506145157-29124-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1765832506145157-29124-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1765832506145157-29124-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251215 21:03:17.397045 6560 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251215 21:03:17.397655 6560 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251215 21:03:17.397818 6560 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251215 21:03:17.404693 6560 rpc_server.cc:307] RPC server started. Bound to: 127.28.113.62:44373
I20251215 21:03:17.404690 6626 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.28.113.62:44373 every 8 connection(s)
I20251215 21:03:17.405791 6560 server_base.cc:1179] Dumped server information to /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1765832506145157-29124-0/minicluster-data/master-0/data/info.pb
I20251215 21:03:17.408629 6627 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 21:03:17.413901 6627 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a1341342868a466a99242e6719fb93fc: Bootstrap starting.
I20251215 21:03:17.414731 29124 external_mini_cluster.cc:1428] Started /tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu as pid 6560
I20251215 21:03:17.414850 29124 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1765832506145157-29124-0/minicluster-data/master-0/wal/instance
I20251215 21:03:17.415132 29124 external_mini_cluster.cc:1468] Setting key 2f4a8a88abcfba5c45a288dc09c49336
I20251215 21:03:17.416586 6627 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a1341342868a466a99242e6719fb93fc: Neither blocks nor log segments found. Creating new log.
I20251215 21:03:17.417259 6627 log.cc:826] T 00000000000000000000000000000000 P a1341342868a466a99242e6719fb93fc: Log is configured to *not* fsync() on all Append() calls
I20251215 21:03:17.418895 6627 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a1341342868a466a99242e6719fb93fc: No bootstrap required, opened a new log
I20251215 21:03:17.421437 6627 raft_consensus.cc:359] T 00000000000000000000000000000000 P a1341342868a466a99242e6719fb93fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1341342868a466a99242e6719fb93fc" member_type: VOTER last_known_addr { host: "127.28.113.62" port: 44373 } }
I20251215 21:03:17.421654 6627 raft_consensus.cc:385] T 00000000000000000000000000000000 P a1341342868a466a99242e6719fb93fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251215 21:03:17.421715 6627 raft_consensus.cc:740] T 00000000000000000000000000000000 P a1341342868a466a99242e6719fb93fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1341342868a466a99242e6719fb93fc, State: Initialized, Role: FOLLOWER
I20251215 21:03:17.422123 6627 consensus_queue.cc:260] T 00000000000000000000000000000000 P a1341342868a466a99242e6719fb93fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1341342868a466a99242e6719fb93fc" member_type: VOTER last_known_addr { host: "127.28.113.62" port: 44373 } }
I20251215 21:03:17.422230 6630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:03:17.416164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46548 (local address 127.28.113.62:44373)
1215 21:03:17.416808 (+ 644us) server_negotiation.cc:207] Beginning negotiation
1215 21:03:17.416817 (+ 9us) server_negotiation.cc:400] Waiting for connection header
1215 21:03:17.416850 (+ 33us) server_negotiation.cc:408] Connection header received
1215 21:03:17.417386 (+ 536us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1215 21:03:17.417409 (+ 23us) server_negotiation.cc:462] Received NEGOTIATE request from client
1215 21:03:17.417752 (+ 343us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1215 21:03:17.418042 (+ 290us) server_negotiation.cc:227] Negotiated authn=SASL
1215 21:03:17.418520 (+ 478us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:03:17.419282 (+ 762us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1215 21:03:17.419917 (+ 635us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:03:17.420210 (+ 293us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:03:17.420706 (+ 496us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1215 21:03:17.420725 (+ 19us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1215 21:03:17.420735 (+ 10us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1215 21:03:17.421116 (+ 381us) server_negotiation.cc:923] calling sasl_server_start()
1215 21:03:17.421166 (+ 50us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1215 21:03:17.421181 (+ 15us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1215 21:03:17.421274 (+ 93us) server_negotiation.cc:1036] Waiting for connection context
1215 21:03:17.421425 (+ 151us) server_negotiation.cc:300] Negotiation successful
1215 21:03:17.421628 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":290,"thread_start_us":122,"threads_started":1}
I20251215 21:03:17.422253 6627 raft_consensus.cc:399] T 00000000000000000000000000000000 P a1341342868a466a99242e6719fb93fc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251215 21:03:17.422353 6627 raft_consensus.cc:493] T 00000000000000000000000000000000 P a1341342868a466a99242e6719fb93fc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251215 21:03:17.422485 6627 raft_consensus.cc:3060] T 00000000000000000000000000000000 P a1341342868a466a99242e6719fb93fc [term 0 FOLLOWER]: Advancing to term 1
I20251215 21:03:17.423461 6627 raft_consensus.cc:515] T 00000000000000000000000000000000 P a1341342868a466a99242e6719fb93fc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1341342868a466a99242e6719fb93fc" member_type: VOTER last_known_addr { host: "127.28.113.62" port: 44373 } }
I20251215 21:03:17.423771 6627 leader_election.cc:304] T 00000000000000000000000000000000 P a1341342868a466a99242e6719fb93fc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a1341342868a466a99242e6719fb93fc; no voters:
I20251215 21:03:17.424057 6627 leader_election.cc:290] T 00000000000000000000000000000000 P a1341342868a466a99242e6719fb93fc [CANDIDATE]: Term 1 election: Requested vote from peers
I20251215 21:03:17.424160 6632 raft_consensus.cc:2804] T 00000000000000000000000000000000 P a1341342868a466a99242e6719fb93fc [term 1 FOLLOWER]: Leader election won for term 1
I20251215 21:03:17.424310 6632 raft_consensus.cc:697] T 00000000000000000000000000000000 P a1341342868a466a99242e6719fb93fc [term 1 LEADER]: Becoming Leader. State: Replica: a1341342868a466a99242e6719fb93fc, State: Running, Role: LEADER
I20251215 21:03:17.424638 6632 consensus_queue.cc:237] T 00000000000000000000000000000000 P a1341342868a466a99242e6719fb93fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1341342868a466a99242e6719fb93fc" member_type: VOTER last_known_addr { host: "127.28.113.62" port: 44373 } }
I20251215 21:03:17.425158 6627 sys_catalog.cc:565] T 00000000000000000000000000000000 P a1341342868a466a99242e6719fb93fc [sys.catalog]: configured and running, proceeding with master startup.
I20251215 21:03:17.426239 6633 sys_catalog.cc:455] T 00000000000000000000000000000000 P a1341342868a466a99242e6719fb93fc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a1341342868a466a99242e6719fb93fc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1341342868a466a99242e6719fb93fc" member_type: VOTER last_known_addr { host: "127.28.113.62" port: 44373 } } }
I20251215 21:03:17.426349 6633 sys_catalog.cc:458] T 00000000000000000000000000000000 P a1341342868a466a99242e6719fb93fc [sys.catalog]: This master's current role is: LEADER
I20251215 21:03:17.426544 6634 sys_catalog.cc:455] T 00000000000000000000000000000000 P a1341342868a466a99242e6719fb93fc [sys.catalog]: SysCatalogTable state changed. Reason: New leader a1341342868a466a99242e6719fb93fc. Latest consensus state: current_term: 1 leader_uuid: "a1341342868a466a99242e6719fb93fc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1341342868a466a99242e6719fb93fc" member_type: VOTER last_known_addr { host: "127.28.113.62" port: 44373 } } }
I20251215 21:03:17.426641 6634 sys_catalog.cc:458] T 00000000000000000000000000000000 P a1341342868a466a99242e6719fb93fc [sys.catalog]: This master's current role is: LEADER
I20251215 21:03:17.426689 6641 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251215 21:03:17.429565 6641 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251215 21:03:17.434520 6641 catalog_manager.cc:1357] Generated new cluster ID: d9188f03db0549e1abb34d833c54825c
I20251215 21:03:17.434597 6641 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251215 21:03:17.455892 6641 catalog_manager.cc:1380] Generated new certificate authority record
I20251215 21:03:17.456966 6641 catalog_manager.cc:1514] Loading token signing keys...
I20251215 21:03:17.463428 6641 catalog_manager.cc:6027] T 00000000000000000000000000000000 P a1341342868a466a99242e6719fb93fc: Generated new TSK 0
I20251215 21:03:17.464165 6641 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251215 21:03:17.470278 29124 external_mini_cluster.cc:1366] Running /tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu
/tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1765832506145157-29124-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1765832506145157-29124-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1765832506145157-29124-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1765832506145157-29124-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.113.1:0
--local_ip_for_outbound_sockets=127.28.113.1
--webserver_interface=127.28.113.1
--webserver_port=0
--tserver_master_addrs=127.28.113.62:44373
--builtin_ntp_servers=127.28.113.20:32999
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20251215 21:03:17.613461 6651 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251215 21:03:17.613755 6651 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251215 21:03:17.613837 6651 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251215 21:03:17.618392 6651 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251215 21:03:17.618479 6651 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251215 21:03:17.618592 6651 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.113.1
I20251215 21:03:17.623864 6651 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.113.20:32999
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1765832506145157-29124-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1765832506145157-29124-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.113.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1765832506145157-29124-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.113.1
--webserver_port=0
--tserver_master_addrs=127.28.113.62:44373
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6651
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.113.1
--log_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1765832506145157-29124-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision dc741732bb6a40213e11a13869fad06a49a209a3
build type DEBUG
built by None at 15 Dec 2025 20:43:14 UTC on 24a791456cd2
build id 9530
I20251215 21:03:17.625291 6651 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251215 21:03:17.626739 6651 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251215 21:03:17.634279 6657 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251215 21:03:17.634334 6659 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251215 21:03:17.634281 6656 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251215 21:03:17.634697 6651 server_base.cc:1047] running on GCE node
I20251215 21:03:17.635090 6651 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251215 21:03:17.635695 6651 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251215 21:03:17.636865 6651 hybrid_clock.cc:648] HybridClock initialized: now 1765832597636839 us; error 36 us; skew 500 ppm
I20251215 21:03:17.638944 6651 webserver.cc:492] Webserver started at http://127.28.113.1:36293/ using document root <none> and password file <none>
I20251215 21:03:17.639593 6651 fs_manager.cc:362] Metadata directory not provided
I20251215 21:03:17.639660 6651 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251215 21:03:17.639858 6651 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251215 21:03:17.641984 6651 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1765832506145157-29124-0/minicluster-data/ts-0/data/instance:
uuid: "81cbb87340a7452f9359334197e05414"
format_stamp: "Formatted at 2025-12-15 21:03:17 on dist-test-slave-bjsf"
server_key: "13b326a8df83dbb71db57725370038df"
server_key_iv: "91a7880fc2714a961d956e07f998784b"
server_key_version: "encryptionkey@0"
I20251215 21:03:17.642495 6651 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1765832506145157-29124-0/minicluster-data/ts-0/wal/instance:
uuid: "81cbb87340a7452f9359334197e05414"
format_stamp: "Formatted at 2025-12-15 21:03:17 on dist-test-slave-bjsf"
server_key: "13b326a8df83dbb71db57725370038df"
server_key_iv: "91a7880fc2714a961d956e07f998784b"
server_key_version: "encryptionkey@0"
I20251215 21:03:17.646600 6651 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20251215 21:03:17.649520 6665 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251215 21:03:17.650665 6651 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20251215 21:03:17.650782 6651 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1765832506145157-29124-0/minicluster-data/ts-0/data,/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1765832506145157-29124-0/minicluster-data/ts-0/wal
uuid: "81cbb87340a7452f9359334197e05414"
format_stamp: "Formatted at 2025-12-15 21:03:17 on dist-test-slave-bjsf"
server_key: "13b326a8df83dbb71db57725370038df"
server_key_iv: "91a7880fc2714a961d956e07f998784b"
server_key_version: "encryptionkey@0"
I20251215 21:03:17.650887 6651 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1765832506145157-29124-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1765832506145157-29124-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1765832506145157-29124-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251215 21:03:17.668973 6651 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251215 21:03:17.669531 6651 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251215 21:03:17.669658 6651 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251215 21:03:17.670133 6651 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251215 21:03:17.671028 6651 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251215 21:03:17.671074 6651 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251215 21:03:17.671113 6651 ts_tablet_manager.cc:616] Registered 0 tablets
I20251215 21:03:17.671132 6651 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251215 21:03:17.679522 6651 rpc_server.cc:307] RPC server started. Bound to: 127.28.113.1:33207
I20251215 21:03:17.679548 6778 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.28.113.1:33207 every 8 connection(s)
I20251215 21:03:17.680469 6651 server_base.cc:1179] Dumped server information to /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1765832506145157-29124-0/minicluster-data/ts-0/data/info.pb
I20251215 21:03:17.685907 29124 external_mini_cluster.cc:1428] Started /tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu as pid 6651
I20251215 21:03:17.686019 29124 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1765832506145157-29124-0/minicluster-data/ts-0/wal/instance
I20251215 21:03:17.686252 29124 external_mini_cluster.cc:1468] Setting key 39990c82f5a9f19d379f5d0f1d2a12f5
I20251215 21:03:17.688153 29124 external_mini_cluster.cc:1366] Running /tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu
/tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1765832506145157-29124-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1765832506145157-29124-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1765832506145157-29124-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1765832506145157-29124-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.113.2:0
--local_ip_for_outbound_sockets=127.28.113.2
--webserver_interface=127.28.113.2
--webserver_port=0
--tserver_master_addrs=127.28.113.62:44373
--builtin_ntp_servers=127.28.113.20:32999
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20251215 21:03:17.688235 6630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:03:17.682204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.113.1:51915 (local address 127.28.113.62:44373)
1215 21:03:17.682325 (+ 121us) server_negotiation.cc:207] Beginning negotiation
1215 21:03:17.682328 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1215 21:03:17.683232 (+ 904us) server_negotiation.cc:408] Connection header received
1215 21:03:17.684059 (+ 827us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1215 21:03:17.684062 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1215 21:03:17.684126 (+ 64us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1215 21:03:17.684204 (+ 78us) server_negotiation.cc:227] Negotiated authn=SASL
1215 21:03:17.685170 (+ 966us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:03:17.685691 (+ 521us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1215 21:03:17.686364 (+ 673us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:03:17.686552 (+ 188us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:03:17.687731 (+ 1179us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1215 21:03:17.687747 (+ 16us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1215 21:03:17.687751 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1215 21:03:17.687763 (+ 12us) server_negotiation.cc:923] calling sasl_server_start()
1215 21:03:17.687785 (+ 22us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1215 21:03:17.687794 (+ 9us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1215 21:03:17.687851 (+ 57us) server_negotiation.cc:1036] Waiting for connection context
1215 21:03:17.688028 (+ 177us) server_negotiation.cc:300] Negotiation successful
1215 21:03:17.688096 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20251215 21:03:17.688925 6782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:03:17.682473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.113.62:44373 (local address 127.28.113.1:51915)
1215 21:03:17.683099 (+ 626us) negotiation.cc:107] Waiting for socket to connect
1215 21:03:17.683134 (+ 35us) client_negotiation.cc:174] Beginning negotiation
1215 21:03:17.683840 (+ 706us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1215 21:03:17.684371 (+ 531us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1215 21:03:17.684379 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
1215 21:03:17.684457 (+ 78us) client_negotiation.cc:189] Negotiated authn=SASL
1215 21:03:17.685021 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:03:17.685033 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:03:17.685801 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1215 21:03:17.685804 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1215 21:03:17.686244 (+ 440us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:03:17.686252 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:03:17.686454 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:03:17.687518 (+ 1064us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
1215 21:03:17.687520 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
1215 21:03:17.687554 (+ 34us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
1215 21:03:17.687613 (+ 59us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1215 21:03:17.687867 (+ 254us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1215 21:03:17.687876 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1215 21:03:17.687881 (+ 5us) client_negotiation.cc:769] Sending connection context
1215 21:03:17.688038 (+ 157us) client_negotiation.cc:240] Negotiation successful
1215 21:03:17.688297 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":93,"threads_started":1}
I20251215 21:03:17.689957 6779 heartbeater.cc:344] Connected to a master server at 127.28.113.62:44373
I20251215 21:03:17.690181 6779 heartbeater.cc:461] Registering TS with master...
I20251215 21:03:17.690708 6779 heartbeater.cc:507] Master 127.28.113.62:44373 requested a full tablet report, sending...
I20251215 21:03:17.692262 6591 ts_manager.cc:194] Registered new tserver with Master: 81cbb87340a7452f9359334197e05414 (127.28.113.1:33207)
I20251215 21:03:17.693876 6591 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.28.113.1:51915
W20251215 21:03:17.791309 6783 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251215 21:03:17.791525 6783 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251215 21:03:17.791569 6783 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251215 21:03:17.794787 6783 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251215 21:03:17.794848 6783 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251215 21:03:17.794915 6783 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.113.2
I20251215 21:03:17.798653 6783 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.113.20:32999
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1765832506145157-29124-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1765832506145157-29124-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.113.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1765832506145157-29124-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.113.2
--webserver_port=0
--tserver_master_addrs=127.28.113.62:44373
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6783
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.113.2
--log_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1765832506145157-29124-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision dc741732bb6a40213e11a13869fad06a49a209a3
build type DEBUG
built by None at 15 Dec 2025 20:43:14 UTC on 24a791456cd2
build id 9530
I20251215 21:03:17.799672 6783 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251215 21:03:17.800727 6783 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251215 21:03:17.807412 6791 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251215 21:03:17.807423 6788 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251215 21:03:17.807523 6783 server_base.cc:1047] running on GCE node
W20251215 21:03:17.807427 6789 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251215 21:03:17.808104 6783 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251215 21:03:17.808624 6783 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251215 21:03:17.809768 6783 hybrid_clock.cc:648] HybridClock initialized: now 1765832597809748 us; error 35 us; skew 500 ppm
I20251215 21:03:17.811520 6783 webserver.cc:492] Webserver started at http://127.28.113.2:46041/ using document root <none> and password file <none>
I20251215 21:03:17.812099 6783 fs_manager.cc:362] Metadata directory not provided
I20251215 21:03:17.812161 6783 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251215 21:03:17.812341 6783 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251215 21:03:17.814028 6783 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1765832506145157-29124-0/minicluster-data/ts-1/data/instance:
uuid: "eb3f839d4389487094bed4de1d3ef913"
format_stamp: "Formatted at 2025-12-15 21:03:17 on dist-test-slave-bjsf"
server_key: "b7e83fe71b814e73849db5778f86dde1"
server_key_iv: "1cfc11573cfd2e3beb9f797b576e52a5"
server_key_version: "encryptionkey@0"
I20251215 21:03:17.814471 6783 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1765832506145157-29124-0/minicluster-data/ts-1/wal/instance:
uuid: "eb3f839d4389487094bed4de1d3ef913"
format_stamp: "Formatted at 2025-12-15 21:03:17 on dist-test-slave-bjsf"
server_key: "b7e83fe71b814e73849db5778f86dde1"
server_key_iv: "1cfc11573cfd2e3beb9f797b576e52a5"
server_key_version: "encryptionkey@0"
I20251215 21:03:17.817761 6783 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20251215 21:03:17.819857 6797 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251215 21:03:17.820839 6783 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20251215 21:03:17.820942 6783 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1765832506145157-29124-0/minicluster-data/ts-1/data,/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1765832506145157-29124-0/minicluster-data/ts-1/wal
uuid: "eb3f839d4389487094bed4de1d3ef913"
format_stamp: "Formatted at 2025-12-15 21:03:17 on dist-test-slave-bjsf"
server_key: "b7e83fe71b814e73849db5778f86dde1"
server_key_iv: "1cfc11573cfd2e3beb9f797b576e52a5"
server_key_version: "encryptionkey@0"
I20251215 21:03:17.821036 6783 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1765832506145157-29124-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1765832506145157-29124-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1765832506145157-29124-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251215 21:03:17.836930 6783 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251215 21:03:17.837533 6783 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251215 21:03:17.837685 6783 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251215 21:03:17.838219 6783 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251215 21:03:17.839191 6783 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251215 21:03:17.839238 6783 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251215 21:03:17.839290 6783 ts_tablet_manager.cc:616] Registered 0 tablets
I20251215 21:03:17.839318 6783 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251215 21:03:17.848050 6783 rpc_server.cc:307] RPC server started. Bound to: 127.28.113.2:45509
I20251215 21:03:17.848074 6910 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.28.113.2:45509 every 8 connection(s)
I20251215 21:03:17.848956 6783 server_base.cc:1179] Dumped server information to /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1765832506145157-29124-0/minicluster-data/ts-1/data/info.pb
I20251215 21:03:17.853639 29124 external_mini_cluster.cc:1428] Started /tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu as pid 6783
I20251215 21:03:17.853758 29124 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1765832506145157-29124-0/minicluster-data/ts-1/wal/instance
I20251215 21:03:17.854075 29124 external_mini_cluster.cc:1468] Setting key 9dc215cd31ab6459aeb79f5da5acf7cb
I20251215 21:03:17.856215 29124 external_mini_cluster.cc:1366] Running /tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu
/tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1765832506145157-29124-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1765832506145157-29124-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1765832506145157-29124-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1765832506145157-29124-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.113.3:0
--local_ip_for_outbound_sockets=127.28.113.3
--webserver_interface=127.28.113.3
--webserver_port=0
--tserver_master_addrs=127.28.113.62:44373
--builtin_ntp_servers=127.28.113.20:32999
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20251215 21:03:17.857247 6630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:03:17.850703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.113.2:46103 (local address 127.28.113.62:44373)
1215 21:03:17.850851 (+ 148us) server_negotiation.cc:207] Beginning negotiation
1215 21:03:17.850856 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1215 21:03:17.851776 (+ 920us) server_negotiation.cc:408] Connection header received
1215 21:03:17.852612 (+ 836us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1215 21:03:17.852616 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1215 21:03:17.852670 (+ 54us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1215 21:03:17.852749 (+ 79us) server_negotiation.cc:227] Negotiated authn=SASL
1215 21:03:17.853701 (+ 952us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:03:17.854424 (+ 723us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1215 21:03:17.855164 (+ 740us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:03:17.855329 (+ 165us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:03:17.856672 (+ 1343us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1215 21:03:17.856688 (+ 16us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1215 21:03:17.856690 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1215 21:03:17.856702 (+ 12us) server_negotiation.cc:923] calling sasl_server_start()
1215 21:03:17.856722 (+ 20us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1215 21:03:17.856729 (+ 7us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1215 21:03:17.856861 (+ 132us) server_negotiation.cc:1036] Waiting for connection context
1215 21:03:17.857034 (+ 173us) server_negotiation.cc:300] Negotiation successful
1215 21:03:17.857099 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20251215 21:03:17.857920 6913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:03:17.851028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.113.62:44373 (local address 127.28.113.2:46103)
1215 21:03:17.851645 (+ 617us) negotiation.cc:107] Waiting for socket to connect
1215 21:03:17.851678 (+ 33us) client_negotiation.cc:174] Beginning negotiation
1215 21:03:17.852418 (+ 740us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1215 21:03:17.852894 (+ 476us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1215 21:03:17.852902 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
1215 21:03:17.852988 (+ 86us) client_negotiation.cc:189] Negotiated authn=SASL
1215 21:03:17.853548 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:03:17.853559 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:03:17.854550 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1215 21:03:17.854553 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1215 21:03:17.855025 (+ 472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:03:17.855035 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:03:17.855245 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:03:17.856458 (+ 1213us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
1215 21:03:17.856459 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
1215 21:03:17.856492 (+ 33us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
1215 21:03:17.856545 (+ 53us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1215 21:03:17.856875 (+ 330us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1215 21:03:17.856885 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1215 21:03:17.856891 (+ 6us) client_negotiation.cc:769] Sending connection context
1215 21:03:17.857027 (+ 136us) client_negotiation.cc:240] Negotiation successful
1215 21:03:17.857265 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":111,"threads_started":1}
I20251215 21:03:17.858915 6911 heartbeater.cc:344] Connected to a master server at 127.28.113.62:44373
I20251215 21:03:17.859150 6911 heartbeater.cc:461] Registering TS with master...
I20251215 21:03:17.859694 6911 heartbeater.cc:507] Master 127.28.113.62:44373 requested a full tablet report, sending...
I20251215 21:03:17.860697 6591 ts_manager.cc:194] Registered new tserver with Master: eb3f839d4389487094bed4de1d3ef913 (127.28.113.2:45509)
I20251215 21:03:17.861447 6591 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.28.113.2:46103
W20251215 21:03:17.959703 6914 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251215 21:03:17.959949 6914 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251215 21:03:17.960035 6914 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251215 21:03:17.963361 6914 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251215 21:03:17.963423 6914 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251215 21:03:17.963496 6914 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.113.3
I20251215 21:03:17.967409 6914 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.113.20:32999
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1765832506145157-29124-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1765832506145157-29124-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.113.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1765832506145157-29124-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.113.3
--webserver_port=0
--tserver_master_addrs=127.28.113.62:44373
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6914
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.113.3
--log_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1765832506145157-29124-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision dc741732bb6a40213e11a13869fad06a49a209a3
build type DEBUG
built by None at 15 Dec 2025 20:43:14 UTC on 24a791456cd2
build id 9530
I20251215 21:03:17.968503 6914 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251215 21:03:17.969628 6914 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251215 21:03:17.976112 6920 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251215 21:03:17.976048 6919 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251215 21:03:17.976291 6914 server_base.cc:1047] running on GCE node
W20251215 21:03:17.976048 6922 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251215 21:03:17.976647 6914 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251215 21:03:17.977192 6914 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251215 21:03:17.978343 6914 hybrid_clock.cc:648] HybridClock initialized: now 1765832597978333 us; error 30 us; skew 500 ppm
I20251215 21:03:17.980132 6914 webserver.cc:492] Webserver started at http://127.28.113.3:46549/ using document root <none> and password file <none>
I20251215 21:03:17.980667 6914 fs_manager.cc:362] Metadata directory not provided
I20251215 21:03:17.980726 6914 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251215 21:03:17.980902 6914 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251215 21:03:17.982604 6914 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1765832506145157-29124-0/minicluster-data/ts-2/data/instance:
uuid: "0a73c40f07304829ab3ffdbb3dc2bcfd"
format_stamp: "Formatted at 2025-12-15 21:03:17 on dist-test-slave-bjsf"
server_key: "feefae8fb4f7c73596d9d986b92d0414"
server_key_iv: "c94bb17daccaef9c2909300c42820a35"
server_key_version: "encryptionkey@0"
I20251215 21:03:17.983079 6914 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1765832506145157-29124-0/minicluster-data/ts-2/wal/instance:
uuid: "0a73c40f07304829ab3ffdbb3dc2bcfd"
format_stamp: "Formatted at 2025-12-15 21:03:17 on dist-test-slave-bjsf"
server_key: "feefae8fb4f7c73596d9d986b92d0414"
server_key_iv: "c94bb17daccaef9c2909300c42820a35"
server_key_version: "encryptionkey@0"
I20251215 21:03:17.986320 6914 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20251215 21:03:17.988585 6928 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251215 21:03:17.989559 6914 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20251215 21:03:17.989661 6914 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1765832506145157-29124-0/minicluster-data/ts-2/data,/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1765832506145157-29124-0/minicluster-data/ts-2/wal
uuid: "0a73c40f07304829ab3ffdbb3dc2bcfd"
format_stamp: "Formatted at 2025-12-15 21:03:17 on dist-test-slave-bjsf"
server_key: "feefae8fb4f7c73596d9d986b92d0414"
server_key_iv: "c94bb17daccaef9c2909300c42820a35"
server_key_version: "encryptionkey@0"
I20251215 21:03:17.989748 6914 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1765832506145157-29124-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1765832506145157-29124-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1765832506145157-29124-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251215 21:03:18.006942 6914 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251215 21:03:18.007539 6914 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251215 21:03:18.007686 6914 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251215 21:03:18.008239 6914 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251215 21:03:18.009184 6914 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251215 21:03:18.009230 6914 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251215 21:03:18.009280 6914 ts_tablet_manager.cc:616] Registered 0 tablets
I20251215 21:03:18.009306 6914 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251215 21:03:18.017696 6914 rpc_server.cc:307] RPC server started. Bound to: 127.28.113.3:33053
I20251215 21:03:18.017720 7041 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.28.113.3:33053 every 8 connection(s)
I20251215 21:03:18.018611 6914 server_base.cc:1179] Dumped server information to /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1765832506145157-29124-0/minicluster-data/ts-2/data/info.pb
I20251215 21:03:18.021536 29124 external_mini_cluster.cc:1428] Started /tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu as pid 6914
I20251215 21:03:18.021648 29124 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1765832506145157-29124-0/minicluster-data/ts-2/wal/instance
I20251215 21:03:18.021957 29124 external_mini_cluster.cc:1468] Setting key d4c584a59edded1fbcf3f3ac93072e3e
I20251215 21:03:18.026901 6630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:03:18.020356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.113.3:59729 (local address 127.28.113.62:44373)
1215 21:03:18.020492 (+ 136us) server_negotiation.cc:207] Beginning negotiation
1215 21:03:18.020496 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1215 21:03:18.021531 (+ 1035us) server_negotiation.cc:408] Connection header received
1215 21:03:18.022511 (+ 980us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1215 21:03:18.022514 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1215 21:03:18.022569 (+ 55us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1215 21:03:18.022638 (+ 69us) server_negotiation.cc:227] Negotiated authn=SASL
1215 21:03:18.023793 (+ 1155us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:03:18.024368 (+ 575us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1215 21:03:18.024968 (+ 600us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:03:18.025101 (+ 133us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:03:18.026410 (+ 1309us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1215 21:03:18.026427 (+ 17us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1215 21:03:18.026430 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1215 21:03:18.026442 (+ 12us) server_negotiation.cc:923] calling sasl_server_start()
1215 21:03:18.026462 (+ 20us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1215 21:03:18.026469 (+ 7us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1215 21:03:18.026527 (+ 58us) server_negotiation.cc:1036] Waiting for connection context
1215 21:03:18.026724 (+ 197us) server_negotiation.cc:300] Negotiation successful
1215 21:03:18.026778 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20251215 21:03:18.027612 7044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:03:18.020632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.113.62:44373 (local address 127.28.113.3:59729)
1215 21:03:18.021366 (+ 734us) negotiation.cc:107] Waiting for socket to connect
1215 21:03:18.021414 (+ 48us) client_negotiation.cc:174] Beginning negotiation
1215 21:03:18.022329 (+ 915us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1215 21:03:18.022818 (+ 489us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1215 21:03:18.022827 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
1215 21:03:18.022917 (+ 90us) client_negotiation.cc:189] Negotiated authn=SASL
1215 21:03:18.023655 (+ 738us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:03:18.023671 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:03:18.024479 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1215 21:03:18.024482 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1215 21:03:18.024871 (+ 389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:03:18.024878 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:03:18.025094 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:03:18.026201 (+ 1107us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
1215 21:03:18.026203 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
1215 21:03:18.026237 (+ 34us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
1215 21:03:18.026293 (+ 56us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1215 21:03:18.026570 (+ 277us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1215 21:03:18.026578 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1215 21:03:18.026582 (+ 4us) client_negotiation.cc:769] Sending connection context
1215 21:03:18.026723 (+ 141us) client_negotiation.cc:240] Negotiation successful
1215 21:03:18.026973 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":93,"threads_started":1}
I20251215 21:03:18.028630 7042 heartbeater.cc:344] Connected to a master server at 127.28.113.62:44373
I20251215 21:03:18.028847 7042 heartbeater.cc:461] Registering TS with master...
I20251215 21:03:18.029387 7042 heartbeater.cc:507] Master 127.28.113.62:44373 requested a full tablet report, sending...
I20251215 21:03:18.030328 6591 ts_manager.cc:194] Registered new tserver with Master: 0a73c40f07304829ab3ffdbb3dc2bcfd (127.28.113.3:33053)
I20251215 21:03:18.030862 6591 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.28.113.3:59729
I20251215 21:03:18.035641 29124 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
W20251215 21:03:18.038125 6630 negotiation.cc:337] Failed RPC negotiation. Trace:
1215 21:03:18.037417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51764 (local address 127.28.113.62:44373)
1215 21:03:18.037534 (+ 117us) server_negotiation.cc:207] Beginning negotiation
1215 21:03:18.037537 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1215 21:03:18.037650 (+ 113us) server_negotiation.cc:408] Connection header received
1215 21:03:18.037764 (+ 114us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1215 21:03:18.037766 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1215 21:03:18.037810 (+ 44us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1215 21:03:18.037864 (+ 54us) server_negotiation.cc:227] Negotiated authn=SASL
1215 21:03:18.038031 (+ 167us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51764: BlockingRecv error: recv got EOF from 127.0.0.1:51764 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
I20251215 21:03:18.038314 29124 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu with pid 6651
I20251215 21:03:18.043395 29124 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu with pid 6783
I20251215 21:03:18.048105 29124 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu with pid 6914
I20251215 21:03:18.052625 29124 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu with pid 6560
2025-12-15T21:03:18Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (828 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2025-12-15T21:03:18Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-12-15T21:03:18Z Disabled control of system clock
I20251215 21:03:18.078709 29124 external_mini_cluster.cc:1366] Running /tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu
/tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1765832506145157-29124-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1765832506145157-29124-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1765832506145157-29124-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1765832506145157-29124-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.113.62:44425
--webserver_interface=127.28.113.62
--webserver_port=0
--builtin_ntp_servers=127.28.113.20:34501
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.113.62:44425
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20251215 21:03:18.184213 7053 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251215 21:03:18.184458 7053 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251215 21:03:18.184509 7053 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251215 21:03:18.187893 7053 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251215 21:03:18.188002 7053 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251215 21:03:18.188019 7053 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251215 21:03:18.188037 7053 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251215 21:03:18.188057 7053 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251215 21:03:18.192130 7053 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.113.20:34501
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1765832506145157-29124-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1765832506145157-29124-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.113.62:44425
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.113.62:44425
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1765832506145157-29124-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.113.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7053
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1765832506145157-29124-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision dc741732bb6a40213e11a13869fad06a49a209a3
build type DEBUG
built by None at 15 Dec 2025 20:43:14 UTC on 24a791456cd2
build id 9530
I20251215 21:03:18.193267 7053 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251215 21:03:18.194375 7053 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251215 21:03:18.199772 7058 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251215 21:03:18.199781 7059 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251215 21:03:18.199842 7061 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251215 21:03:18.200241 7053 server_base.cc:1047] running on GCE node
I20251215 21:03:18.200758 7053 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251215 21:03:18.201695 7053 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251215 21:03:18.202808 7053 hybrid_clock.cc:648] HybridClock initialized: now 1765832598202798 us; error 31 us; skew 500 ppm
I20251215 21:03:18.204715 7053 webserver.cc:492] Webserver started at http://127.28.113.62:37467/ using document root <none> and password file <none>
I20251215 21:03:18.205273 7053 fs_manager.cc:362] Metadata directory not provided
I20251215 21:03:18.205327 7053 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251215 21:03:18.205511 7053 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251215 21:03:18.207195 7053 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1765832506145157-29124-0/minicluster-data/master-0/data/instance:
uuid: "7afbd4db6b1046b288b33251ab085349"
format_stamp: "Formatted at 2025-12-15 21:03:18 on dist-test-slave-bjsf"
server_key: "9fa9e3236f80f314e6c00e4b17ac5364"
server_key_iv: "92cab7f956e21d71b26812a38eba640c"
server_key_version: "encryptionkey@0"
I20251215 21:03:18.207664 7053 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1765832506145157-29124-0/minicluster-data/master-0/wal/instance:
uuid: "7afbd4db6b1046b288b33251ab085349"
format_stamp: "Formatted at 2025-12-15 21:03:18 on dist-test-slave-bjsf"
server_key: "9fa9e3236f80f314e6c00e4b17ac5364"
server_key_iv: "92cab7f956e21d71b26812a38eba640c"
server_key_version: "encryptionkey@0"
I20251215 21:03:18.210937 7053 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20251215 21:03:18.213369 7067 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251215 21:03:18.214443 7053 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20251215 21:03:18.214560 7053 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1765832506145157-29124-0/minicluster-data/master-0/data,/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1765832506145157-29124-0/minicluster-data/master-0/wal
uuid: "7afbd4db6b1046b288b33251ab085349"
format_stamp: "Formatted at 2025-12-15 21:03:18 on dist-test-slave-bjsf"
server_key: "9fa9e3236f80f314e6c00e4b17ac5364"
server_key_iv: "92cab7f956e21d71b26812a38eba640c"
server_key_version: "encryptionkey@0"
I20251215 21:03:18.214661 7053 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1765832506145157-29124-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1765832506145157-29124-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1765832506145157-29124-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251215 21:03:18.219437 7053 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251215 21:03:18.220055 7053 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251215 21:03:18.220206 7053 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251215 21:03:18.226926 7053 rpc_server.cc:307] RPC server started. Bound to: 127.28.113.62:44425
I20251215 21:03:18.226951 7119 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.28.113.62:44425 every 8 connection(s)
I20251215 21:03:18.228247 7053 server_base.cc:1179] Dumped server information to /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1765832506145157-29124-0/minicluster-data/master-0/data/info.pb
I20251215 21:03:18.231165 7120 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 21:03:18.233481 29124 external_mini_cluster.cc:1428] Started /tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu as pid 7053
I20251215 21:03:18.233587 29124 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1765832506145157-29124-0/minicluster-data/master-0/wal/instance
I20251215 21:03:18.233827 29124 external_mini_cluster.cc:1468] Setting key b583c90945aad93eccea24613d86794e
I20251215 21:03:18.237329 7120 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7afbd4db6b1046b288b33251ab085349: Bootstrap starting.
I20251215 21:03:18.239004 7123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:03:18.235170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49114 (local address 127.28.113.62:44425)
1215 21:03:18.235775 (+ 605us) server_negotiation.cc:207] Beginning negotiation
1215 21:03:18.235783 (+ 8us) server_negotiation.cc:400] Waiting for connection header
1215 21:03:18.235819 (+ 36us) server_negotiation.cc:408] Connection header received
1215 21:03:18.236399 (+ 580us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1215 21:03:18.236418 (+ 19us) server_negotiation.cc:462] Received NEGOTIATE request from client
1215 21:03:18.236749 (+ 331us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1215 21:03:18.237048 (+ 299us) server_negotiation.cc:227] Negotiated authn=SASL
1215 21:03:18.237905 (+ 857us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1215 21:03:18.237910 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1215 21:03:18.237920 (+ 10us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1215 21:03:18.237930 (+ 10us) server_negotiation.cc:923] calling sasl_server_start()
1215 21:03:18.237972 (+ 42us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1215 21:03:18.237986 (+ 14us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1215 21:03:18.238101 (+ 115us) server_negotiation.cc:1036] Waiting for connection context
1215 21:03:18.238203 (+ 102us) server_negotiation.cc:300] Negotiation successful
1215 21:03:18.238440 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":258,"thread_start_us":112,"threads_started":1}
I20251215 21:03:18.239809 7120 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7afbd4db6b1046b288b33251ab085349: Neither blocks nor log segments found. Creating new log.
I20251215 21:03:18.240605 7120 log.cc:826] T 00000000000000000000000000000000 P 7afbd4db6b1046b288b33251ab085349: Log is configured to *not* fsync() on all Append() calls
I20251215 21:03:18.242470 7120 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7afbd4db6b1046b288b33251ab085349: No bootstrap required, opened a new log
I20251215 21:03:18.245157 7120 raft_consensus.cc:359] T 00000000000000000000000000000000 P 7afbd4db6b1046b288b33251ab085349 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7afbd4db6b1046b288b33251ab085349" member_type: VOTER last_known_addr { host: "127.28.113.62" port: 44425 } }
I20251215 21:03:18.245369 7120 raft_consensus.cc:385] T 00000000000000000000000000000000 P 7afbd4db6b1046b288b33251ab085349 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251215 21:03:18.245419 7120 raft_consensus.cc:740] T 00000000000000000000000000000000 P 7afbd4db6b1046b288b33251ab085349 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7afbd4db6b1046b288b33251ab085349, State: Initialized, Role: FOLLOWER
I20251215 21:03:18.245859 7120 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7afbd4db6b1046b288b33251ab085349 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7afbd4db6b1046b288b33251ab085349" member_type: VOTER last_known_addr { host: "127.28.113.62" port: 44425 } }
I20251215 21:03:18.245990 7120 raft_consensus.cc:399] T 00000000000000000000000000000000 P 7afbd4db6b1046b288b33251ab085349 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251215 21:03:18.246052 7120 raft_consensus.cc:493] T 00000000000000000000000000000000 P 7afbd4db6b1046b288b33251ab085349 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251215 21:03:18.246147 7120 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 7afbd4db6b1046b288b33251ab085349 [term 0 FOLLOWER]: Advancing to term 1
I20251215 21:03:18.247015 7120 raft_consensus.cc:515] T 00000000000000000000000000000000 P 7afbd4db6b1046b288b33251ab085349 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7afbd4db6b1046b288b33251ab085349" member_type: VOTER last_known_addr { host: "127.28.113.62" port: 44425 } }
I20251215 21:03:18.247360 7120 leader_election.cc:304] T 00000000000000000000000000000000 P 7afbd4db6b1046b288b33251ab085349 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7afbd4db6b1046b288b33251ab085349; no voters:
I20251215 21:03:18.247637 7120 leader_election.cc:290] T 00000000000000000000000000000000 P 7afbd4db6b1046b288b33251ab085349 [CANDIDATE]: Term 1 election: Requested vote from peers
I20251215 21:03:18.247777 7125 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 7afbd4db6b1046b288b33251ab085349 [term 1 FOLLOWER]: Leader election won for term 1
I20251215 21:03:18.248013 7125 raft_consensus.cc:697] T 00000000000000000000000000000000 P 7afbd4db6b1046b288b33251ab085349 [term 1 LEADER]: Becoming Leader. State: Replica: 7afbd4db6b1046b288b33251ab085349, State: Running, Role: LEADER
I20251215 21:03:18.248309 7125 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7afbd4db6b1046b288b33251ab085349 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7afbd4db6b1046b288b33251ab085349" member_type: VOTER last_known_addr { host: "127.28.113.62" port: 44425 } }
I20251215 21:03:18.248762 7120 sys_catalog.cc:565] T 00000000000000000000000000000000 P 7afbd4db6b1046b288b33251ab085349 [sys.catalog]: configured and running, proceeding with master startup.
I20251215 21:03:18.249756 7126 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7afbd4db6b1046b288b33251ab085349 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7afbd4db6b1046b288b33251ab085349" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7afbd4db6b1046b288b33251ab085349" member_type: VOTER last_known_addr { host: "127.28.113.62" port: 44425 } } }
I20251215 21:03:18.249869 7126 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7afbd4db6b1046b288b33251ab085349 [sys.catalog]: This master's current role is: LEADER
I20251215 21:03:18.250281 7127 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7afbd4db6b1046b288b33251ab085349 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7afbd4db6b1046b288b33251ab085349. Latest consensus state: current_term: 1 leader_uuid: "7afbd4db6b1046b288b33251ab085349" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7afbd4db6b1046b288b33251ab085349" member_type: VOTER last_known_addr { host: "127.28.113.62" port: 44425 } } }
I20251215 21:03:18.250368 7127 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7afbd4db6b1046b288b33251ab085349 [sys.catalog]: This master's current role is: LEADER
W20251215 21:03:18.253391 7140 catalog_manager.cc:1568] T 00000000000000000000000000000000 P 7afbd4db6b1046b288b33251ab085349: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20251215 21:03:18.253469 7140 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20251215 21:03:18.253541 7134 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251215 21:03:18.254242 7134 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251215 21:03:18.259649 7134 catalog_manager.cc:1357] Generated new cluster ID: 8984a642571f40c0b86da06ad3d65cdd
I20251215 21:03:18.259723 7134 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251215 21:03:18.275426 7134 catalog_manager.cc:1380] Generated new certificate authority record
I20251215 21:03:18.275745 7134 catalog_manager.cc:1514] Loading token signing keys...
I20251215 21:03:18.285563 7134 catalog_manager.cc:6027] T 00000000000000000000000000000000 P 7afbd4db6b1046b288b33251ab085349: Generated new TSK 0
I20251215 21:03:18.286311 7134 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251215 21:03:18.293828 29124 external_mini_cluster.cc:1366] Running /tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu
/tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1765832506145157-29124-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1765832506145157-29124-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1765832506145157-29124-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1765832506145157-29124-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.113.1:0
--local_ip_for_outbound_sockets=127.28.113.1
--webserver_interface=127.28.113.1
--webserver_port=0
--tserver_master_addrs=127.28.113.62:44425
--builtin_ntp_servers=127.28.113.20:34501
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20251215 21:03:18.403918 7144 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251215 21:03:18.404199 7144 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251215 21:03:18.404269 7144 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251215 21:03:18.407603 7144 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251215 21:03:18.407672 7144 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251215 21:03:18.407752 7144 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.113.1
I20251215 21:03:18.411650 7144 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.113.20:34501
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1765832506145157-29124-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1765832506145157-29124-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.113.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1765832506145157-29124-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.113.1
--webserver_port=0
--tserver_master_addrs=127.28.113.62:44425
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7144
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.113.1
--log_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1765832506145157-29124-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision dc741732bb6a40213e11a13869fad06a49a209a3
build type DEBUG
built by None at 15 Dec 2025 20:43:14 UTC on 24a791456cd2
build id 9530
I20251215 21:03:18.412804 7144 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251215 21:03:18.413926 7144 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251215 21:03:18.420498 7150 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251215 21:03:18.420656 7144 server_base.cc:1047] running on GCE node
W20251215 21:03:18.420507 7152 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251215 21:03:18.420518 7149 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251215 21:03:18.421222 7144 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251215 21:03:18.421793 7144 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251215 21:03:18.422945 7144 hybrid_clock.cc:648] HybridClock initialized: now 1765832598422927 us; error 46 us; skew 500 ppm
I20251215 21:03:18.424861 7144 webserver.cc:492] Webserver started at http://127.28.113.1:42229/ using document root <none> and password file <none>
I20251215 21:03:18.425438 7144 fs_manager.cc:362] Metadata directory not provided
I20251215 21:03:18.425496 7144 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251215 21:03:18.425678 7144 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251215 21:03:18.427390 7144 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1765832506145157-29124-0/minicluster-data/ts-0/data/instance:
uuid: "8fee4ba63d0a484caef91db37d8732e8"
format_stamp: "Formatted at 2025-12-15 21:03:18 on dist-test-slave-bjsf"
server_key: "588b1571aec565824915997e36bb947a"
server_key_iv: "7791c4b68d68326fff1c327970849b1f"
server_key_version: "encryptionkey@0"
I20251215 21:03:18.427851 7144 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1765832506145157-29124-0/minicluster-data/ts-0/wal/instance:
uuid: "8fee4ba63d0a484caef91db37d8732e8"
format_stamp: "Formatted at 2025-12-15 21:03:18 on dist-test-slave-bjsf"
server_key: "588b1571aec565824915997e36bb947a"
server_key_iv: "7791c4b68d68326fff1c327970849b1f"
server_key_version: "encryptionkey@0"
I20251215 21:03:18.431303 7144 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20251215 21:03:18.433615 7158 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251215 21:03:18.434669 7144 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20251215 21:03:18.434788 7144 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1765832506145157-29124-0/minicluster-data/ts-0/data,/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1765832506145157-29124-0/minicluster-data/ts-0/wal
uuid: "8fee4ba63d0a484caef91db37d8732e8"
format_stamp: "Formatted at 2025-12-15 21:03:18 on dist-test-slave-bjsf"
server_key: "588b1571aec565824915997e36bb947a"
server_key_iv: "7791c4b68d68326fff1c327970849b1f"
server_key_version: "encryptionkey@0"
I20251215 21:03:18.434882 7144 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1765832506145157-29124-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1765832506145157-29124-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1765832506145157-29124-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251215 21:03:18.439708 7144 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251215 21:03:18.440348 7144 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251215 21:03:18.440508 7144 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251215 21:03:18.441020 7144 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251215 21:03:18.442004 7144 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251215 21:03:18.442051 7144 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251215 21:03:18.442102 7144 ts_tablet_manager.cc:616] Registered 0 tablets
I20251215 21:03:18.442129 7144 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251215 21:03:18.450850 7144 rpc_server.cc:307] RPC server started. Bound to: 127.28.113.1:37091
I20251215 21:03:18.450879 7271 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.28.113.1:37091 every 8 connection(s)
I20251215 21:03:18.451794 7144 server_base.cc:1179] Dumped server information to /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1765832506145157-29124-0/minicluster-data/ts-0/data/info.pb
I20251215 21:03:18.456995 7123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:03:18.453498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.113.1:48777 (local address 127.28.113.62:44425)
1215 21:03:18.453631 (+ 133us) server_negotiation.cc:207] Beginning negotiation
1215 21:03:18.453635 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1215 21:03:18.454587 (+ 952us) server_negotiation.cc:408] Connection header received
1215 21:03:18.455387 (+ 800us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1215 21:03:18.455391 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1215 21:03:18.455439 (+ 48us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1215 21:03:18.455518 (+ 79us) server_negotiation.cc:227] Negotiated authn=SASL
1215 21:03:18.456550 (+ 1032us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1215 21:03:18.456554 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1215 21:03:18.456557 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1215 21:03:18.456570 (+ 13us) server_negotiation.cc:923] calling sasl_server_start()
1215 21:03:18.456592 (+ 22us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1215 21:03:18.456599 (+ 7us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1215 21:03:18.456680 (+ 81us) server_negotiation.cc:1036] Waiting for connection context
1215 21:03:18.456807 (+ 127us) server_negotiation.cc:300] Negotiation successful
1215 21:03:18.456857 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20251215 21:03:18.457703 7274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:03:18.453782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.113.62:44425 (local address 127.28.113.1:48777)
1215 21:03:18.454451 (+ 669us) negotiation.cc:107] Waiting for socket to connect
1215 21:03:18.454487 (+ 36us) client_negotiation.cc:174] Beginning negotiation
1215 21:03:18.455190 (+ 703us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1215 21:03:18.455676 (+ 486us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1215 21:03:18.455685 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
1215 21:03:18.455769 (+ 84us) client_negotiation.cc:189] Negotiated authn=SASL
1215 21:03:18.456322 (+ 553us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
1215 21:03:18.456323 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
1215 21:03:18.456358 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
1215 21:03:18.456413 (+ 55us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1215 21:03:18.456668 (+ 255us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1215 21:03:18.456675 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1215 21:03:18.456680 (+ 5us) client_negotiation.cc:769] Sending connection context
1215 21:03:18.456867 (+ 187us) client_negotiation.cc:240] Negotiation successful
1215 21:03:18.457098 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":134,"threads_started":1}
I20251215 21:03:18.458757 7272 heartbeater.cc:344] Connected to a master server at 127.28.113.62:44425
I20251215 21:03:18.459000 7272 heartbeater.cc:461] Registering TS with master...
I20251215 21:03:18.459409 7272 heartbeater.cc:507] Master 127.28.113.62:44425 requested a full tablet report, sending...
I20251215 21:03:18.459756 29124 external_mini_cluster.cc:1428] Started /tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu as pid 7144
I20251215 21:03:18.459883 29124 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1765832506145157-29124-0/minicluster-data/ts-0/wal/instance
I20251215 21:03:18.460170 29124 external_mini_cluster.cc:1468] Setting key 72a13f5b84ef4fa8633fb3541c91be50
I20251215 21:03:18.460918 7084 ts_manager.cc:194] Registered new tserver with Master: 8fee4ba63d0a484caef91db37d8732e8 (127.28.113.1:37091)
I20251215 21:03:18.462178 29124 external_mini_cluster.cc:1366] Running /tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu
/tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1765832506145157-29124-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1765832506145157-29124-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1765832506145157-29124-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1765832506145157-29124-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.113.2:0
--local_ip_for_outbound_sockets=127.28.113.2
--webserver_interface=127.28.113.2
--webserver_port=0
--tserver_master_addrs=127.28.113.62:44425
--builtin_ntp_servers=127.28.113.20:34501
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20251215 21:03:18.568114 7275 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251215 21:03:18.568357 7275 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251215 21:03:18.568423 7275 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251215 21:03:18.571784 7275 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251215 21:03:18.571871 7275 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251215 21:03:18.572047 7275 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.113.2
I20251215 21:03:18.575889 7275 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.113.20:34501
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1765832506145157-29124-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1765832506145157-29124-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.113.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1765832506145157-29124-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.113.2
--webserver_port=0
--tserver_master_addrs=127.28.113.62:44425
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7275
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.113.2
--log_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1765832506145157-29124-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision dc741732bb6a40213e11a13869fad06a49a209a3
build type DEBUG
built by None at 15 Dec 2025 20:43:14 UTC on 24a791456cd2
build id 9530
I20251215 21:03:18.577033 7275 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251215 21:03:18.578151 7275 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251215 21:03:18.584872 7281 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251215 21:03:18.584884 7280 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251215 21:03:18.585016 7283 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251215 21:03:18.585505 7275 server_base.cc:1047] running on GCE node
I20251215 21:03:18.585883 7275 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251215 21:03:18.586440 7275 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251215 21:03:18.587595 7275 hybrid_clock.cc:648] HybridClock initialized: now 1765832598587580 us; error 29 us; skew 500 ppm
I20251215 21:03:18.589339 7275 webserver.cc:492] Webserver started at http://127.28.113.2:33339/ using document root <none> and password file <none>
I20251215 21:03:18.589856 7275 fs_manager.cc:362] Metadata directory not provided
I20251215 21:03:18.589907 7275 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251215 21:03:18.590080 7275 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251215 21:03:18.591704 7275 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1765832506145157-29124-0/minicluster-data/ts-1/data/instance:
uuid: "49b7f3a9ddc141febebb5447dd5753d0"
format_stamp: "Formatted at 2025-12-15 21:03:18 on dist-test-slave-bjsf"
server_key: "326a173079454611747eb42c127d11ef"
server_key_iv: "c01547c7eb1c696eac7e771ff3e091b1"
server_key_version: "encryptionkey@0"
I20251215 21:03:18.592168 7275 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1765832506145157-29124-0/minicluster-data/ts-1/wal/instance:
uuid: "49b7f3a9ddc141febebb5447dd5753d0"
format_stamp: "Formatted at 2025-12-15 21:03:18 on dist-test-slave-bjsf"
server_key: "326a173079454611747eb42c127d11ef"
server_key_iv: "c01547c7eb1c696eac7e771ff3e091b1"
server_key_version: "encryptionkey@0"
I20251215 21:03:18.595311 7275 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20251215 21:03:18.597579 7289 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251215 21:03:18.598767 7275 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20251215 21:03:18.598870 7275 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1765832506145157-29124-0/minicluster-data/ts-1/data,/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1765832506145157-29124-0/minicluster-data/ts-1/wal
uuid: "49b7f3a9ddc141febebb5447dd5753d0"
format_stamp: "Formatted at 2025-12-15 21:03:18 on dist-test-slave-bjsf"
server_key: "326a173079454611747eb42c127d11ef"
server_key_iv: "c01547c7eb1c696eac7e771ff3e091b1"
server_key_version: "encryptionkey@0"
I20251215 21:03:18.598968 7275 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1765832506145157-29124-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1765832506145157-29124-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1765832506145157-29124-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251215 21:03:18.605085 7275 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251215 21:03:18.605657 7275 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251215 21:03:18.605774 7275 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251215 21:03:18.606271 7275 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251215 21:03:18.607317 7275 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251215 21:03:18.607364 7275 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251215 21:03:18.607411 7275 ts_tablet_manager.cc:616] Registered 0 tablets
I20251215 21:03:18.607430 7275 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251215 21:03:18.617970 7275 rpc_server.cc:307] RPC server started. Bound to: 127.28.113.2:45913
I20251215 21:03:18.617990 7402 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.28.113.2:45913 every 8 connection(s)
I20251215 21:03:18.619145 7275 server_base.cc:1179] Dumped server information to /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1765832506145157-29124-0/minicluster-data/ts-1/data/info.pb
I20251215 21:03:18.624451 7123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:03:18.621024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.113.2:34643 (local address 127.28.113.62:44425)
1215 21:03:18.621175 (+ 151us) server_negotiation.cc:207] Beginning negotiation
1215 21:03:18.621180 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1215 21:03:18.622107 (+ 927us) server_negotiation.cc:408] Connection header received
1215 21:03:18.622841 (+ 734us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1215 21:03:18.622844 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1215 21:03:18.622884 (+ 40us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1215 21:03:18.622955 (+ 71us) server_negotiation.cc:227] Negotiated authn=SASL
1215 21:03:18.624104 (+ 1149us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1215 21:03:18.624107 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1215 21:03:18.624109 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1215 21:03:18.624118 (+ 9us) server_negotiation.cc:923] calling sasl_server_start()
1215 21:03:18.624134 (+ 16us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1215 21:03:18.624141 (+ 7us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1215 21:03:18.624207 (+ 66us) server_negotiation.cc:1036] Waiting for connection context
1215 21:03:18.624310 (+ 103us) server_negotiation.cc:300] Negotiation successful
1215 21:03:18.624342 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20251215 21:03:18.625171 7405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:03:18.621339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.113.62:44425 (local address 127.28.113.2:34643)
1215 21:03:18.621968 (+ 629us) negotiation.cc:107] Waiting for socket to connect
1215 21:03:18.622009 (+ 41us) client_negotiation.cc:174] Beginning negotiation
1215 21:03:18.622669 (+ 660us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1215 21:03:18.623154 (+ 485us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1215 21:03:18.623163 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
1215 21:03:18.623252 (+ 89us) client_negotiation.cc:189] Negotiated authn=SASL
1215 21:03:18.623835 (+ 583us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
1215 21:03:18.623836 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
1215 21:03:18.623867 (+ 31us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
1215 21:03:18.623945 (+ 78us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1215 21:03:18.624199 (+ 254us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1215 21:03:18.624204 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1215 21:03:18.624207 (+ 3us) client_negotiation.cc:769] Sending connection context
1215 21:03:18.624364 (+ 157us) client_negotiation.cc:240] Negotiation successful
1215 21:03:18.624537 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":96,"threads_started":1}
I20251215 21:03:18.626145 7403 heartbeater.cc:344] Connected to a master server at 127.28.113.62:44425
I20251215 21:03:18.626360 7403 heartbeater.cc:461] Registering TS with master...
I20251215 21:03:18.626747 7403 heartbeater.cc:507] Master 127.28.113.62:44425 requested a full tablet report, sending...
I20251215 21:03:18.627743 7084 ts_manager.cc:194] Registered new tserver with Master: 49b7f3a9ddc141febebb5447dd5753d0 (127.28.113.2:45913)
I20251215 21:03:18.629266 29124 external_mini_cluster.cc:1428] Started /tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu as pid 7275
I20251215 21:03:18.629382 29124 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1765832506145157-29124-0/minicluster-data/ts-1/wal/instance
I20251215 21:03:18.629644 29124 external_mini_cluster.cc:1468] Setting key 18403d1a536f6c3b5e549e0638573bc5
I20251215 21:03:18.631325 29124 external_mini_cluster.cc:1366] Running /tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu
/tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1765832506145157-29124-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1765832506145157-29124-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1765832506145157-29124-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1765832506145157-29124-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.113.3:0
--local_ip_for_outbound_sockets=127.28.113.3
--webserver_interface=127.28.113.3
--webserver_port=0
--tserver_master_addrs=127.28.113.62:44425
--builtin_ntp_servers=127.28.113.20:34501
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20251215 21:03:18.733104 7406 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251215 21:03:18.733348 7406 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251215 21:03:18.733421 7406 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251215 21:03:18.736696 7406 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251215 21:03:18.736758 7406 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251215 21:03:18.736864 7406 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.113.3
I20251215 21:03:18.740694 7406 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.113.20:34501
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1765832506145157-29124-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1765832506145157-29124-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.113.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1765832506145157-29124-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.113.3
--webserver_port=0
--tserver_master_addrs=127.28.113.62:44425
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7406
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.113.3
--log_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1765832506145157-29124-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision dc741732bb6a40213e11a13869fad06a49a209a3
build type DEBUG
built by None at 15 Dec 2025 20:43:14 UTC on 24a791456cd2
build id 9530
I20251215 21:03:18.741851 7406 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251215 21:03:18.742969 7406 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251215 21:03:18.749182 7414 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251215 21:03:18.749190 7412 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251215 21:03:18.749317 7406 server_base.cc:1047] running on GCE node
W20251215 21:03:18.749178 7411 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251215 21:03:18.749995 7406 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251215 21:03:18.750566 7406 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251215 21:03:18.751710 7406 hybrid_clock.cc:648] HybridClock initialized: now 1765832598751685 us; error 43 us; skew 500 ppm
I20251215 21:03:18.753525 7406 webserver.cc:492] Webserver started at http://127.28.113.3:39721/ using document root <none> and password file <none>
I20251215 21:03:18.754071 7406 fs_manager.cc:362] Metadata directory not provided
I20251215 21:03:18.754124 7406 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251215 21:03:18.754299 7406 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251215 21:03:18.755942 7406 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1765832506145157-29124-0/minicluster-data/ts-2/data/instance:
uuid: "465e35b988e24a6ea04838c38432dafd"
format_stamp: "Formatted at 2025-12-15 21:03:18 on dist-test-slave-bjsf"
server_key: "12e36996a599147c4dd1f842a17528b7"
server_key_iv: "e8479f9ae50bafa011669e57a7e2ca33"
server_key_version: "encryptionkey@0"
I20251215 21:03:18.756444 7406 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1765832506145157-29124-0/minicluster-data/ts-2/wal/instance:
uuid: "465e35b988e24a6ea04838c38432dafd"
format_stamp: "Formatted at 2025-12-15 21:03:18 on dist-test-slave-bjsf"
server_key: "12e36996a599147c4dd1f842a17528b7"
server_key_iv: "e8479f9ae50bafa011669e57a7e2ca33"
server_key_version: "encryptionkey@0"
I20251215 21:03:18.759594 7406 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.001s sys 0.001s
I20251215 21:03:18.761943 7420 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251215 21:03:18.762939 7406 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20251215 21:03:18.763046 7406 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1765832506145157-29124-0/minicluster-data/ts-2/data,/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1765832506145157-29124-0/minicluster-data/ts-2/wal
uuid: "465e35b988e24a6ea04838c38432dafd"
format_stamp: "Formatted at 2025-12-15 21:03:18 on dist-test-slave-bjsf"
server_key: "12e36996a599147c4dd1f842a17528b7"
server_key_iv: "e8479f9ae50bafa011669e57a7e2ca33"
server_key_version: "encryptionkey@0"
I20251215 21:03:18.763132 7406 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1765832506145157-29124-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1765832506145157-29124-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1765832506145157-29124-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251215 21:03:18.767685 7406 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251215 21:03:18.768251 7406 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251215 21:03:18.768399 7406 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251215 21:03:18.768875 7406 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251215 21:03:18.769784 7406 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251215 21:03:18.769830 7406 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251215 21:03:18.769878 7406 ts_tablet_manager.cc:616] Registered 0 tablets
I20251215 21:03:18.769907 7406 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251215 21:03:18.778147 7406 rpc_server.cc:307] RPC server started. Bound to: 127.28.113.3:37893
I20251215 21:03:18.778190 7533 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.28.113.3:37893 every 8 connection(s)
I20251215 21:03:18.779035 7406 server_base.cc:1179] Dumped server information to /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1765832506145157-29124-0/minicluster-data/ts-2/data/info.pb
I20251215 21:03:18.783897 7123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:03:18.780673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.113.3:46037 (local address 127.28.113.62:44425)
1215 21:03:18.780799 (+ 126us) server_negotiation.cc:207] Beginning negotiation
1215 21:03:18.780803 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1215 21:03:18.781648 (+ 845us) server_negotiation.cc:408] Connection header received
1215 21:03:18.782413 (+ 765us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1215 21:03:18.782416 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1215 21:03:18.782459 (+ 43us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1215 21:03:18.782542 (+ 83us) server_negotiation.cc:227] Negotiated authn=SASL
1215 21:03:18.783475 (+ 933us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1215 21:03:18.783478 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1215 21:03:18.783480 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1215 21:03:18.783491 (+ 11us) server_negotiation.cc:923] calling sasl_server_start()
1215 21:03:18.783511 (+ 20us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1215 21:03:18.783519 (+ 8us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1215 21:03:18.783597 (+ 78us) server_negotiation.cc:1036] Waiting for connection context
1215 21:03:18.783743 (+ 146us) server_negotiation.cc:300] Negotiation successful
1215 21:03:18.783777 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20251215 21:03:18.784731 7536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:03:18.780937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.113.62:44425 (local address 127.28.113.3:46037)
1215 21:03:18.781526 (+ 589us) negotiation.cc:107] Waiting for socket to connect
1215 21:03:18.781558 (+ 32us) client_negotiation.cc:174] Beginning negotiation
1215 21:03:18.782206 (+ 648us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1215 21:03:18.782677 (+ 471us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1215 21:03:18.782684 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
1215 21:03:18.782776 (+ 92us) client_negotiation.cc:189] Negotiated authn=SASL
1215 21:03:18.783279 (+ 503us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
1215 21:03:18.783280 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
1215 21:03:18.783308 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
1215 21:03:18.783358 (+ 50us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1215 21:03:18.783591 (+ 233us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1215 21:03:18.783596 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1215 21:03:18.783599 (+ 3us) client_negotiation.cc:769] Sending connection context
1215 21:03:18.783813 (+ 214us) client_negotiation.cc:240] Negotiation successful
1215 21:03:18.784056 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":96,"threads_started":1}
I20251215 21:03:18.785662 7534 heartbeater.cc:344] Connected to a master server at 127.28.113.62:44425
I20251215 21:03:18.785900 7534 heartbeater.cc:461] Registering TS with master...
I20251215 21:03:18.786145 29124 external_mini_cluster.cc:1428] Started /tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu as pid 7406
I20251215 21:03:18.786243 29124 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1765832506145157-29124-0/minicluster-data/ts-2/wal/instance
I20251215 21:03:18.786326 7534 heartbeater.cc:507] Master 127.28.113.62:44425 requested a full tablet report, sending...
I20251215 21:03:18.786465 29124 external_mini_cluster.cc:1468] Setting key 38c943bc8fb33e5667fbd2688b5f029d
I20251215 21:03:18.787288 7084 ts_manager.cc:194] Registered new tserver with Master: 465e35b988e24a6ea04838c38432dafd (127.28.113.3:37893)
I20251215 21:03:18.788370 29124 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
W20251215 21:03:18.790729 7123 negotiation.cc:337] Failed RPC negotiation. Trace:
1215 21:03:18.790001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49124 (local address 127.28.113.62:44425)
1215 21:03:18.790124 (+ 123us) server_negotiation.cc:207] Beginning negotiation
1215 21:03:18.790128 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1215 21:03:18.790250 (+ 122us) server_negotiation.cc:408] Connection header received
1215 21:03:18.790364 (+ 114us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1215 21:03:18.790368 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1215 21:03:18.790400 (+ 32us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1215 21:03:18.790461 (+ 61us) server_negotiation.cc:227] Negotiated authn=SASL
1215 21:03:18.790648 (+ 187us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49124: BlockingRecv error: recv got EOF from 127.0.0.1:49124 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
I20251215 21:03:18.791111 29124 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu with pid 7144
I20251215 21:03:18.796438 29124 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu with pid 7275
I20251215 21:03:18.800865 29124 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu with pid 7406
I20251215 21:03:18.805116 29124 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu with pid 7053
2025-12-15T21:03:18Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (752 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765832506145157-29124-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Dec 15 21:03:18 dist-test-slave-bjsf krb5kdc[7544](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Dec 15 21:03:18 dist-test-slave-bjsf krb5kdc[7544](info): set up 2 sockets
Dec 15 21:03:18 dist-test-slave-bjsf krb5kdc[7544](info): commencing operation
krb5kdc: starting...
W20251215 21:03:20.628396 29124 mini_kdc.cc:121] Time spent starting KDC: real 1.804s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765832506145157-29124-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765832506145157-29124-0/krb5kdc/test-user.keytab.
Dec 15 21:03:20 dist-test-slave-bjsf krb5kdc[7544](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765832600, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-12-15T21:03:20Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-12-15T21:03:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.113.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.113.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.113.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765832506145157-29124-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.28.113.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765832506145157-29124-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.28.113.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.28.113.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.28.113.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765832506145157-29124-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.28.113.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765832506145157-29124-0/krb5kdc/kudu.keytab.
I20251215 21:03:20.754890 29124 external_mini_cluster.cc:1366] Running /tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu
/tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765832506145157-29124-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765832506145157-29124-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765832506145157-29124-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765832506145157-29124-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.113.62:35521
--webserver_interface=127.28.113.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765832506145157-29124-0/krb5kdc/kudu.keytab
--principal=kudu/127.28.113.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.28.113.20:32937
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.113.62:35521
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765832506145157-29124-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765832506145157-29124-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765832506145157-29124-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251215 21:03:20.859805 7560 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251215 21:03:20.860100 7560 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251215 21:03:20.860177 7560 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251215 21:03:20.863539 7560 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251215 21:03:20.863617 7560 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251215 21:03:20.863644 7560 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251215 21:03:20.863662 7560 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251215 21:03:20.863680 7560 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251215 21:03:20.868044 7560 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.113.20:32937
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765832506145157-29124-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765832506145157-29124-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.113.62:35521
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765832506145157-29124-0/krb5kdc/kudu.keytab
--principal=kudu/127.28.113.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.113.62:35521
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765832506145157-29124-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.113.62
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7560
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765832506145157-29124-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision dc741732bb6a40213e11a13869fad06a49a209a3
build type DEBUG
built by None at 15 Dec 2025 20:43:14 UTC on 24a791456cd2
build id 9530
I20251215 21:03:20.869097 7560 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251215 21:03:20.869937 7560 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251215 21:03:20.875365 7566 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251215 21:03:20.875416 7565 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251215 21:03:20.875398 7568 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251215 21:03:20.875553 7560 server_base.cc:1047] running on GCE node
I20251215 21:03:20.876152 7560 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251215 21:03:20.877034 7560 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251215 21:03:20.878211 7560 hybrid_clock.cc:648] HybridClock initialized: now 1765832600878186 us; error 39 us; skew 500 ppm
Dec 15 21:03:20 dist-test-slave-bjsf krb5kdc[7544](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765832600, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.113.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251215 21:03:20.881001 7560 init.cc:377] Logged in from keytab as kudu/127.28.113.62@KRBTEST.COM (short username kudu)
I20251215 21:03:20.882093 7560 webserver.cc:492] Webserver started at http://127.28.113.62:46453/ using document root <none> and password file <none>
I20251215 21:03:20.882637 7560 fs_manager.cc:362] Metadata directory not provided
I20251215 21:03:20.882689 7560 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251215 21:03:20.882887 7560 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251215 21:03:20.884632 7560 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765832506145157-29124-0/minicluster-data/master-0/data/instance:
uuid: "ee0bd4d6b3d94ea3b6373a5365e9a832"
format_stamp: "Formatted at 2025-12-15 21:03:20 on dist-test-slave-bjsf"
server_key: "cc3bd7e093a2849ebb01532e1e867917"
server_key_iv: "e9d54858ca4286b2dc0e115e2a0349ee"
server_key_version: "encryptionkey@0"
I20251215 21:03:20.885099 7560 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765832506145157-29124-0/minicluster-data/master-0/wal/instance:
uuid: "ee0bd4d6b3d94ea3b6373a5365e9a832"
format_stamp: "Formatted at 2025-12-15 21:03:20 on dist-test-slave-bjsf"
server_key: "cc3bd7e093a2849ebb01532e1e867917"
server_key_iv: "e9d54858ca4286b2dc0e115e2a0349ee"
server_key_version: "encryptionkey@0"
I20251215 21:03:20.888612 7560 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20251215 21:03:20.890851 7575 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251215 21:03:20.891898 7560 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20251215 21:03:20.892042 7560 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765832506145157-29124-0/minicluster-data/master-0/data,/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765832506145157-29124-0/minicluster-data/master-0/wal
uuid: "ee0bd4d6b3d94ea3b6373a5365e9a832"
format_stamp: "Formatted at 2025-12-15 21:03:20 on dist-test-slave-bjsf"
server_key: "cc3bd7e093a2849ebb01532e1e867917"
server_key_iv: "e9d54858ca4286b2dc0e115e2a0349ee"
server_key_version: "encryptionkey@0"
I20251215 21:03:20.892139 7560 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765832506145157-29124-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765832506145157-29124-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765832506145157-29124-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251215 21:03:20.903852 7560 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251215 21:03:20.907102 7560 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251215 21:03:20.907285 7560 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251215 21:03:20.914881 7560 rpc_server.cc:307] RPC server started. Bound to: 127.28.113.62:35521
I20251215 21:03:20.914896 7627 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.28.113.62:35521 every 8 connection(s)
I20251215 21:03:20.916049 7560 server_base.cc:1179] Dumped server information to /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765832506145157-29124-0/minicluster-data/master-0/data/info.pb
I20251215 21:03:20.918915 7628 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 21:03:20.920039 29124 external_mini_cluster.cc:1428] Started /tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu as pid 7560
I20251215 21:03:20.920181 29124 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765832506145157-29124-0/minicluster-data/master-0/wal/instance
I20251215 21:03:20.920466 29124 external_mini_cluster.cc:1468] Setting key e611fdcab988aeb4912b790434ac533d
I20251215 21:03:20.925412 7628 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ee0bd4d6b3d94ea3b6373a5365e9a832: Bootstrap starting.
I20251215 21:03:20.927881 7628 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ee0bd4d6b3d94ea3b6373a5365e9a832: Neither blocks nor log segments found. Creating new log.
Dec 15 21:03:20 dist-test-slave-bjsf krb5kdc[7544](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765832600, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.113.62@KRBTEST.COM
I20251215 21:03:20.928890 7628 log.cc:826] T 00000000000000000000000000000000 P ee0bd4d6b3d94ea3b6373a5365e9a832: Log is configured to *not* fsync() on all Append() calls
I20251215 21:03:20.931214 7628 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ee0bd4d6b3d94ea3b6373a5365e9a832: No bootstrap required, opened a new log
I20251215 21:03:20.934147 7631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:03:20.921860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33910 (local address 127.28.113.62:35521)
1215 21:03:20.922345 (+ 485us) server_negotiation.cc:207] Beginning negotiation
1215 21:03:20.922355 (+ 10us) server_negotiation.cc:400] Waiting for connection header
1215 21:03:20.922393 (+ 38us) server_negotiation.cc:408] Connection header received
1215 21:03:20.922932 (+ 539us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1215 21:03:20.922966 (+ 34us) server_negotiation.cc:462] Received NEGOTIATE request from client
1215 21:03:20.923278 (+ 312us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1215 21:03:20.923605 (+ 327us) server_negotiation.cc:227] Negotiated authn=SASL
1215 21:03:20.924609 (+ 1004us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:03:20.925768 (+ 1159us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1215 21:03:20.926445 (+ 677us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:03:20.926737 (+ 292us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:03:20.929132 (+ 2395us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1215 21:03:20.929174 (+ 42us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1215 21:03:20.929187 (+ 13us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1215 21:03:20.929227 (+ 40us) server_negotiation.cc:923] calling sasl_server_start()
1215 21:03:20.931326 (+ 2099us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1215 21:03:20.931742 (+ 416us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1215 21:03:20.931748 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1215 21:03:20.931753 (+ 5us) server_negotiation.cc:968] calling sasl_server_step()
1215 21:03:20.931848 (+ 95us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1215 21:03:20.932169 (+ 321us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1215 21:03:20.932177 (+ 8us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1215 21:03:20.932179 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1215 21:03:20.932604 (+ 425us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1215 21:03:20.932802 (+ 198us) server_negotiation.cc:1036] Waiting for connection context
1215 21:03:20.933097 (+ 295us) server_negotiation.cc:300] Negotiation successful
1215 21:03:20.933376 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":319,"thread_start_us":135,"threads_started":1}
I20251215 21:03:20.934523 7628 raft_consensus.cc:359] T 00000000000000000000000000000000 P ee0bd4d6b3d94ea3b6373a5365e9a832 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee0bd4d6b3d94ea3b6373a5365e9a832" member_type: VOTER last_known_addr { host: "127.28.113.62" port: 35521 } }
I20251215 21:03:20.934737 7628 raft_consensus.cc:385] T 00000000000000000000000000000000 P ee0bd4d6b3d94ea3b6373a5365e9a832 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251215 21:03:20.934791 7628 raft_consensus.cc:740] T 00000000000000000000000000000000 P ee0bd4d6b3d94ea3b6373a5365e9a832 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee0bd4d6b3d94ea3b6373a5365e9a832, State: Initialized, Role: FOLLOWER
I20251215 21:03:20.935356 7628 consensus_queue.cc:260] T 00000000000000000000000000000000 P ee0bd4d6b3d94ea3b6373a5365e9a832 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee0bd4d6b3d94ea3b6373a5365e9a832" member_type: VOTER last_known_addr { host: "127.28.113.62" port: 35521 } }
I20251215 21:03:20.935494 7628 raft_consensus.cc:399] T 00000000000000000000000000000000 P ee0bd4d6b3d94ea3b6373a5365e9a832 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251215 21:03:20.935554 7628 raft_consensus.cc:493] T 00000000000000000000000000000000 P ee0bd4d6b3d94ea3b6373a5365e9a832 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251215 21:03:20.935649 7628 raft_consensus.cc:3060] T 00000000000000000000000000000000 P ee0bd4d6b3d94ea3b6373a5365e9a832 [term 0 FOLLOWER]: Advancing to term 1
I20251215 21:03:20.936703 7628 raft_consensus.cc:515] T 00000000000000000000000000000000 P ee0bd4d6b3d94ea3b6373a5365e9a832 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee0bd4d6b3d94ea3b6373a5365e9a832" member_type: VOTER last_known_addr { host: "127.28.113.62" port: 35521 } }
I20251215 21:03:20.937033 7628 leader_election.cc:304] T 00000000000000000000000000000000 P ee0bd4d6b3d94ea3b6373a5365e9a832 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ee0bd4d6b3d94ea3b6373a5365e9a832; no voters:
I20251215 21:03:20.937310 7628 leader_election.cc:290] T 00000000000000000000000000000000 P ee0bd4d6b3d94ea3b6373a5365e9a832 [CANDIDATE]: Term 1 election: Requested vote from peers
I20251215 21:03:20.937652 7634 raft_consensus.cc:2804] T 00000000000000000000000000000000 P ee0bd4d6b3d94ea3b6373a5365e9a832 [term 1 FOLLOWER]: Leader election won for term 1
I20251215 21:03:20.937885 7634 raft_consensus.cc:697] T 00000000000000000000000000000000 P ee0bd4d6b3d94ea3b6373a5365e9a832 [term 1 LEADER]: Becoming Leader. State: Replica: ee0bd4d6b3d94ea3b6373a5365e9a832, State: Running, Role: LEADER
I20251215 21:03:20.938212 7634 consensus_queue.cc:237] T 00000000000000000000000000000000 P ee0bd4d6b3d94ea3b6373a5365e9a832 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee0bd4d6b3d94ea3b6373a5365e9a832" member_type: VOTER last_known_addr { host: "127.28.113.62" port: 35521 } }
I20251215 21:03:20.938503 7628 sys_catalog.cc:565] T 00000000000000000000000000000000 P ee0bd4d6b3d94ea3b6373a5365e9a832 [sys.catalog]: configured and running, proceeding with master startup.
I20251215 21:03:20.939921 7633 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee0bd4d6b3d94ea3b6373a5365e9a832 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ee0bd4d6b3d94ea3b6373a5365e9a832" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee0bd4d6b3d94ea3b6373a5365e9a832" member_type: VOTER last_known_addr { host: "127.28.113.62" port: 35521 } } }
I20251215 21:03:20.939947 7635 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee0bd4d6b3d94ea3b6373a5365e9a832 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ee0bd4d6b3d94ea3b6373a5365e9a832. Latest consensus state: current_term: 1 leader_uuid: "ee0bd4d6b3d94ea3b6373a5365e9a832" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee0bd4d6b3d94ea3b6373a5365e9a832" member_type: VOTER last_known_addr { host: "127.28.113.62" port: 35521 } } }
I20251215 21:03:20.940052 7633 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee0bd4d6b3d94ea3b6373a5365e9a832 [sys.catalog]: This master's current role is: LEADER
I20251215 21:03:20.940085 7635 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee0bd4d6b3d94ea3b6373a5365e9a832 [sys.catalog]: This master's current role is: LEADER
I20251215 21:03:20.940424 7643 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251215 21:03:20.943038 7643 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251215 21:03:20.948823 7643 catalog_manager.cc:1357] Generated new cluster ID: a474bd8c2695475baf120fb96373e90b
I20251215 21:03:20.948908 7643 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251215 21:03:20.961913 7643 catalog_manager.cc:1380] Generated new certificate authority record
I20251215 21:03:20.963233 7643 catalog_manager.cc:1514] Loading token signing keys...
I20251215 21:03:20.978868 7643 catalog_manager.cc:6027] T 00000000000000000000000000000000 P ee0bd4d6b3d94ea3b6373a5365e9a832: Generated new TSK 0
I20251215 21:03:20.979794 7643 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.113.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.113.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.113.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765832506145157-29124-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.28.113.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765832506145157-29124-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.28.113.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.28.113.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.28.113.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765832506145157-29124-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.28.113.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765832506145157-29124-0/krb5kdc/kudu.keytab.
I20251215 21:03:21.024799 29124 external_mini_cluster.cc:1366] Running /tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu
/tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765832506145157-29124-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765832506145157-29124-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765832506145157-29124-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765832506145157-29124-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.113.1:0
--local_ip_for_outbound_sockets=127.28.113.1
--webserver_interface=127.28.113.1
--webserver_port=0
--tserver_master_addrs=127.28.113.62:35521
--keytab_file=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765832506145157-29124-0/krb5kdc/kudu.keytab
--principal=kudu/127.28.113.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.28.113.20:32937
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765832506145157-29124-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765832506145157-29124-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765832506145157-29124-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251215 21:03:21.130208 7656 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251215 21:03:21.130437 7656 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251215 21:03:21.130493 7656 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251215 21:03:21.133730 7656 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251215 21:03:21.133800 7656 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251215 21:03:21.133881 7656 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.113.1
I20251215 21:03:21.138051 7656 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.113.20:32937
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765832506145157-29124-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765832506145157-29124-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765832506145157-29124-0/krb5kdc/kudu.keytab
--principal=kudu/127.28.113.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.113.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765832506145157-29124-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.113.1
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.28.113.62:35521
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7656
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.113.1
--log_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765832506145157-29124-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision dc741732bb6a40213e11a13869fad06a49a209a3
build type DEBUG
built by None at 15 Dec 2025 20:43:14 UTC on 24a791456cd2
build id 9530
I20251215 21:03:21.139052 7656 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251215 21:03:21.139977 7656 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251215 21:03:21.146169 7662 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251215 21:03:21.146186 7664 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251215 21:03:21.146392 7656 server_base.cc:1047] running on GCE node
W20251215 21:03:21.146425 7661 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251215 21:03:21.146845 7656 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251215 21:03:21.147364 7656 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251215 21:03:21.148494 7656 hybrid_clock.cc:648] HybridClock initialized: now 1765832601148476 us; error 34 us; skew 500 ppm
Dec 15 21:03:21 dist-test-slave-bjsf krb5kdc[7544](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765832601, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.113.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251215 21:03:21.150914 7656 init.cc:377] Logged in from keytab as kudu/127.28.113.1@KRBTEST.COM (short username kudu)
I20251215 21:03:21.151893 7656 webserver.cc:492] Webserver started at http://127.28.113.1:44277/ using document root <none> and password file <none>
I20251215 21:03:21.152442 7656 fs_manager.cc:362] Metadata directory not provided
I20251215 21:03:21.152491 7656 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251215 21:03:21.152666 7656 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251215 21:03:21.154280 7656 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765832506145157-29124-0/minicluster-data/ts-0/data/instance:
uuid: "e6b9743daabe409084b61f874bd0cb25"
format_stamp: "Formatted at 2025-12-15 21:03:21 on dist-test-slave-bjsf"
server_key: "f77725c58e91c36974cb3d3abf9b10f7"
server_key_iv: "0a3120dfbb7ad9e8bb30c0a92ebe6831"
server_key_version: "encryptionkey@0"
I20251215 21:03:21.154706 7656 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765832506145157-29124-0/minicluster-data/ts-0/wal/instance:
uuid: "e6b9743daabe409084b61f874bd0cb25"
format_stamp: "Formatted at 2025-12-15 21:03:21 on dist-test-slave-bjsf"
server_key: "f77725c58e91c36974cb3d3abf9b10f7"
server_key_iv: "0a3120dfbb7ad9e8bb30c0a92ebe6831"
server_key_version: "encryptionkey@0"
I20251215 21:03:21.157976 7656 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20251215 21:03:21.160219 7671 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251215 21:03:21.161212 7656 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20251215 21:03:21.161307 7656 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765832506145157-29124-0/minicluster-data/ts-0/data,/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765832506145157-29124-0/minicluster-data/ts-0/wal
uuid: "e6b9743daabe409084b61f874bd0cb25"
format_stamp: "Formatted at 2025-12-15 21:03:21 on dist-test-slave-bjsf"
server_key: "f77725c58e91c36974cb3d3abf9b10f7"
server_key_iv: "0a3120dfbb7ad9e8bb30c0a92ebe6831"
server_key_version: "encryptionkey@0"
I20251215 21:03:21.161399 7656 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765832506145157-29124-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765832506145157-29124-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765832506145157-29124-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251215 21:03:21.197067 7656 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251215 21:03:21.202575 7656 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251215 21:03:21.202749 7656 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251215 21:03:21.203316 7656 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251215 21:03:21.204212 7656 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251215 21:03:21.204258 7656 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251215 21:03:21.204298 7656 ts_tablet_manager.cc:616] Registered 0 tablets
I20251215 21:03:21.204326 7656 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251215 21:03:21.212926 7656 rpc_server.cc:307] RPC server started. Bound to: 127.28.113.1:41735
I20251215 21:03:21.212954 7784 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.28.113.1:41735 every 8 connection(s)
I20251215 21:03:21.213831 7656 server_base.cc:1179] Dumped server information to /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765832506145157-29124-0/minicluster-data/ts-0/data/info.pb
I20251215 21:03:21.220544 29124 external_mini_cluster.cc:1428] Started /tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu as pid 7656
I20251215 21:03:21.220662 29124 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765832506145157-29124-0/minicluster-data/ts-0/wal/instance
I20251215 21:03:21.220947 29124 external_mini_cluster.cc:1468] Setting key dd5d0fefa4bbe9435ee1171095b13add
Dec 15 21:03:21 dist-test-slave-bjsf krb5kdc[7544](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765832601, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.113.1@KRBTEST.COM for kudu/127.28.113.62@KRBTEST.COM
I20251215 21:03:21.226207 7631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:03:21.215602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.113.1:34663 (local address 127.28.113.62:35521)
1215 21:03:21.215744 (+ 142us) server_negotiation.cc:207] Beginning negotiation
1215 21:03:21.215748 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1215 21:03:21.216448 (+ 700us) server_negotiation.cc:408] Connection header received
1215 21:03:21.217232 (+ 784us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1215 21:03:21.217234 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1215 21:03:21.217289 (+ 55us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1215 21:03:21.217360 (+ 71us) server_negotiation.cc:227] Negotiated authn=SASL
1215 21:03:21.218941 (+ 1581us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:03:21.219441 (+ 500us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1215 21:03:21.220040 (+ 599us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:03:21.220170 (+ 130us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:03:21.222615 (+ 2445us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1215 21:03:21.222633 (+ 18us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1215 21:03:21.222635 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1215 21:03:21.222660 (+ 25us) server_negotiation.cc:923] calling sasl_server_start()
1215 21:03:21.224287 (+ 1627us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1215 21:03:21.224762 (+ 475us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1215 21:03:21.224766 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1215 21:03:21.224767 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1215 21:03:21.224820 (+ 53us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1215 21:03:21.225089 (+ 269us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1215 21:03:21.225092 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1215 21:03:21.225094 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1215 21:03:21.225282 (+ 188us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1215 21:03:21.225377 (+ 95us) server_negotiation.cc:1036] Waiting for connection context
1215 21:03:21.225885 (+ 508us) server_negotiation.cc:300] Negotiation successful
1215 21:03:21.226039 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20251215 21:03:21.226763 7787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:03:21.215856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.113.62:35521 (local address 127.28.113.1:34663)
1215 21:03:21.216313 (+ 457us) negotiation.cc:107] Waiting for socket to connect
1215 21:03:21.216349 (+ 36us) client_negotiation.cc:174] Beginning negotiation
1215 21:03:21.217048 (+ 699us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1215 21:03:21.217504 (+ 456us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1215 21:03:21.217511 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
1215 21:03:21.217883 (+ 372us) client_negotiation.cc:189] Negotiated authn=SASL
1215 21:03:21.218816 (+ 933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:03:21.218830 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:03:21.219526 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1215 21:03:21.219528 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1215 21:03:21.219907 (+ 379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:03:21.219913 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:03:21.220136 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:03:21.220741 (+ 605us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1215 21:03:21.220775 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
1215 21:03:21.222467 (+ 1692us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1215 21:03:21.224409 (+ 1942us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1215 21:03:21.224415 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1215 21:03:21.224426 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
1215 21:03:21.224661 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1215 21:03:21.224905 (+ 244us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1215 21:03:21.224907 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1215 21:03:21.224908 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1215 21:03:21.225017 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1215 21:03:21.225369 (+ 352us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1215 21:03:21.225374 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1215 21:03:21.225643 (+ 269us) client_negotiation.cc:769] Sending connection context
1215 21:03:21.225823 (+ 180us) client_negotiation.cc:240] Negotiation successful
1215 21:03:21.226058 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":132,"threads_started":1}
I20251215 21:03:21.227943 7785 heartbeater.cc:344] Connected to a master server at 127.28.113.62:35521
I20251215 21:03:21.228250 7785 heartbeater.cc:461] Registering TS with master...
I20251215 21:03:21.228890 7785 heartbeater.cc:507] Master 127.28.113.62:35521 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.28.113.2@KRBTEST.COM; defaulting to no policy
I20251215 21:03:21.230399 7592 ts_manager.cc:194] Registered new tserver with Master: e6b9743daabe409084b61f874bd0cb25 (127.28.113.1:41735)
I20251215 21:03:21.231598 7592 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.113.1@KRBTEST.COM'} at 127.28.113.1:34663
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.113.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.113.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765832506145157-29124-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.28.113.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765832506145157-29124-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.28.113.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.28.113.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.28.113.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765832506145157-29124-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.28.113.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765832506145157-29124-0/krb5kdc/kudu.keytab.
I20251215 21:03:21.266227 29124 external_mini_cluster.cc:1366] Running /tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu
/tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765832506145157-29124-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765832506145157-29124-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765832506145157-29124-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765832506145157-29124-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.113.2:0
--local_ip_for_outbound_sockets=127.28.113.2
--webserver_interface=127.28.113.2
--webserver_port=0
--tserver_master_addrs=127.28.113.62:35521
--keytab_file=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765832506145157-29124-0/krb5kdc/kudu.keytab
--principal=kudu/127.28.113.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.28.113.20:32937
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765832506145157-29124-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765832506145157-29124-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765832506145157-29124-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251215 21:03:21.371886 7792 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251215 21:03:21.372184 7792 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251215 21:03:21.372265 7792 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251215 21:03:21.375669 7792 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251215 21:03:21.375747 7792 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251215 21:03:21.375865 7792 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.113.2
I20251215 21:03:21.380262 7792 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.113.20:32937
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765832506145157-29124-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765832506145157-29124-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765832506145157-29124-0/krb5kdc/kudu.keytab
--principal=kudu/127.28.113.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.113.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765832506145157-29124-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.113.2
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.28.113.62:35521
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7792
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.113.2
--log_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765832506145157-29124-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision dc741732bb6a40213e11a13869fad06a49a209a3
build type DEBUG
built by None at 15 Dec 2025 20:43:14 UTC on 24a791456cd2
build id 9530
I20251215 21:03:21.381335 7792 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251215 21:03:21.382148 7792 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251215 21:03:21.388569 7798 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251215 21:03:21.388608 7797 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251215 21:03:21.388738 7792 server_base.cc:1047] running on GCE node
W20251215 21:03:21.388741 7800 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251215 21:03:21.389216 7792 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251215 21:03:21.389778 7792 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251215 21:03:21.390930 7792 hybrid_clock.cc:648] HybridClock initialized: now 1765832601390917 us; error 27 us; skew 500 ppm
Dec 15 21:03:21 dist-test-slave-bjsf krb5kdc[7544](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765832601, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.113.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251215 21:03:21.393462 7792 init.cc:377] Logged in from keytab as kudu/127.28.113.2@KRBTEST.COM (short username kudu)
I20251215 21:03:21.394475 7792 webserver.cc:492] Webserver started at http://127.28.113.2:44893/ using document root <none> and password file <none>
I20251215 21:03:21.395006 7792 fs_manager.cc:362] Metadata directory not provided
I20251215 21:03:21.395058 7792 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251215 21:03:21.395241 7792 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251215 21:03:21.396996 7792 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765832506145157-29124-0/minicluster-data/ts-1/data/instance:
uuid: "3432246dc9284624876b603d75b91697"
format_stamp: "Formatted at 2025-12-15 21:03:21 on dist-test-slave-bjsf"
server_key: "2f5930b0c51a6986a471953f4c631f29"
server_key_iv: "8432af6e3eb401c79c69df9d36e51648"
server_key_version: "encryptionkey@0"
I20251215 21:03:21.397447 7792 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765832506145157-29124-0/minicluster-data/ts-1/wal/instance:
uuid: "3432246dc9284624876b603d75b91697"
format_stamp: "Formatted at 2025-12-15 21:03:21 on dist-test-slave-bjsf"
server_key: "2f5930b0c51a6986a471953f4c631f29"
server_key_iv: "8432af6e3eb401c79c69df9d36e51648"
server_key_version: "encryptionkey@0"
I20251215 21:03:21.400674 7792 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20251215 21:03:21.402779 7807 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251215 21:03:21.403753 7792 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20251215 21:03:21.403859 7792 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765832506145157-29124-0/minicluster-data/ts-1/data,/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765832506145157-29124-0/minicluster-data/ts-1/wal
uuid: "3432246dc9284624876b603d75b91697"
format_stamp: "Formatted at 2025-12-15 21:03:21 on dist-test-slave-bjsf"
server_key: "2f5930b0c51a6986a471953f4c631f29"
server_key_iv: "8432af6e3eb401c79c69df9d36e51648"
server_key_version: "encryptionkey@0"
I20251215 21:03:21.403944 7792 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765832506145157-29124-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765832506145157-29124-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765832506145157-29124-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251215 21:03:21.419368 7792 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251215 21:03:21.422255 7792 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251215 21:03:21.422430 7792 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251215 21:03:21.422966 7792 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251215 21:03:21.423825 7792 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251215 21:03:21.423871 7792 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251215 21:03:21.423923 7792 ts_tablet_manager.cc:616] Registered 0 tablets
I20251215 21:03:21.423947 7792 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251215 21:03:21.432467 7792 rpc_server.cc:307] RPC server started. Bound to: 127.28.113.2:40065
I20251215 21:03:21.432493 7920 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.28.113.2:40065 every 8 connection(s)
I20251215 21:03:21.433446 7792 server_base.cc:1179] Dumped server information to /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765832506145157-29124-0/minicluster-data/ts-1/data/info.pb
I20251215 21:03:21.441215 29124 external_mini_cluster.cc:1428] Started /tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu as pid 7792
I20251215 21:03:21.441334 29124 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765832506145157-29124-0/minicluster-data/ts-1/wal/instance
Dec 15 21:03:21 dist-test-slave-bjsf krb5kdc[7544](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765832601, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.113.2@KRBTEST.COM for kudu/127.28.113.62@KRBTEST.COM
I20251215 21:03:21.441568 29124 external_mini_cluster.cc:1468] Setting key 05731a9aef3043ac8e5bbf1566493503
I20251215 21:03:21.445571 7631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:03:21.435125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.113.2:40387 (local address 127.28.113.62:35521)
1215 21:03:21.435270 (+ 145us) server_negotiation.cc:207] Beginning negotiation
1215 21:03:21.435275 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1215 21:03:21.435989 (+ 714us) server_negotiation.cc:408] Connection header received
1215 21:03:21.436792 (+ 803us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1215 21:03:21.436795 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1215 21:03:21.436845 (+ 50us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1215 21:03:21.436919 (+ 74us) server_negotiation.cc:227] Negotiated authn=SASL
1215 21:03:21.438498 (+ 1579us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:03:21.438999 (+ 501us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1215 21:03:21.439677 (+ 678us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:03:21.439812 (+ 135us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:03:21.441931 (+ 2119us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1215 21:03:21.441952 (+ 21us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1215 21:03:21.441954 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1215 21:03:21.441981 (+ 27us) server_negotiation.cc:923] calling sasl_server_start()
1215 21:03:21.443599 (+ 1618us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1215 21:03:21.444099 (+ 500us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1215 21:03:21.444102 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1215 21:03:21.444104 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1215 21:03:21.444153 (+ 49us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1215 21:03:21.444400 (+ 247us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1215 21:03:21.444402 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1215 21:03:21.444404 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1215 21:03:21.444586 (+ 182us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1215 21:03:21.444672 (+ 86us) server_negotiation.cc:1036] Waiting for connection context
1215 21:03:21.445293 (+ 621us) server_negotiation.cc:300] Negotiation successful
1215 21:03:21.445418 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20251215 21:03:21.446201 7923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:03:21.435424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.113.62:35521 (local address 127.28.113.2:40387)
1215 21:03:21.435824 (+ 400us) negotiation.cc:107] Waiting for socket to connect
1215 21:03:21.435859 (+ 35us) client_negotiation.cc:174] Beginning negotiation
1215 21:03:21.436604 (+ 745us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1215 21:03:21.437059 (+ 455us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1215 21:03:21.437066 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
1215 21:03:21.437460 (+ 394us) client_negotiation.cc:189] Negotiated authn=SASL
1215 21:03:21.438376 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:03:21.438391 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:03:21.439087 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1215 21:03:21.439090 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1215 21:03:21.439571 (+ 481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:03:21.439581 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:03:21.439772 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:03:21.440367 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1215 21:03:21.440390 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
1215 21:03:21.441789 (+ 1399us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1215 21:03:21.443707 (+ 1918us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1215 21:03:21.443712 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1215 21:03:21.443723 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
1215 21:03:21.444002 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1215 21:03:21.444223 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1215 21:03:21.444225 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1215 21:03:21.444227 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1215 21:03:21.444325 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1215 21:03:21.444685 (+ 360us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1215 21:03:21.444692 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1215 21:03:21.445004 (+ 312us) client_negotiation.cc:769] Sending connection context
1215 21:03:21.445249 (+ 245us) client_negotiation.cc:240] Negotiation successful
1215 21:03:21.445485 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":95,"threads_started":1}
I20251215 21:03:21.447252 7921 heartbeater.cc:344] Connected to a master server at 127.28.113.62:35521
I20251215 21:03:21.447479 7921 heartbeater.cc:461] Registering TS with master...
I20251215 21:03:21.448021 7921 heartbeater.cc:507] Master 127.28.113.62:35521 requested a full tablet report, sending...
I20251215 21:03:21.449004 7592 ts_manager.cc:194] Registered new tserver with Master: 3432246dc9284624876b603d75b91697 (127.28.113.2:40065)
I20251215 21:03:21.449540 7592 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.113.2@KRBTEST.COM'} at 127.28.113.2:40387
WARNING: no policy specified for kudu/127.28.113.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.113.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.113.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765832506145157-29124-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.28.113.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765832506145157-29124-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.28.113.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.28.113.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.28.113.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765832506145157-29124-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.28.113.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765832506145157-29124-0/krb5kdc/kudu.keytab.
I20251215 21:03:21.484097 29124 external_mini_cluster.cc:1366] Running /tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu
/tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765832506145157-29124-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765832506145157-29124-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765832506145157-29124-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765832506145157-29124-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.113.3:0
--local_ip_for_outbound_sockets=127.28.113.3
--webserver_interface=127.28.113.3
--webserver_port=0
--tserver_master_addrs=127.28.113.62:35521
--keytab_file=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765832506145157-29124-0/krb5kdc/kudu.keytab
--principal=kudu/127.28.113.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.28.113.20:32937
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765832506145157-29124-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765832506145157-29124-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765832506145157-29124-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251215 21:03:21.588431 7928 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251215 21:03:21.588661 7928 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251215 21:03:21.588739 7928 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251215 21:03:21.592083 7928 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251215 21:03:21.592159 7928 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251215 21:03:21.592264 7928 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.113.3
I20251215 21:03:21.596485 7928 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.113.20:32937
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765832506145157-29124-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765832506145157-29124-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765832506145157-29124-0/krb5kdc/kudu.keytab
--principal=kudu/127.28.113.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.113.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765832506145157-29124-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.113.3
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.28.113.62:35521
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7928
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.113.3
--log_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765832506145157-29124-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision dc741732bb6a40213e11a13869fad06a49a209a3
build type DEBUG
built by None at 15 Dec 2025 20:43:14 UTC on 24a791456cd2
build id 9530
I20251215 21:03:21.597507 7928 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251215 21:03:21.598325 7928 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251215 21:03:21.604372 7933 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251215 21:03:21.604429 7928 server_base.cc:1047] running on GCE node
W20251215 21:03:21.604347 7934 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251215 21:03:21.604347 7936 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251215 21:03:21.604899 7928 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251215 21:03:21.605412 7928 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251215 21:03:21.606559 7928 hybrid_clock.cc:648] HybridClock initialized: now 1765832601606548 us; error 27 us; skew 500 ppm
Dec 15 21:03:21 dist-test-slave-bjsf krb5kdc[7544](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765832601, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.113.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251215 21:03:21.609170 7928 init.cc:377] Logged in from keytab as kudu/127.28.113.3@KRBTEST.COM (short username kudu)
I20251215 21:03:21.610150 7928 webserver.cc:492] Webserver started at http://127.28.113.3:37263/ using document root <none> and password file <none>
I20251215 21:03:21.610661 7928 fs_manager.cc:362] Metadata directory not provided
I20251215 21:03:21.610708 7928 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251215 21:03:21.610875 7928 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251215 21:03:21.612538 7928 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765832506145157-29124-0/minicluster-data/ts-2/data/instance:
uuid: "c2cdb9aa2aa948ea980b3d0d773d60d0"
format_stamp: "Formatted at 2025-12-15 21:03:21 on dist-test-slave-bjsf"
server_key: "156dab34ce9644542f30aca8d1e29124"
server_key_iv: "e51bd86dae19a3a019c33b5a3c8ed3fe"
server_key_version: "encryptionkey@0"
I20251215 21:03:21.612967 7928 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765832506145157-29124-0/minicluster-data/ts-2/wal/instance:
uuid: "c2cdb9aa2aa948ea980b3d0d773d60d0"
format_stamp: "Formatted at 2025-12-15 21:03:21 on dist-test-slave-bjsf"
server_key: "156dab34ce9644542f30aca8d1e29124"
server_key_iv: "e51bd86dae19a3a019c33b5a3c8ed3fe"
server_key_version: "encryptionkey@0"
I20251215 21:03:21.616217 7928 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20251215 21:03:21.618245 7943 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251215 21:03:21.619331 7928 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20251215 21:03:21.619475 7928 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765832506145157-29124-0/minicluster-data/ts-2/data,/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765832506145157-29124-0/minicluster-data/ts-2/wal
uuid: "c2cdb9aa2aa948ea980b3d0d773d60d0"
format_stamp: "Formatted at 2025-12-15 21:03:21 on dist-test-slave-bjsf"
server_key: "156dab34ce9644542f30aca8d1e29124"
server_key_iv: "e51bd86dae19a3a019c33b5a3c8ed3fe"
server_key_version: "encryptionkey@0"
I20251215 21:03:21.619582 7928 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765832506145157-29124-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765832506145157-29124-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765832506145157-29124-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251215 21:03:21.636826 7928 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251215 21:03:21.639786 7928 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251215 21:03:21.639987 7928 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251215 21:03:21.640580 7928 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251215 21:03:21.641454 7928 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251215 21:03:21.641502 7928 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251215 21:03:21.641563 7928 ts_tablet_manager.cc:616] Registered 0 tablets
I20251215 21:03:21.641592 7928 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251215 21:03:21.650025 7928 rpc_server.cc:307] RPC server started. Bound to: 127.28.113.3:43437
I20251215 21:03:21.650065 8056 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.28.113.3:43437 every 8 connection(s)
I20251215 21:03:21.651005 7928 server_base.cc:1179] Dumped server information to /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765832506145157-29124-0/minicluster-data/ts-2/data/info.pb
Dec 15 21:03:21 dist-test-slave-bjsf krb5kdc[7544](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765832601, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.113.3@KRBTEST.COM for kudu/127.28.113.62@KRBTEST.COM
I20251215 21:03:21.659405 29124 external_mini_cluster.cc:1428] Started /tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu as pid 7928
I20251215 21:03:21.659521 29124 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765832506145157-29124-0/minicluster-data/ts-2/wal/instance
I20251215 21:03:21.659788 29124 external_mini_cluster.cc:1468] Setting key 3f47811ee4bc6e7e051a8682fbc8bb0e
I20251215 21:03:21.662783 7631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:03:21.652684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.113.3:46841 (local address 127.28.113.62:35521)
1215 21:03:21.652819 (+ 135us) server_negotiation.cc:207] Beginning negotiation
1215 21:03:21.652823 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1215 21:03:21.653448 (+ 625us) server_negotiation.cc:408] Connection header received
1215 21:03:21.654255 (+ 807us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1215 21:03:21.654259 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1215 21:03:21.654317 (+ 58us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1215 21:03:21.654390 (+ 73us) server_negotiation.cc:227] Negotiated authn=SASL
1215 21:03:21.655994 (+ 1604us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:03:21.656503 (+ 509us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1215 21:03:21.657087 (+ 584us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:03:21.657257 (+ 170us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:03:21.659283 (+ 2026us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1215 21:03:21.659305 (+ 22us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1215 21:03:21.659308 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1215 21:03:21.659334 (+ 26us) server_negotiation.cc:923] calling sasl_server_start()
1215 21:03:21.660898 (+ 1564us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1215 21:03:21.661392 (+ 494us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1215 21:03:21.661396 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1215 21:03:21.661398 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1215 21:03:21.661443 (+ 45us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1215 21:03:21.661714 (+ 271us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1215 21:03:21.661718 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1215 21:03:21.661720 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1215 21:03:21.661891 (+ 171us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1215 21:03:21.661985 (+ 94us) server_negotiation.cc:1036] Waiting for connection context
1215 21:03:21.662485 (+ 500us) server_negotiation.cc:300] Negotiation successful
1215 21:03:21.662608 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20251215 21:03:21.663417 8059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:03:21.652914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.113.62:35521 (local address 127.28.113.3:46841)
1215 21:03:21.653323 (+ 409us) negotiation.cc:107] Waiting for socket to connect
1215 21:03:21.653358 (+ 35us) client_negotiation.cc:174] Beginning negotiation
1215 21:03:21.654069 (+ 711us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1215 21:03:21.654549 (+ 480us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1215 21:03:21.654556 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
1215 21:03:21.654928 (+ 372us) client_negotiation.cc:189] Negotiated authn=SASL
1215 21:03:21.655822 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:03:21.655836 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:03:21.656612 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1215 21:03:21.656615 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1215 21:03:21.656986 (+ 371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:03:21.656992 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:03:21.657201 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:03:21.657759 (+ 558us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1215 21:03:21.657781 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
1215 21:03:21.659125 (+ 1344us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1215 21:03:21.661035 (+ 1910us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1215 21:03:21.661042 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1215 21:03:21.661054 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
1215 21:03:21.661296 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1215 21:03:21.661522 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1215 21:03:21.661525 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1215 21:03:21.661526 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1215 21:03:21.661629 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1215 21:03:21.661998 (+ 369us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1215 21:03:21.662006 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1215 21:03:21.662250 (+ 244us) client_negotiation.cc:769] Sending connection context
1215 21:03:21.662436 (+ 186us) client_negotiation.cc:240] Negotiation successful
1215 21:03:21.662704 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":96,"threads_started":1}
I20251215 21:03:21.664453 8057 heartbeater.cc:344] Connected to a master server at 127.28.113.62:35521
I20251215 21:03:21.664686 8057 heartbeater.cc:461] Registering TS with master...
I20251215 21:03:21.665227 8057 heartbeater.cc:507] Master 127.28.113.62:35521 requested a full tablet report, sending...
I20251215 21:03:21.666217 7592 ts_manager.cc:194] Registered new tserver with Master: c2cdb9aa2aa948ea980b3d0d773d60d0 (127.28.113.3:43437)
I20251215 21:03:21.666747 7592 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.113.3@KRBTEST.COM'} at 127.28.113.3:46841
I20251215 21:03:21.673354 29124 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20251215 21:03:21.681313 7631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:03:21.675121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33924 (local address 127.28.113.62:35521)
1215 21:03:21.675244 (+ 123us) server_negotiation.cc:207] Beginning negotiation
1215 21:03:21.675249 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1215 21:03:21.675384 (+ 135us) server_negotiation.cc:408] Connection header received
1215 21:03:21.675517 (+ 133us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1215 21:03:21.675521 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1215 21:03:21.675568 (+ 47us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1215 21:03:21.675627 (+ 59us) server_negotiation.cc:227] Negotiated authn=SASL
1215 21:03:21.676485 (+ 858us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:03:21.676978 (+ 493us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1215 21:03:21.677588 (+ 610us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:03:21.677758 (+ 170us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:03:21.678848 (+ 1090us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1215 21:03:21.678867 (+ 19us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1215 21:03:21.678870 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1215 21:03:21.678895 (+ 25us) server_negotiation.cc:923] calling sasl_server_start()
1215 21:03:21.679987 (+ 1092us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1215 21:03:21.680346 (+ 359us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1215 21:03:21.680350 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1215 21:03:21.680352 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1215 21:03:21.680394 (+ 42us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1215 21:03:21.680577 (+ 183us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1215 21:03:21.680581 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1215 21:03:21.680583 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1215 21:03:21.680754 (+ 171us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1215 21:03:21.680869 (+ 115us) server_negotiation.cc:1036] Waiting for connection context
1215 21:03:21.681058 (+ 189us) server_negotiation.cc:300] Negotiation successful
1215 21:03:21.681163 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20251215 21:03:21.683629 7592 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33924:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251215 21:03:21.685968 7592 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251215 21:03:21.698414 8068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:03:21.694020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.113.1:41735 (local address 127.0.0.1:52564)
1215 21:03:21.695103 (+ 1083us) negotiation.cc:107] Waiting for socket to connect
1215 21:03:21.695115 (+ 12us) client_negotiation.cc:174] Beginning negotiation
1215 21:03:21.695237 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1215 21:03:21.695838 (+ 601us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1215 21:03:21.695841 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1215 21:03:21.695859 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1215 21:03:21.696099 (+ 240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:03:21.696107 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:03:21.697114 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1215 21:03:21.697117 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1215 21:03:21.697917 (+ 800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:03:21.697925 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:03:21.698024 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:03:21.698083 (+ 59us) client_negotiation.cc:769] Sending connection context
1215 21:03:21.698169 (+ 86us) client_negotiation.cc:240] Negotiation successful
1215 21:03:21.698270 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1022,"thread_start_us":88,"threads_started":1}
I20251215 21:03:21.698714 8072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:03:21.694960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.113.2:40065 (local address 127.0.0.1:53692)
1215 21:03:21.695179 (+ 219us) negotiation.cc:107] Waiting for socket to connect
1215 21:03:21.695192 (+ 13us) client_negotiation.cc:174] Beginning negotiation
1215 21:03:21.695275 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1215 21:03:21.696004 (+ 729us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1215 21:03:21.696008 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
1215 21:03:21.696024 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1215 21:03:21.696246 (+ 222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:03:21.696254 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:03:21.697584 (+ 1330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1215 21:03:21.697587 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1215 21:03:21.698421 (+ 834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:03:21.698429 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:03:21.698503 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:03:21.698517 (+ 14us) client_negotiation.cc:769] Sending connection context
1215 21:03:21.698552 (+ 35us) client_negotiation.cc:240] Negotiation successful
1215 21:03:21.698601 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":90,"threads_started":1}
I20251215 21:03:21.698951 8071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:03:21.694308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52564 (local address 127.28.113.1:41735)
1215 21:03:21.694986 (+ 678us) server_negotiation.cc:207] Beginning negotiation
1215 21:03:21.694992 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1215 21:03:21.695180 (+ 188us) server_negotiation.cc:408] Connection header received
1215 21:03:21.695389 (+ 209us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1215 21:03:21.695394 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1215 21:03:21.695544 (+ 150us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1215 21:03:21.695681 (+ 137us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1215 21:03:21.696223 (+ 542us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:03:21.696994 (+ 771us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1215 21:03:21.698058 (+ 1064us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:03:21.698567 (+ 509us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:03:21.698682 (+ 115us) server_negotiation.cc:1036] Waiting for connection context
1215 21:03:21.698744 (+ 62us) server_negotiation.cc:300] Negotiation successful
1215 21:03:21.698824 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":578,"thread_start_us":82,"threads_started":1}
I20251215 21:03:21.699391 8069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:03:21.693957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39790 (local address 127.28.113.3:43437)
1215 21:03:21.694284 (+ 327us) server_negotiation.cc:207] Beginning negotiation
1215 21:03:21.694291 (+ 7us) server_negotiation.cc:400] Waiting for connection header
1215 21:03:21.694728 (+ 437us) server_negotiation.cc:408] Connection header received
1215 21:03:21.694989 (+ 261us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1215 21:03:21.694995 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
1215 21:03:21.695196 (+ 201us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1215 21:03:21.695327 (+ 131us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1215 21:03:21.695918 (+ 591us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:03:21.697085 (+ 1167us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1215 21:03:21.698490 (+ 1405us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:03:21.699011 (+ 521us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:03:21.699107 (+ 96us) server_negotiation.cc:1036] Waiting for connection context
1215 21:03:21.699178 (+ 71us) server_negotiation.cc:300] Negotiation successful
1215 21:03:21.699260 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":216,"thread_start_us":121,"threads_started":1}
I20251215 21:03:21.699676 8070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:03:21.693852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.113.3:43437 (local address 127.0.0.1:39790)
1215 21:03:21.694609 (+ 757us) negotiation.cc:107] Waiting for socket to connect
1215 21:03:21.694648 (+ 39us) client_negotiation.cc:174] Beginning negotiation
1215 21:03:21.694794 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1215 21:03:21.695497 (+ 703us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1215 21:03:21.695502 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
1215 21:03:21.695521 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1215 21:03:21.695801 (+ 280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:03:21.695810 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:03:21.697226 (+ 1416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1215 21:03:21.697230 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1215 21:03:21.698128 (+ 898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:03:21.698138 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:03:21.698251 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:03:21.698270 (+ 19us) client_negotiation.cc:769] Sending connection context
1215 21:03:21.698311 (+ 41us) client_negotiation.cc:240] Negotiation successful
1215 21:03:21.698362 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":593,"thread_start_us":89,"threads_started":1}
I20251215 21:03:21.700228 8073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:03:21.695356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53692 (local address 127.28.113.2:40065)
1215 21:03:21.695624 (+ 268us) server_negotiation.cc:207] Beginning negotiation
1215 21:03:21.695629 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1215 21:03:21.695647 (+ 18us) server_negotiation.cc:408] Connection header received
1215 21:03:21.695711 (+ 64us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1215 21:03:21.695715 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1215 21:03:21.695845 (+ 130us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1215 21:03:21.696055 (+ 210us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1215 21:03:21.696354 (+ 299us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:03:21.697466 (+ 1112us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1215 21:03:21.699299 (+ 1833us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:03:21.699799 (+ 500us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:03:21.699899 (+ 100us) server_negotiation.cc:1036] Waiting for connection context
1215 21:03:21.700003 (+ 104us) server_negotiation.cc:300] Negotiation successful
1215 21:03:21.700091 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":172,"thread_start_us":100,"threads_started":1}
I20251215 21:03:21.701740 7719 tablet_service.cc:1505] Processing CreateTablet for tablet 50e91d6cd81c423da372c9438daf956d (DEFAULT_TABLE table=test-table [id=c89ac7418a684f648f347304c496a0b9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251215 21:03:21.702596 7719 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 50e91d6cd81c423da372c9438daf956d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 21:03:21.702358 7991 tablet_service.cc:1505] Processing CreateTablet for tablet 50e91d6cd81c423da372c9438daf956d (DEFAULT_TABLE table=test-table [id=c89ac7418a684f648f347304c496a0b9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251215 21:03:21.702358 7855 tablet_service.cc:1505] Processing CreateTablet for tablet 50e91d6cd81c423da372c9438daf956d (DEFAULT_TABLE table=test-table [id=c89ac7418a684f648f347304c496a0b9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251215 21:03:21.703287 7991 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 50e91d6cd81c423da372c9438daf956d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 21:03:21.703287 7855 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 50e91d6cd81c423da372c9438daf956d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 21:03:21.707939 8074 tablet_bootstrap.cc:492] T 50e91d6cd81c423da372c9438daf956d P e6b9743daabe409084b61f874bd0cb25: Bootstrap starting.
I20251215 21:03:21.709877 8075 tablet_bootstrap.cc:492] T 50e91d6cd81c423da372c9438daf956d P 3432246dc9284624876b603d75b91697: Bootstrap starting.
I20251215 21:03:21.709877 8076 tablet_bootstrap.cc:492] T 50e91d6cd81c423da372c9438daf956d P c2cdb9aa2aa948ea980b3d0d773d60d0: Bootstrap starting.
I20251215 21:03:21.710677 8074 tablet_bootstrap.cc:654] T 50e91d6cd81c423da372c9438daf956d P e6b9743daabe409084b61f874bd0cb25: Neither blocks nor log segments found. Creating new log.
I20251215 21:03:21.711362 8074 log.cc:826] T 50e91d6cd81c423da372c9438daf956d P e6b9743daabe409084b61f874bd0cb25: Log is configured to *not* fsync() on all Append() calls
I20251215 21:03:21.711906 8075 tablet_bootstrap.cc:654] T 50e91d6cd81c423da372c9438daf956d P 3432246dc9284624876b603d75b91697: Neither blocks nor log segments found. Creating new log.
I20251215 21:03:21.711915 8076 tablet_bootstrap.cc:654] T 50e91d6cd81c423da372c9438daf956d P c2cdb9aa2aa948ea980b3d0d773d60d0: Neither blocks nor log segments found. Creating new log.
I20251215 21:03:21.712742 8075 log.cc:826] T 50e91d6cd81c423da372c9438daf956d P 3432246dc9284624876b603d75b91697: Log is configured to *not* fsync() on all Append() calls
I20251215 21:03:21.712742 8076 log.cc:826] T 50e91d6cd81c423da372c9438daf956d P c2cdb9aa2aa948ea980b3d0d773d60d0: Log is configured to *not* fsync() on all Append() calls
I20251215 21:03:21.713110 8074 tablet_bootstrap.cc:492] T 50e91d6cd81c423da372c9438daf956d P e6b9743daabe409084b61f874bd0cb25: No bootstrap required, opened a new log
I20251215 21:03:21.713310 8074 ts_tablet_manager.cc:1403] T 50e91d6cd81c423da372c9438daf956d P e6b9743daabe409084b61f874bd0cb25: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.003s
I20251215 21:03:21.714474 8075 tablet_bootstrap.cc:492] T 50e91d6cd81c423da372c9438daf956d P 3432246dc9284624876b603d75b91697: No bootstrap required, opened a new log
I20251215 21:03:21.714731 8075 ts_tablet_manager.cc:1403] T 50e91d6cd81c423da372c9438daf956d P 3432246dc9284624876b603d75b91697: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20251215 21:03:21.714803 8076 tablet_bootstrap.cc:492] T 50e91d6cd81c423da372c9438daf956d P c2cdb9aa2aa948ea980b3d0d773d60d0: No bootstrap required, opened a new log
I20251215 21:03:21.714964 8076 ts_tablet_manager.cc:1403] T 50e91d6cd81c423da372c9438daf956d P c2cdb9aa2aa948ea980b3d0d773d60d0: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20251215 21:03:21.716373 8074 raft_consensus.cc:359] T 50e91d6cd81c423da372c9438daf956d P e6b9743daabe409084b61f874bd0cb25 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2cdb9aa2aa948ea980b3d0d773d60d0" member_type: VOTER last_known_addr { host: "127.28.113.3" port: 43437 } } peers { permanent_uuid: "e6b9743daabe409084b61f874bd0cb25" member_type: VOTER last_known_addr { host: "127.28.113.1" port: 41735 } } peers { permanent_uuid: "3432246dc9284624876b603d75b91697" member_type: VOTER last_known_addr { host: "127.28.113.2" port: 40065 } }
I20251215 21:03:21.716557 8074 raft_consensus.cc:385] T 50e91d6cd81c423da372c9438daf956d P e6b9743daabe409084b61f874bd0cb25 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251215 21:03:21.716609 8074 raft_consensus.cc:740] T 50e91d6cd81c423da372c9438daf956d P e6b9743daabe409084b61f874bd0cb25 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6b9743daabe409084b61f874bd0cb25, State: Initialized, Role: FOLLOWER
I20251215 21:03:21.717010 8074 consensus_queue.cc:260] T 50e91d6cd81c423da372c9438daf956d P e6b9743daabe409084b61f874bd0cb25 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2cdb9aa2aa948ea980b3d0d773d60d0" member_type: VOTER last_known_addr { host: "127.28.113.3" port: 43437 } } peers { permanent_uuid: "e6b9743daabe409084b61f874bd0cb25" member_type: VOTER last_known_addr { host: "127.28.113.1" port: 41735 } } peers { permanent_uuid: "3432246dc9284624876b603d75b91697" member_type: VOTER last_known_addr { host: "127.28.113.2" port: 40065 } }
I20251215 21:03:21.717657 7785 heartbeater.cc:499] Master 127.28.113.62:35521 was elected leader, sending a full tablet report...
I20251215 21:03:21.717809 8074 ts_tablet_manager.cc:1434] T 50e91d6cd81c423da372c9438daf956d P e6b9743daabe409084b61f874bd0cb25: Time spent starting tablet: real 0.004s user 0.004s sys 0.001s
I20251215 21:03:21.718384 8075 raft_consensus.cc:359] T 50e91d6cd81c423da372c9438daf956d P 3432246dc9284624876b603d75b91697 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2cdb9aa2aa948ea980b3d0d773d60d0" member_type: VOTER last_known_addr { host: "127.28.113.3" port: 43437 } } peers { permanent_uuid: "e6b9743daabe409084b61f874bd0cb25" member_type: VOTER last_known_addr { host: "127.28.113.1" port: 41735 } } peers { permanent_uuid: "3432246dc9284624876b603d75b91697" member_type: VOTER last_known_addr { host: "127.28.113.2" port: 40065 } }
I20251215 21:03:21.718585 8075 raft_consensus.cc:385] T 50e91d6cd81c423da372c9438daf956d P 3432246dc9284624876b603d75b91697 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251215 21:03:21.718637 8075 raft_consensus.cc:740] T 50e91d6cd81c423da372c9438daf956d P 3432246dc9284624876b603d75b91697 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3432246dc9284624876b603d75b91697, State: Initialized, Role: FOLLOWER
I20251215 21:03:21.718681 8076 raft_consensus.cc:359] T 50e91d6cd81c423da372c9438daf956d P c2cdb9aa2aa948ea980b3d0d773d60d0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2cdb9aa2aa948ea980b3d0d773d60d0" member_type: VOTER last_known_addr { host: "127.28.113.3" port: 43437 } } peers { permanent_uuid: "e6b9743daabe409084b61f874bd0cb25" member_type: VOTER last_known_addr { host: "127.28.113.1" port: 41735 } } peers { permanent_uuid: "3432246dc9284624876b603d75b91697" member_type: VOTER last_known_addr { host: "127.28.113.2" port: 40065 } }
I20251215 21:03:21.718870 8076 raft_consensus.cc:385] T 50e91d6cd81c423da372c9438daf956d P c2cdb9aa2aa948ea980b3d0d773d60d0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251215 21:03:21.718919 8076 raft_consensus.cc:740] T 50e91d6cd81c423da372c9438daf956d P c2cdb9aa2aa948ea980b3d0d773d60d0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2cdb9aa2aa948ea980b3d0d773d60d0, State: Initialized, Role: FOLLOWER
I20251215 21:03:21.719043 8075 consensus_queue.cc:260] T 50e91d6cd81c423da372c9438daf956d P 3432246dc9284624876b603d75b91697 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2cdb9aa2aa948ea980b3d0d773d60d0" member_type: VOTER last_known_addr { host: "127.28.113.3" port: 43437 } } peers { permanent_uuid: "e6b9743daabe409084b61f874bd0cb25" member_type: VOTER last_known_addr { host: "127.28.113.1" port: 41735 } } peers { permanent_uuid: "3432246dc9284624876b603d75b91697" member_type: VOTER last_known_addr { host: "127.28.113.2" port: 40065 } }
I20251215 21:03:21.719286 8076 consensus_queue.cc:260] T 50e91d6cd81c423da372c9438daf956d P c2cdb9aa2aa948ea980b3d0d773d60d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2cdb9aa2aa948ea980b3d0d773d60d0" member_type: VOTER last_known_addr { host: "127.28.113.3" port: 43437 } } peers { permanent_uuid: "e6b9743daabe409084b61f874bd0cb25" member_type: VOTER last_known_addr { host: "127.28.113.1" port: 41735 } } peers { permanent_uuid: "3432246dc9284624876b603d75b91697" member_type: VOTER last_known_addr { host: "127.28.113.2" port: 40065 } }
I20251215 21:03:21.719743 7921 heartbeater.cc:499] Master 127.28.113.62:35521 was elected leader, sending a full tablet report...
I20251215 21:03:21.719857 8075 ts_tablet_manager.cc:1434] T 50e91d6cd81c423da372c9438daf956d P 3432246dc9284624876b603d75b91697: Time spent starting tablet: real 0.005s user 0.000s sys 0.004s
I20251215 21:03:21.720074 8076 ts_tablet_manager.cc:1434] T 50e91d6cd81c423da372c9438daf956d P c2cdb9aa2aa948ea980b3d0d773d60d0: Time spent starting tablet: real 0.005s user 0.000s sys 0.005s
I20251215 21:03:21.720414 8057 heartbeater.cc:499] Master 127.28.113.62:35521 was elected leader, sending a full tablet report...
I20251215 21:03:21.867831 8082 raft_consensus.cc:493] T 50e91d6cd81c423da372c9438daf956d P c2cdb9aa2aa948ea980b3d0d773d60d0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 21:03:21.868067 8082 raft_consensus.cc:515] T 50e91d6cd81c423da372c9438daf956d P c2cdb9aa2aa948ea980b3d0d773d60d0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2cdb9aa2aa948ea980b3d0d773d60d0" member_type: VOTER last_known_addr { host: "127.28.113.3" port: 43437 } } peers { permanent_uuid: "e6b9743daabe409084b61f874bd0cb25" member_type: VOTER last_known_addr { host: "127.28.113.1" port: 41735 } } peers { permanent_uuid: "3432246dc9284624876b603d75b91697" member_type: VOTER last_known_addr { host: "127.28.113.2" port: 40065 } }
I20251215 21:03:21.868341 8080 raft_consensus.cc:493] T 50e91d6cd81c423da372c9438daf956d P e6b9743daabe409084b61f874bd0cb25 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251215 21:03:21.868495 8080 raft_consensus.cc:515] T 50e91d6cd81c423da372c9438daf956d P e6b9743daabe409084b61f874bd0cb25 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2cdb9aa2aa948ea980b3d0d773d60d0" member_type: VOTER last_known_addr { host: "127.28.113.3" port: 43437 } } peers { permanent_uuid: "e6b9743daabe409084b61f874bd0cb25" member_type: VOTER last_known_addr { host: "127.28.113.1" port: 41735 } } peers { permanent_uuid: "3432246dc9284624876b603d75b91697" member_type: VOTER last_known_addr { host: "127.28.113.2" port: 40065 } }
I20251215 21:03:21.869477 8080 leader_election.cc:290] T 50e91d6cd81c423da372c9438daf956d P e6b9743daabe409084b61f874bd0cb25 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c2cdb9aa2aa948ea980b3d0d773d60d0 (127.28.113.3:43437), 3432246dc9284624876b603d75b91697 (127.28.113.2:40065)
I20251215 21:03:21.869729 8082 leader_election.cc:290] T 50e91d6cd81c423da372c9438daf956d P c2cdb9aa2aa948ea980b3d0d773d60d0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e6b9743daabe409084b61f874bd0cb25 (127.28.113.1:41735), 3432246dc9284624876b603d75b91697 (127.28.113.2:40065)
I20251215 21:03:21.873116 8059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:03:21.869689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.113.1:41735 (local address 127.28.113.3:33933)
1215 21:03:21.869902 (+ 213us) negotiation.cc:107] Waiting for socket to connect
1215 21:03:21.869930 (+ 28us) client_negotiation.cc:174] Beginning negotiation
1215 21:03:21.870023 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1215 21:03:21.870629 (+ 606us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1215 21:03:21.870632 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1215 21:03:21.870648 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1215 21:03:21.870879 (+ 231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:03:21.870889 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:03:21.871858 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1215 21:03:21.871861 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1215 21:03:21.872754 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:03:21.872762 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:03:21.872861 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:03:21.872882 (+ 21us) client_negotiation.cc:769] Sending connection context
1215 21:03:21.872930 (+ 48us) client_negotiation.cc:240] Negotiation successful
1215 21:03:21.872988 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130}
I20251215 21:03:21.873880 8083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:03:21.869690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.113.3:43437 (local address 127.28.113.1:48105)
1215 21:03:21.870044 (+ 354us) negotiation.cc:107] Waiting for socket to connect
1215 21:03:21.870059 (+ 15us) client_negotiation.cc:174] Beginning negotiation
1215 21:03:21.870157 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1215 21:03:21.870806 (+ 649us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1215 21:03:21.870809 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1215 21:03:21.870825 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1215 21:03:21.871107 (+ 282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:03:21.871116 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:03:21.872075 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1215 21:03:21.872078 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1215 21:03:21.872931 (+ 853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:03:21.872939 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:03:21.873600 (+ 661us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:03:21.873616 (+ 16us) client_negotiation.cc:769] Sending connection context
1215 21:03:21.873686 (+ 70us) client_negotiation.cc:240] Negotiation successful
1215 21:03:21.873743 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":91,"threads_started":1}
I20251215 21:03:21.873905 8073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:03:21.869806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.113.3:56929 (local address 127.28.113.2:40065)
1215 21:03:21.869937 (+ 131us) server_negotiation.cc:207] Beginning negotiation
1215 21:03:21.869942 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1215 21:03:21.870455 (+ 513us) server_negotiation.cc:408] Connection header received
1215 21:03:21.870643 (+ 188us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1215 21:03:21.870647 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1215 21:03:21.870694 (+ 47us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1215 21:03:21.870731 (+ 37us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1215 21:03:21.871234 (+ 503us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:03:21.871908 (+ 674us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1215 21:03:21.872972 (+ 1064us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:03:21.873438 (+ 466us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:03:21.873474 (+ 36us) server_negotiation.cc:1036] Waiting for connection context
1215 21:03:21.873730 (+ 256us) server_negotiation.cc:300] Negotiation successful
1215 21:03:21.873797 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20251215 21:03:21.874053 8071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:03:21.870280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.113.3:33933 (local address 127.28.113.1:41735)
1215 21:03:21.870379 (+ 99us) server_negotiation.cc:207] Beginning negotiation
1215 21:03:21.870383 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1215 21:03:21.870394 (+ 11us) server_negotiation.cc:408] Connection header received
1215 21:03:21.870428 (+ 34us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1215 21:03:21.870431 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1215 21:03:21.870466 (+ 35us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1215 21:03:21.870525 (+ 59us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1215 21:03:21.871032 (+ 507us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:03:21.871742 (+ 710us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1215 21:03:21.873382 (+ 1640us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:03:21.873818 (+ 436us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:03:21.873847 (+ 29us) server_negotiation.cc:1036] Waiting for connection context
1215 21:03:21.873888 (+ 41us) server_negotiation.cc:300] Negotiation successful
1215 21:03:21.873940 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40,"spinlock_wait_cycles":2816}
I20251215 21:03:21.874279 8069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:03:21.870148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.113.1:48105 (local address 127.28.113.3:43437)
1215 21:03:21.870254 (+ 106us) server_negotiation.cc:207] Beginning negotiation
1215 21:03:21.870257 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1215 21:03:21.870270 (+ 13us) server_negotiation.cc:408] Connection header received
1215 21:03:21.870313 (+ 43us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1215 21:03:21.870316 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1215 21:03:21.870360 (+ 44us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1215 21:03:21.870428 (+ 68us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1215 21:03:21.871228 (+ 800us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:03:21.871904 (+ 676us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1215 21:03:21.873047 (+ 1143us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:03:21.873509 (+ 462us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:03:21.873542 (+ 33us) server_negotiation.cc:1036] Waiting for connection context
1215 21:03:21.874112 (+ 570us) server_negotiation.cc:300] Negotiation successful
1215 21:03:21.874174 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20251215 21:03:21.874460 8085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:03:21.870023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.113.2:40065 (local address 127.28.113.1:57557)
1215 21:03:21.870470 (+ 447us) negotiation.cc:107] Waiting for socket to connect
1215 21:03:21.870481 (+ 11us) client_negotiation.cc:174] Beginning negotiation
1215 21:03:21.870573 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1215 21:03:21.871459 (+ 886us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1215 21:03:21.871462 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1215 21:03:21.871478 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1215 21:03:21.871762 (+ 284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:03:21.871769 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:03:21.872863 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1215 21:03:21.872867 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1215 21:03:21.873708 (+ 841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:03:21.873716 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:03:21.873800 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:03:21.873815 (+ 15us) client_negotiation.cc:769] Sending connection context
1215 21:03:21.873855 (+ 40us) client_negotiation.cc:240] Negotiation successful
1215 21:03:21.873898 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":386,"spinlock_wait_cycles":768,"thread_start_us":135,"threads_started":1}
I20251215 21:03:21.874636 8084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:03:21.869700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.113.2:40065 (local address 127.28.113.3:56929)
1215 21:03:21.870383 (+ 683us) negotiation.cc:107] Waiting for socket to connect
1215 21:03:21.870396 (+ 13us) client_negotiation.cc:174] Beginning negotiation
1215 21:03:21.870524 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1215 21:03:21.870834 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1215 21:03:21.870836 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1215 21:03:21.870849 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1215 21:03:21.871112 (+ 263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:03:21.871120 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:03:21.872056 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1215 21:03:21.872060 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1215 21:03:21.872856 (+ 796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:03:21.872863 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:03:21.873592 (+ 729us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:03:21.873607 (+ 15us) client_negotiation.cc:769] Sending connection context
1215 21:03:21.874157 (+ 550us) client_negotiation.cc:240] Negotiation successful
1215 21:03:21.874213 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":615,"mutex_wait_us":264,"thread_start_us":107,"threads_started":1}
I20251215 21:03:21.874733 8086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:03:21.870591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.113.1:57557 (local address 127.28.113.2:40065)
1215 21:03:21.871151 (+ 560us) server_negotiation.cc:207] Beginning negotiation
1215 21:03:21.871155 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1215 21:03:21.871181 (+ 26us) server_negotiation.cc:408] Connection header received
1215 21:03:21.871225 (+ 44us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1215 21:03:21.871228 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1215 21:03:21.871275 (+ 47us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1215 21:03:21.871347 (+ 72us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1215 21:03:21.871877 (+ 530us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:03:21.872740 (+ 863us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1215 21:03:21.874003 (+ 1263us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:03:21.874468 (+ 465us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:03:21.874499 (+ 31us) server_negotiation.cc:1036] Waiting for connection context
1215 21:03:21.874540 (+ 41us) server_negotiation.cc:300] Negotiation successful
1215 21:03:21.874592 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":487,"thread_start_us":109,"threads_started":1}
I20251215 21:03:21.874861 7739 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "50e91d6cd81c423da372c9438daf956d" candidate_uuid: "c2cdb9aa2aa948ea980b3d0d773d60d0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6b9743daabe409084b61f874bd0cb25" is_pre_election: true
I20251215 21:03:21.874874 8011 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "50e91d6cd81c423da372c9438daf956d" candidate_uuid: "e6b9743daabe409084b61f874bd0cb25" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2cdb9aa2aa948ea980b3d0d773d60d0" is_pre_election: true
I20251215 21:03:21.875061 8011 raft_consensus.cc:2468] T 50e91d6cd81c423da372c9438daf956d P c2cdb9aa2aa948ea980b3d0d773d60d0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6b9743daabe409084b61f874bd0cb25 in term 0.
I20251215 21:03:21.875061 7739 raft_consensus.cc:2468] T 50e91d6cd81c423da372c9438daf956d P e6b9743daabe409084b61f874bd0cb25 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c2cdb9aa2aa948ea980b3d0d773d60d0 in term 0.
I20251215 21:03:21.875241 7875 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "50e91d6cd81c423da372c9438daf956d" candidate_uuid: "c2cdb9aa2aa948ea980b3d0d773d60d0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3432246dc9284624876b603d75b91697" is_pre_election: true
I20251215 21:03:21.875335 7874 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "50e91d6cd81c423da372c9438daf956d" candidate_uuid: "e6b9743daabe409084b61f874bd0cb25" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3432246dc9284624876b603d75b91697" is_pre_election: true
I20251215 21:03:21.875499 7874 raft_consensus.cc:2468] T 50e91d6cd81c423da372c9438daf956d P 3432246dc9284624876b603d75b91697 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6b9743daabe409084b61f874bd0cb25 in term 0.
I20251215 21:03:21.875514 7946 leader_election.cc:304] T 50e91d6cd81c423da372c9438daf956d P c2cdb9aa2aa948ea980b3d0d773d60d0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c2cdb9aa2aa948ea980b3d0d773d60d0, e6b9743daabe409084b61f874bd0cb25; no voters:
I20251215 21:03:21.875764 8082 raft_consensus.cc:2804] T 50e91d6cd81c423da372c9438daf956d P c2cdb9aa2aa948ea980b3d0d773d60d0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251215 21:03:21.875835 8082 raft_consensus.cc:493] T 50e91d6cd81c423da372c9438daf956d P c2cdb9aa2aa948ea980b3d0d773d60d0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 21:03:21.875878 8082 raft_consensus.cc:3060] T 50e91d6cd81c423da372c9438daf956d P c2cdb9aa2aa948ea980b3d0d773d60d0 [term 0 FOLLOWER]: Advancing to term 1
I20251215 21:03:21.876075 7672 leader_election.cc:304] T 50e91d6cd81c423da372c9438daf956d P e6b9743daabe409084b61f874bd0cb25 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3432246dc9284624876b603d75b91697, e6b9743daabe409084b61f874bd0cb25; no voters:
I20251215 21:03:21.876286 8080 raft_consensus.cc:2804] T 50e91d6cd81c423da372c9438daf956d P e6b9743daabe409084b61f874bd0cb25 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251215 21:03:21.876365 8080 raft_consensus.cc:493] T 50e91d6cd81c423da372c9438daf956d P e6b9743daabe409084b61f874bd0cb25 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251215 21:03:21.876410 8080 raft_consensus.cc:3060] T 50e91d6cd81c423da372c9438daf956d P e6b9743daabe409084b61f874bd0cb25 [term 0 FOLLOWER]: Advancing to term 1
I20251215 21:03:21.877058 8082 raft_consensus.cc:515] T 50e91d6cd81c423da372c9438daf956d P c2cdb9aa2aa948ea980b3d0d773d60d0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2cdb9aa2aa948ea980b3d0d773d60d0" member_type: VOTER last_known_addr { host: "127.28.113.3" port: 43437 } } peers { permanent_uuid: "e6b9743daabe409084b61f874bd0cb25" member_type: VOTER last_known_addr { host: "127.28.113.1" port: 41735 } } peers { permanent_uuid: "3432246dc9284624876b603d75b91697" member_type: VOTER last_known_addr { host: "127.28.113.2" port: 40065 } }
I20251215 21:03:21.877465 8082 leader_election.cc:290] T 50e91d6cd81c423da372c9438daf956d P c2cdb9aa2aa948ea980b3d0d773d60d0 [CANDIDATE]: Term 1 election: Requested vote from peers e6b9743daabe409084b61f874bd0cb25 (127.28.113.1:41735), 3432246dc9284624876b603d75b91697 (127.28.113.2:40065)
I20251215 21:03:21.877444 8080 raft_consensus.cc:515] T 50e91d6cd81c423da372c9438daf956d P e6b9743daabe409084b61f874bd0cb25 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2cdb9aa2aa948ea980b3d0d773d60d0" member_type: VOTER last_known_addr { host: "127.28.113.3" port: 43437 } } peers { permanent_uuid: "e6b9743daabe409084b61f874bd0cb25" member_type: VOTER last_known_addr { host: "127.28.113.1" port: 41735 } } peers { permanent_uuid: "3432246dc9284624876b603d75b91697" member_type: VOTER last_known_addr { host: "127.28.113.2" port: 40065 } }
I20251215 21:03:21.877770 8080 leader_election.cc:290] T 50e91d6cd81c423da372c9438daf956d P e6b9743daabe409084b61f874bd0cb25 [CANDIDATE]: Term 1 election: Requested vote from peers c2cdb9aa2aa948ea980b3d0d773d60d0 (127.28.113.3:43437), 3432246dc9284624876b603d75b91697 (127.28.113.2:40065)
I20251215 21:03:21.877782 7874 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "50e91d6cd81c423da372c9438daf956d" candidate_uuid: "c2cdb9aa2aa948ea980b3d0d773d60d0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3432246dc9284624876b603d75b91697"
I20251215 21:03:21.877786 7739 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "50e91d6cd81c423da372c9438daf956d" candidate_uuid: "c2cdb9aa2aa948ea980b3d0d773d60d0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6b9743daabe409084b61f874bd0cb25"
I20251215 21:03:21.877897 7874 raft_consensus.cc:3060] T 50e91d6cd81c423da372c9438daf956d P 3432246dc9284624876b603d75b91697 [term 0 FOLLOWER]: Advancing to term 1
I20251215 21:03:21.877972 7739 raft_consensus.cc:2393] T 50e91d6cd81c423da372c9438daf956d P e6b9743daabe409084b61f874bd0cb25 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate c2cdb9aa2aa948ea980b3d0d773d60d0 in current term 1: Already voted for candidate e6b9743daabe409084b61f874bd0cb25 in this term.
I20251215 21:03:21.878113 8011 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "50e91d6cd81c423da372c9438daf956d" candidate_uuid: "e6b9743daabe409084b61f874bd0cb25" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2cdb9aa2aa948ea980b3d0d773d60d0"
I20251215 21:03:21.878249 8011 raft_consensus.cc:2393] T 50e91d6cd81c423da372c9438daf956d P c2cdb9aa2aa948ea980b3d0d773d60d0 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate e6b9743daabe409084b61f874bd0cb25 in current term 1: Already voted for candidate c2cdb9aa2aa948ea980b3d0d773d60d0 in this term.
I20251215 21:03:21.878379 7875 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "50e91d6cd81c423da372c9438daf956d" candidate_uuid: "e6b9743daabe409084b61f874bd0cb25" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3432246dc9284624876b603d75b91697"
I20251215 21:03:21.878736 7672 leader_election.cc:304] T 50e91d6cd81c423da372c9438daf956d P e6b9743daabe409084b61f874bd0cb25 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e6b9743daabe409084b61f874bd0cb25; no voters: 3432246dc9284624876b603d75b91697, c2cdb9aa2aa948ea980b3d0d773d60d0
I20251215 21:03:21.878949 8080 raft_consensus.cc:2749] T 50e91d6cd81c423da372c9438daf956d P e6b9743daabe409084b61f874bd0cb25 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20251215 21:03:21.879020 7874 raft_consensus.cc:2468] T 50e91d6cd81c423da372c9438daf956d P 3432246dc9284624876b603d75b91697 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c2cdb9aa2aa948ea980b3d0d773d60d0 in term 1.
I20251215 21:03:21.879293 7944 leader_election.cc:304] T 50e91d6cd81c423da372c9438daf956d P c2cdb9aa2aa948ea980b3d0d773d60d0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3432246dc9284624876b603d75b91697, c2cdb9aa2aa948ea980b3d0d773d60d0; no voters: e6b9743daabe409084b61f874bd0cb25
I20251215 21:03:21.879479 8082 raft_consensus.cc:2804] T 50e91d6cd81c423da372c9438daf956d P c2cdb9aa2aa948ea980b3d0d773d60d0 [term 1 FOLLOWER]: Leader election won for term 1
I20251215 21:03:21.879686 8082 raft_consensus.cc:697] T 50e91d6cd81c423da372c9438daf956d P c2cdb9aa2aa948ea980b3d0d773d60d0 [term 1 LEADER]: Becoming Leader. State: Replica: c2cdb9aa2aa948ea980b3d0d773d60d0, State: Running, Role: LEADER
I20251215 21:03:21.880028 8082 consensus_queue.cc:237] T 50e91d6cd81c423da372c9438daf956d P c2cdb9aa2aa948ea980b3d0d773d60d0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2cdb9aa2aa948ea980b3d0d773d60d0" member_type: VOTER last_known_addr { host: "127.28.113.3" port: 43437 } } peers { permanent_uuid: "e6b9743daabe409084b61f874bd0cb25" member_type: VOTER last_known_addr { host: "127.28.113.1" port: 41735 } } peers { permanent_uuid: "3432246dc9284624876b603d75b91697" member_type: VOTER last_known_addr { host: "127.28.113.2" port: 40065 } }
I20251215 21:03:21.883087 7591 catalog_manager.cc:5654] T 50e91d6cd81c423da372c9438daf956d P c2cdb9aa2aa948ea980b3d0d773d60d0 reported cstate change: term changed from 0 to 1, leader changed from <none> to c2cdb9aa2aa948ea980b3d0d773d60d0 (127.28.113.3). New cstate: current_term: 1 leader_uuid: "c2cdb9aa2aa948ea980b3d0d773d60d0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2cdb9aa2aa948ea980b3d0d773d60d0" member_type: VOTER last_known_addr { host: "127.28.113.3" port: 43437 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e6b9743daabe409084b61f874bd0cb25" member_type: VOTER last_known_addr { host: "127.28.113.1" port: 41735 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3432246dc9284624876b603d75b91697" member_type: VOTER last_known_addr { host: "127.28.113.2" port: 40065 } health_report { overall_health: UNKNOWN } } }
W20251215 21:03:21.901870 8058 tablet.cc:2378] T 50e91d6cd81c423da372c9438daf956d P c2cdb9aa2aa948ea980b3d0d773d60d0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20251215 21:03:21.932217 8069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:03:21.929498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39792 (local address 127.28.113.3:43437)
1215 21:03:21.929629 (+ 131us) server_negotiation.cc:207] Beginning negotiation
1215 21:03:21.929632 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1215 21:03:21.929657 (+ 25us) server_negotiation.cc:408] Connection header received
1215 21:03:21.929734 (+ 77us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1215 21:03:21.929736 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1215 21:03:21.929793 (+ 57us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1215 21:03:21.929864 (+ 71us) server_negotiation.cc:227] Negotiated authn=TOKEN
1215 21:03:21.930147 (+ 283us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:03:21.930616 (+ 469us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1215 21:03:21.931215 (+ 599us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:03:21.931342 (+ 127us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:03:21.931430 (+ 88us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1215 21:03:21.931864 (+ 434us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1215 21:03:21.931980 (+ 116us) server_negotiation.cc:1036] Waiting for connection context
1215 21:03:21.932060 (+ 80us) server_negotiation.cc:300] Negotiation successful
1215 21:03:21.932109 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
W20251215 21:03:21.934448 7922 tablet.cc:2378] T 50e91d6cd81c423da372c9438daf956d P 3432246dc9284624876b603d75b91697: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20251215 21:03:21.937512 7739 raft_consensus.cc:1275] T 50e91d6cd81c423da372c9438daf956d P e6b9743daabe409084b61f874bd0cb25 [term 1 FOLLOWER]: Refusing update from remote peer c2cdb9aa2aa948ea980b3d0d773d60d0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251215 21:03:21.937511 7874 raft_consensus.cc:1275] T 50e91d6cd81c423da372c9438daf956d P 3432246dc9284624876b603d75b91697 [term 1 FOLLOWER]: Refusing update from remote peer c2cdb9aa2aa948ea980b3d0d773d60d0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251215 21:03:21.938107 8087 consensus_queue.cc:1048] T 50e91d6cd81c423da372c9438daf956d P c2cdb9aa2aa948ea980b3d0d773d60d0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3432246dc9284624876b603d75b91697" member_type: VOTER last_known_addr { host: "127.28.113.2" port: 40065 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251215 21:03:21.938278 8082 consensus_queue.cc:1048] T 50e91d6cd81c423da372c9438daf956d P c2cdb9aa2aa948ea980b3d0d773d60d0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6b9743daabe409084b61f874bd0cb25" member_type: VOTER last_known_addr { host: "127.28.113.1" port: 41735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251215 21:03:21.945798 8091 mvcc.cc:204] Tried to move back new op lower bound from 7232850337529192448 to 7232850337303318528. Current Snapshot: MvccSnapshot[applied={T|T < 7232850337529192448}]
I20251215 21:03:21.946308 8090 mvcc.cc:204] Tried to move back new op lower bound from 7232850337529192448 to 7232850337303318528. Current Snapshot: MvccSnapshot[applied={T|T < 7232850337529192448 or (T in {7232850337529192448})}]
I20251215 21:03:21.952679 7592 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33924:
table { table_name: "test-table" } modify_external_catalogs: true
I20251215 21:03:21.952818 7592 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33924:
table { table_name: "test-table" } modify_external_catalogs: true
I20251215 21:03:21.954934 7592 catalog_manager.cc:5941] T 00000000000000000000000000000000 P ee0bd4d6b3d94ea3b6373a5365e9a832: Sending DeleteTablet for 3 replicas of tablet 50e91d6cd81c423da372c9438daf956d
I20251215 21:03:21.955629 7855 tablet_service.cc:1552] Processing DeleteTablet for tablet 50e91d6cd81c423da372c9438daf956d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-15 21:03:21 UTC) from {username='kudu', principal='kudu/127.28.113.62@KRBTEST.COM'} at 127.0.0.1:53692
I20251215 21:03:21.956059 7719 tablet_service.cc:1552] Processing DeleteTablet for tablet 50e91d6cd81c423da372c9438daf956d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-15 21:03:21 UTC) from {username='kudu', principal='kudu/127.28.113.62@KRBTEST.COM'} at 127.0.0.1:52564
I20251215 21:03:21.956103 29124 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu with pid 7656
I20251215 21:03:21.957911 7991 tablet_service.cc:1552] Processing DeleteTablet for tablet 50e91d6cd81c423da372c9438daf956d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-15 21:03:21 UTC) from {username='kudu', principal='kudu/127.28.113.62@KRBTEST.COM'} at 127.0.0.1:39790
I20251215 21:03:21.958340 8101 tablet_replica.cc:333] T 50e91d6cd81c423da372c9438daf956d P c2cdb9aa2aa948ea980b3d0d773d60d0: stopping tablet replica
I20251215 21:03:21.958627 8101 raft_consensus.cc:2243] T 50e91d6cd81c423da372c9438daf956d P c2cdb9aa2aa948ea980b3d0d773d60d0 [term 1 LEADER]: Raft consensus shutting down.
I20251215 21:03:21.958985 8101 raft_consensus.cc:2272] T 50e91d6cd81c423da372c9438daf956d P c2cdb9aa2aa948ea980b3d0d773d60d0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251215 21:03:21.960342 8101 ts_tablet_manager.cc:1916] T 50e91d6cd81c423da372c9438daf956d P c2cdb9aa2aa948ea980b3d0d773d60d0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251215 21:03:21.961941 8100 tablet_replica.cc:333] T 50e91d6cd81c423da372c9438daf956d P 3432246dc9284624876b603d75b91697: stopping tablet replica
W20251215 21:03:21.962011 7578 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.28.113.1:41735 (error 108)
I20251215 21:03:21.962188 8100 raft_consensus.cc:2243] T 50e91d6cd81c423da372c9438daf956d P 3432246dc9284624876b603d75b91697 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251215 21:03:21.962491 8100 raft_consensus.cc:2272] T 50e91d6cd81c423da372c9438daf956d P 3432246dc9284624876b603d75b91697 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251215 21:03:21.962736 29124 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu with pid 7792
I20251215 21:03:21.964049 8070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:03:21.963096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.113.1:41735 (local address 127.0.0.1:52580)
1215 21:03:21.963865 (+ 769us) negotiation.cc:107] Waiting for socket to connect
1215 21:03:21.963931 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.113.1:41735: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":680}
I20251215 21:03:21.964186 8101 ts_tablet_manager.cc:1929] T 50e91d6cd81c423da372c9438daf956d P c2cdb9aa2aa948ea980b3d0d773d60d0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
W20251215 21:03:21.964198 7578 catalog_manager.cc:4712] TS e6b9743daabe409084b61f874bd0cb25 (127.28.113.1:41735): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 50e91d6cd81c423da372c9438daf956d: Network error: Client connection negotiation failed: client connection to 127.28.113.1:41735: connect: Connection refused (error 111)
I20251215 21:03:21.964280 8101 log.cc:1199] T 50e91d6cd81c423da372c9438daf956d P c2cdb9aa2aa948ea980b3d0d773d60d0: Deleting WAL directory at /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765832506145157-29124-0/minicluster-data/ts-2/wal/wals/50e91d6cd81c423da372c9438daf956d
I20251215 21:03:21.964586 8101 ts_tablet_manager.cc:1950] T 50e91d6cd81c423da372c9438daf956d P c2cdb9aa2aa948ea980b3d0d773d60d0: Deleting consensus metadata
I20251215 21:03:21.965597 7576 catalog_manager.cc:4985] TS c2cdb9aa2aa948ea980b3d0d773d60d0 (127.28.113.3:43437): tablet 50e91d6cd81c423da372c9438daf956d (table test-table [id=c89ac7418a684f648f347304c496a0b9]) successfully deleted
I20251215 21:03:21.968438 29124 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu with pid 7928
I20251215 21:03:21.969192 8070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:03:21.968619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.113.2:40065 (local address 127.0.0.1:53702)
1215 21:03:21.969073 (+ 454us) negotiation.cc:107] Waiting for socket to connect
1215 21:03:21.969119 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.113.2:40065: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":380}
I20251215 21:03:21.973224 29124 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu with pid 7560
2025-12-15T21:03:21Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (3169 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765832506145157-29124-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Dec 15 21:03:22 dist-test-slave-bjsf krb5kdc[8105](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Dec 15 21:03:22 dist-test-slave-bjsf krb5kdc[8105](info): set up 2 sockets
Dec 15 21:03:22 dist-test-slave-bjsf krb5kdc[8105](info): commencing operation
krb5kdc: starting...
W20251215 21:03:23.812449 29124 mini_kdc.cc:121] Time spent starting KDC: real 1.819s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765832506145157-29124-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765832506145157-29124-0/krb5kdc/test-user.keytab.
Dec 15 21:03:23 dist-test-slave-bjsf krb5kdc[8105](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765832603, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-12-15T21:03:23Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-12-15T21:03:23Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.28.113.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.28.113.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.28.113.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765832506145157-29124-0/krb5kdc/rangeradmin_127.28.113.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.28.113.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765832506145157-29124-0/krb5kdc/rangeradmin_127.28.113.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.28.113.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.28.113.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.28.113.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765832506145157-29124-0/krb5kdc/rangerlookup_127.28.113.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.28.113.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765832506145157-29124-0/krb5kdc/rangerlookup_127.28.113.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.28.113.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.28.113.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.28.113.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765832506145157-29124-0/krb5kdc/HTTP_127.28.113.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.28.113.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765832506145157-29124-0/krb5kdc/HTTP_127.28.113.20@KRBTEST.COM.keytab.
I20251215 21:03:23.956593 29124 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765832506145157-29124-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default "max_connections" ... 100
selecting default "shared_buffers" ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
initdb: hint: You can change this by editing pg_hba.conf or using the option -A, or --auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-task4Kz2CZ/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765832506145157-29124-0/minicluster-data/postgres -l logfile start
2025-12-15 21:03:26.180 UTC [8136] LOG: starting PostgreSQL 17.2 on x86_64-pc-linux-gnu, compiled by gcc (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0, 64-bit
2025-12-15 21:03:26.180 UTC [8136] LOG: listening on IPv4 address "127.28.113.20", port 34593
2025-12-15 21:03:26.184 UTC [8136] LOG: listening on Unix socket "/tmp/.s.PGSQL.34593"
2025-12-15 21:03:26.189 UTC [8141] LOG: database system was shut down at 2025-12-15 21:03:25 UTC
2025-12-15 21:03:26.193 UTC [8136] LOG: database system is ready to accept connections
I20251215 21:03:27.936157 29124 mini_postgres.cc:96] Postgres bound to 34593
2025-12-15 21:03:27.941 UTC [8147] FATAL: database "slave" does not exist
127.28.113.20:34593 - accepting connections
I20251215 21:03:27.942297 29124 mini_ranger.cc:161] Starting Ranger...
I20251215 21:03:27.959270 29124 mini_ranger.cc:84] Created miniranger Postgres user
I20251215 21:03:28.017045 29124 mini_ranger.cc:87] Created ranger Postgres database
I20251215 21:03:28.017151 29124 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765832506145157-29124-0/minicluster-data/ranger-admin
2025-12-15 21:03:28,297 [I] DB FLAVOR :POSTGRES
2025-12-15 21:03:28,297 [I] --------- Verifying Ranger DB connection ---------
2025-12-15 21:03:28,297 [I] Checking connection..
2025-12-15 21:03:28,297 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4Kz2CZ/build/debug/bin/postgresql.jar:/tmp/dist-test-task4Kz2CZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.113.20:34593/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2025-12-15 21:03:28,605 [I] Checking connection passed.
2025-12-15 21:03:28,605 [I] --------- Verifying version history table ---------
2025-12-15 21:03:28,605 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4Kz2CZ/build/debug/bin/postgresql.jar:/tmp/dist-test-task4Kz2CZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.113.20:34593/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2025-12-15 21:03:28,926 [I] Table x_db_version_h does not exist in database ranger
2025-12-15 21:03:28,926 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4Kz2CZ/build/debug/bin/postgresql.jar:/tmp/dist-test-task4Kz2CZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.113.20:34593/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2025-12-15 21:03:29,234 [I] Table x_db_version_h does not exist in database ranger
2025-12-15 21:03:29,234 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2025-12-15 21:03:29,234 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4Kz2CZ/build/debug/bin/postgresql.jar:/tmp/dist-test-task4Kz2CZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.113.20:34593/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task4Kz2CZ/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2025-12-15 21:03:29.523 UTC [8214] WARNING: there is no transaction in progress
2025-12-15 21:03:29,528 [I] create_dbversion_catalog.sql file imported successfully
2025-12-15 21:03:29,529 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4Kz2CZ/build/debug/bin/postgresql.jar:/tmp/dist-test-task4Kz2CZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.113.20:34593/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2025-12-15 21:03:29,828 [I] Table x_db_version_h already exists in database 'ranger'
2025-12-15 21:03:29,828 [I] --------- Importing Ranger Core DB Schema ---------
2025-12-15 21:03:29,829 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4Kz2CZ/build/debug/bin/postgresql.jar:/tmp/dist-test-task4Kz2CZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.113.20:34593/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2025-12-15 21:03:30,108 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4Kz2CZ/build/debug/bin/postgresql.jar:/tmp/dist-test-task4Kz2CZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.113.20:34593/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2025-12-15 21:03:30,398 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4Kz2CZ/build/debug/bin/postgresql.jar:/tmp/dist-test-task4Kz2CZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.113.20:34593/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-bjsf.c.gcp-upstream.internal','N') ;"
2025-12-15 21:03:30,712 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4Kz2CZ/build/debug/bin/postgresql.jar:/tmp/dist-test-task4Kz2CZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.113.20:34593/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2025-12-15 21:03:31,036 [I] Table x_portal_user does not exist in database ranger
2025-12-15 21:03:31,036 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4Kz2CZ/build/debug/bin/postgresql.jar:/tmp/dist-test-task4Kz2CZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.113.20:34593/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2025-12-15 21:03:31,350 [I] Table x_policy_ref_group does not exist in database ranger
2025-12-15 21:03:31,350 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4Kz2CZ/build/debug/bin/postgresql.jar:/tmp/dist-test-task4Kz2CZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.113.20:34593/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2025-12-15 21:03:31,651 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4Kz2CZ/build/debug/bin/postgresql.jar:/tmp/dist-test-task4Kz2CZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.113.20:34593/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2025-12-15 21:03:31,947 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2025-12-15 21:03:31,947 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4Kz2CZ/build/debug/bin/postgresql.jar:/tmp/dist-test-task4Kz2CZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.113.20:34593/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task4Kz2CZ/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2025-12-15 21:03:32.727 UTC [8331] WARNING: there is no transaction in progress
2025-12-15 21:03:32.741 UTC [8331] WARNING: there is no transaction in progress
2025-12-15 21:03:32.755 UTC [8331] WARNING: there is no transaction in progress
2025-12-15 21:03:32.768 UTC [8331] WARNING: there is no transaction in progress
2025-12-15 21:03:32.782 UTC [8331] WARNING: there is no transaction in progress
2025-12-15 21:03:32.792 UTC [8331] WARNING: there is no transaction in progress
2025-12-15 21:03:32.835 UTC [8331] WARNING: there is no transaction in progress
2025-12-15 21:03:32.840 UTC [8331] WARNING: there is no transaction in progress
2025-12-15 21:03:32.849 UTC [8331] WARNING: there is no transaction in progress
2025-12-15 21:03:32.857 UTC [8331] WARNING: there is no transaction in progress
2025-12-15 21:03:32.866 UTC [8331] WARNING: there is no transaction in progress
2025-12-15 21:03:32.876 UTC [8331] WARNING: there is no transaction in progress
2025-12-15 21:03:32.883 UTC [8331] WARNING: there is no transaction in progress
2025-12-15 21:03:32.886 UTC [8331] WARNING: there is no transaction in progress
2025-12-15 21:03:32.889 UTC [8331] WARNING: there is no transaction in progress
2025-12-15 21:03:32.892 UTC [8331] WARNING: there is no transaction in progress
2025-12-15 21:03:33,238 [I] ranger_core_db_postgres.sql file imported successfully
2025-12-15 21:03:33,239 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4Kz2CZ/build/debug/bin/postgresql.jar:/tmp/dist-test-task4Kz2CZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.113.20:34593/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2025-12-15 21:03:33,556 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2025-12-15 21:03:33,556 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4Kz2CZ/build/debug/bin/postgresql.jar:/tmp/dist-test-task4Kz2CZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.113.20:34593/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2025-12-15 21:03:33,882 [I] Table x_portal_user already exists in database 'ranger'
2025-12-15 21:03:33,882 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4Kz2CZ/build/debug/bin/postgresql.jar:/tmp/dist-test-task4Kz2CZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.113.20:34593/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2025-12-15 21:03:34,189 [I] Table x_policy_ref_group already exists in database 'ranger'
2025-12-15 21:03:34,189 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4Kz2CZ/build/debug/bin/postgresql.jar:/tmp/dist-test-task4Kz2CZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.113.20:34593/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2025-12-15 21:03:34,501 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4Kz2CZ/build/debug/bin/postgresql.jar:/tmp/dist-test-task4Kz2CZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.113.20:34593/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2025-12-15 21:03:34,826 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4Kz2CZ/build/debug/bin/postgresql.jar:/tmp/dist-test-task4Kz2CZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.113.20:34593/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-bjsf.c.gcp-upstream.internal';"
2025-12-15 21:03:35,138 [I] CORE_DB_SCHEMA import status has been updated
2025-12-15 21:03:35,139 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4Kz2CZ/build/debug/bin/postgresql.jar:/tmp/dist-test-task4Kz2CZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.113.20:34593/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2025-12-15 21:03:35,441 [I] DB_PATCHES have already been applied
I20251215 21:03:35.446738 29124 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765832506145157-29124-0/minicluster-data/ranger-admin:/tmp/dist-test-task4Kz2CZ/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-task4Kz2CZ/thirdparty/src/hadoop-3.4.1/*:/tmp/dist-test-task4Kz2CZ/build/debug/bin/postgresql.jar:/tmp/dist-test-task4Kz2CZ/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20251215 21:03:35.446808 29124 mini_ranger.cc:193] Using host: 127.28.113.20
I20251215 21:03:35.449553 29124 mini_ranger.cc:227] Ranger admin URL: http://127.28.113.20:49361
Dec 15, 2025 9:03:35 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Dec 15, 2025 9:03:35 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Dec 15, 2025 9:03:35 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765832506145157-29124-0/minicluster-data/ranger-admin/ews/webapp
Dec 15, 2025 9:03:35 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765832506145157-29124-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Dec 15, 2025 9:03:35 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765832506145157-29124-0/minicluster-data/ranger-admin/ews/webapp] .....
Dec 15, 2025 9:03:35 PM org.apache.catalina.core.StandardContext setPath
WARNING: A context path must either be an empty string or start with a '/' and do not end with a '/'. The path [/] does not meet these criteria and has been changed to []
Dec 15, 2025 9:03:36 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765832506145157-29124-0/minicluster-data/ranger-admin/ews/webapp].
Dec 15, 2025 9:03:36 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
Dec 15, 2025 9:03:36 PM org.apache.coyote.AbstractProtocol init
INFO: Initializing ProtocolHandler ["http-bio-49361"]
Dec 15, 2025 9:03:36 PM org.apache.catalina.core.StandardService startInternal
INFO: Starting service [Tomcat]
Dec 15, 2025 9:03:36 PM org.apache.catalina.core.StandardEngine startInternal
INFO: Starting Servlet Engine: Apache Tomcat/7.0.105
Dec 15, 2025 9:03:36 PM org.apache.catalina.loader.WebappClassLoaderBase validateJarFile
INFO: validateJarFile(/tmp/dist-test-task4Kz2CZ/thirdparty/src/ranger-2.1.0-admin/ews/webapp/WEB-INF/lib/javax.servlet-api-3.1.0.jar) - jar not loaded. See Servlet Spec 3.0, section 10.7.2. Offending class: javax/servlet/Servlet.class
Dec 15, 2025 9:03:36 PM org.apache.catalina.loader.WebappClassLoaderBase validateJarFile
INFO: validateJarFile(/tmp/dist-test-task4Kz2CZ/thirdparty/src/ranger-2.1.0-admin/ews/webapp/WEB-INF/lib/tomcat-el-api-7.0.105.jar) - jar not loaded. See Servlet Spec 3.0, section 10.7.2. Offending class: javax/el/Expression.class
Dec 15, 2025 9:03:36 PM org.apache.catalina.startup.ContextConfig getDefaultWebXmlFragment
INFO: No global web.xml found
Dec 15, 2025 9:03:37 PM org.apache.catalina.startup.TaglibUriRule body
INFO: TLD skipped. URI: urn:com:sun:jersey:api:view is already defined
Dec 15, 2025 9:03:37 PM org.apache.catalina.startup.TldConfig execute
INFO: At least one JAR was scanned for TLDs yet contained no TLDs. Enable debug logging for this logger for a complete list of JARs that were scanned but no TLDs were found in them. Skipping unneeded JARs during scanning can improve startup time and JSP compilation time.
Dec 15, 2025 9:03:37 PM org.apache.catalina.core.ApplicationContext log
INFO: Set web app root system property: 'webapp.root' = [/tmp/dist-test-task4Kz2CZ/thirdparty/src/ranger-2.1.0-admin/ews/webapp/]
I20251215 21:03:37.435678 29124 mini_ranger.cc:160] Time spent starting Ranger: real 9.493s user 0.001s sys 0.009s
Dec 15, 2025 9:03:37 PM org.apache.catalina.core.ApplicationContext log
INFO: Initializing Spring root WebApplicationContext
Dec 15, 2025 9:03:55 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Dec 15, 2025 9:03:55 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
Dec 15, 2025 9:03:55 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Dec 15, 2025 9:03:55 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Dec 15, 2025 9:03:55 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Dec 15, 2025 9:03:55 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Dec 15, 2025 9:03:55 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Dec 15, 2025 9:03:55 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Dec 15, 2025 9:03:55 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Dec 15, 2025 9:03:55 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Dec 15, 2025 9:03:55 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Dec 15, 2025 9:03:55 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Dec 15, 2025 9:03:55 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Dec 15, 2025 9:03:55 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Dec 15, 2025 9:03:55 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Dec 15, 2025 9:03:55 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Dec 15, 2025 9:03:55 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Dec 15, 2025 9:03:56 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
Dec 15, 2025 9:03:56 PM org.apache.coyote.AbstractProtocol start
INFO: Starting ProtocolHandler ["http-bio-49361"]
I20251215 21:03:56.766721 29124 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.28.113.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.28.113.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.28.113.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765832506145157-29124-0/krb5kdc/rangerkms_127.28.113.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.28.113.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765832506145157-29124-0/krb5kdc/rangerkms_127.28.113.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.28.113.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.28.113.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.28.113.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765832506145157-29124-0/krb5kdc/HTTP_127.28.113.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.28.113.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765832506145157-29124-0/krb5kdc/HTTP_127.28.113.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Dec 15 21:03:56 dist-test-slave-bjsf krb5kdc[8105](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765832636, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20251215 21:03:56.837589 29124 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20251215 21:03:56.846834 29124 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20251215 21:03:56.942050 29124 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20251215 21:03:56.942160 29124 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765832506145157-29124-0/minicluster-data/ranger-kms
I20251215 21:03:56.942199 29124 mini_ranger_kms.cc:227] Using postgres at 127.28.113.20:34593
2025-12-15 21:03:57,132 [I] DB FLAVOR :POSTGRES
2025-12-15 21:03:57,133 [I] --------- Verifying Ranger DB connection ---------
2025-12-15 21:03:57,133 [I] Checking connection
2025-12-15 21:03:57,133 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4Kz2CZ/build/debug/bin/postgresql.jar:/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765832506145157-29124-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.113.20:34593/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2025-12-15 21:03:57,457 [I] connection success
2025-12-15 21:03:57,457 [I] --------- Verifying Ranger DB tables ---------
2025-12-15 21:03:57,457 [I] Verifying table ranger_masterkey in database rangerkms
2025-12-15 21:03:57,457 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4Kz2CZ/build/debug/bin/postgresql.jar:/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765832506145157-29124-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.113.20:34593/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2025-12-15 21:03:57,748 [I] Table ranger_masterkey does not exist in database rangerkms
2025-12-15 21:03:57,748 [I] --------- Importing Ranger Core DB Schema ---------
2025-12-15 21:03:57,748 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2025-12-15 21:03:57,748 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4Kz2CZ/build/debug/bin/postgresql.jar:/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765832506145157-29124-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.113.20:34593/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765832506145157-29124-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2025-12-15 21:03:58,070 [I] kms_core_db_postgres.sql DB schema imported successfully
I20251215 21:03:58.554355 29124 mini_ranger_kms.cc:310] Created kms service
I20251215 21:03:58.744983 29124 mini_ranger_kms.cc:326] Created kudu user
I20251215 21:03:58.785745 29124 mini_ranger_kms.cc:343] Created rangerkms user
Dec 15 21:03:58 dist-test-slave-bjsf krb5kdc[8105](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765832636, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.28.113.20@KRBTEST.COM
I20251215 21:03:59.167156 29124 mini_ranger_kms.cc:384] Added ranger policy
I20251215 21:03:59.167344 29124 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765832506145157-29124-0/minicluster-data/ranger-kms:/tmp/dist-test-task4Kz2CZ/build/debug/bin/postgresql.jar:/tmp/dist-test-task4Kz2CZ/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-task4Kz2CZ/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20251215 21:03:59.167368 29124 mini_ranger_kms.cc:240] Using host: 127.28.113.20
I20251215 21:03:59.170449 29124 mini_ranger_kms.cc:276] Ranger KMS PID: 8523
I20251215 21:03:59.170549 29124 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.28.113.20:46169
Dec 15, 2025 9:04:00 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Dec 15, 2025 9:04:00 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Dec 15, 2025 9:04:00 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765832506145157-29124-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Dec 15, 2025 9:04:00 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765832506145157-29124-0/minicluster-data/ranger-kms/ews/webapp] .....
Dec 15, 2025 9:04:00 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765832506145157-29124-0/minicluster-data/ranger-kms/ews/webapp].
Dec 15, 2025 9:04:00 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Dec 15, 2025 9:04:00 PM org.apache.coyote.AbstractProtocol init
INFO: Initializing ProtocolHandler ["http-bio-46169"]
Dec 15, 2025 9:04:00 PM org.apache.catalina.core.StandardService startInternal
INFO: Starting service [Tomcat]
Dec 15, 2025 9:04:00 PM org.apache.catalina.core.StandardEngine startInternal
INFO: Starting Servlet Engine: Apache Tomcat/7.0.105
Dec 15, 2025 9:04:00 PM org.apache.catalina.startup.ContextConfig getDefaultWebXmlFragment
INFO: No global web.xml found
I20251215 21:04:01.204145 29124 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 4.367s user 0.020s sys 0.155s
I20251215 21:04:01.204286 29124 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20251215 21:04:01.204383 29124 mini_ranger_kms.cc:401] 127.28.113.20:46169/kms/v1/keys
Dec 15, 2025 9:04:01 PM org.apache.catalina.startup.TldConfig execute
INFO: At least one JAR was scanned for TLDs yet contained no TLDs. Enable debug logging for this logger for a complete list of JARs that were scanned but no TLDs were found in them. Skipping unneeded JARs during scanning can improve startup time and JSP compilation time.
Dec 15 21:04:01 dist-test-slave-bjsf krb5kdc[8105](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1765832641, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.28.113.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Dec 15 21:04:01 dist-test-slave-bjsf krb5kdc[8105](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765832641, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.28.113.20@KRBTEST.COM for HTTP/127.28.113.20@KRBTEST.COM
Dec 15 21:04:02 dist-test-slave-bjsf krb5kdc[8105](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765832641, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.28.113.20@KRBTEST.COM for HTTP/127.28.113.20@KRBTEST.COM
Dec 15 21:04:04 dist-test-slave-bjsf krb5kdc[8105](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765832644, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.28.113.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.113.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.113.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765832506145157-29124-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.28.113.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765832506145157-29124-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.28.113.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.28.113.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.28.113.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765832506145157-29124-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.28.113.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765832506145157-29124-0/krb5kdc/kudu.keytab.
I20251215 21:04:04.362807 29124 external_mini_cluster.cc:1366] Running /tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu
/tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765832506145157-29124-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765832506145157-29124-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765832506145157-29124-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765832506145157-29124-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.113.62:44055
--webserver_interface=127.28.113.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765832506145157-29124-0/krb5kdc/kudu.keytab
--principal=kudu/127.28.113.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.28.113.20:46035
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.113.62:44055
--ranger_config_path=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765832506145157-29124-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.28.113.20:46169/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765832506145157-29124-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765832506145157-29124-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765832506145157-29124-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251215 21:04:04.482353 8576 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251215 21:04:04.482587 8576 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251215 21:04:04.482641 8576 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251215 21:04:04.485872 8576 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251215 21:04:04.485932 8576 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251215 21:04:04.485954 8576 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251215 21:04:04.485972 8576 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251215 21:04:04.486011 8576 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251215 21:04:04.490307 8576 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.113.20:46035
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765832506145157-29124-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765832506145157-29124-0/minicluster-data/master-0/wal
--ranger_kms_url=127.28.113.20:46169/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.28.113.62:44055
--ranger_config_path=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765832506145157-29124-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765832506145157-29124-0/krb5kdc/kudu.keytab
--principal=kudu/127.28.113.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.113.62:44055
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765832506145157-29124-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.113.62
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8576
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765832506145157-29124-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision dc741732bb6a40213e11a13869fad06a49a209a3
build type DEBUG
built by None at 15 Dec 2025 20:43:14 UTC on 24a791456cd2
build id 9530
I20251215 21:04:04.491318 8576 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251215 21:04:04.492179 8576 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251215 21:04:04.497602 8584 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251215 21:04:04.497596 8581 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251215 21:04:04.497622 8582 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251215 21:04:04.498118 8576 server_base.cc:1047] running on GCE node
I20251215 21:04:04.498589 8576 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251215 21:04:04.499415 8576 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251215 21:04:04.500550 8576 hybrid_clock.cc:648] HybridClock initialized: now 1765832644500535 us; error 39 us; skew 500 ppm
Dec 15 21:04:04 dist-test-slave-bjsf krb5kdc[8105](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765832644, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.113.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251215 21:04:04.502985 8576 init.cc:377] Logged in from keytab as kudu/127.28.113.62@KRBTEST.COM (short username kudu)
I20251215 21:04:04.503973 8576 webserver.cc:492] Webserver started at http://127.28.113.62:37365/ using document root <none> and password file <none>
I20251215 21:04:04.504484 8576 fs_manager.cc:362] Metadata directory not provided
I20251215 21:04:04.504532 8576 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251215 21:04:04.504701 8576 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
Dec 15 21:04:04 dist-test-slave-bjsf krb5kdc[8105](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765832644, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.113.62@KRBTEST.COM for HTTP/127.28.113.20@KRBTEST.COM
I20251215 21:04:04.550082 8576 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765832506145157-29124-0/minicluster-data/master-0/data/instance:
uuid: "9c00e5c4afd343249484cad9d76eee0d"
format_stamp: "Formatted at 2025-12-15 21:04:04 on dist-test-slave-bjsf"
server_key: "842dc6ca172445d9f3bf9bb9db4e5108"
server_key_iv: "07b0f5583d159dcbe034056d483a3e08"
server_key_version: "kuduclusterkey@0"
I20251215 21:04:04.550729 8576 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765832506145157-29124-0/minicluster-data/master-0/wal/instance:
uuid: "9c00e5c4afd343249484cad9d76eee0d"
format_stamp: "Formatted at 2025-12-15 21:04:04 on dist-test-slave-bjsf"
server_key: "842dc6ca172445d9f3bf9bb9db4e5108"
server_key_iv: "07b0f5583d159dcbe034056d483a3e08"
server_key_version: "kuduclusterkey@0"
I20251215 21:04:04.555192 8576 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20251215 21:04:04.572057 8591 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251215 21:04:04.573297 8576 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20251215 21:04:04.573431 8576 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765832506145157-29124-0/minicluster-data/master-0/data,/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765832506145157-29124-0/minicluster-data/master-0/wal
uuid: "9c00e5c4afd343249484cad9d76eee0d"
format_stamp: "Formatted at 2025-12-15 21:04:04 on dist-test-slave-bjsf"
server_key: "842dc6ca172445d9f3bf9bb9db4e5108"
server_key_iv: "07b0f5583d159dcbe034056d483a3e08"
server_key_version: "kuduclusterkey@0"
I20251215 21:04:04.573563 8576 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765832506145157-29124-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765832506145157-29124-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765832506145157-29124-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251215 21:04:04.602445 8576 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251215 21:04:04.606132 8576 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251215 21:04:04.606331 8576 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251215 21:04:04.615257 8643 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.28.113.62:44055 every 8 connection(s)
I20251215 21:04:04.615278 8576 rpc_server.cc:307] RPC server started. Bound to: 127.28.113.62:44055
I20251215 21:04:04.616549 8576 server_base.cc:1179] Dumped server information to /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765832506145157-29124-0/minicluster-data/master-0/data/info.pb
I20251215 21:04:04.619194 8644 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251215 21:04:04.619648 29124 external_mini_cluster.cc:1428] Started /tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu as pid 8576
I20251215 21:04:04.619781 29124 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765832506145157-29124-0/minicluster-data/master-0/wal/instance
I20251215 21:04:04.620067 29124 external_mini_cluster.cc:1468] Setting key ae07ece03d0e6ff3d995b193f1647b22
I20251215 21:04:04.624778 8644 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9c00e5c4afd343249484cad9d76eee0d: Bootstrap starting.
I20251215 21:04:04.627156 8644 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9c00e5c4afd343249484cad9d76eee0d: Neither blocks nor log segments found. Creating new log.
I20251215 21:04:04.628042 8644 log.cc:826] T 00000000000000000000000000000000 P 9c00e5c4afd343249484cad9d76eee0d: Log is configured to *not* fsync() on all Append() calls
Dec 15 21:04:04 dist-test-slave-bjsf krb5kdc[8105](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765832644, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.113.62@KRBTEST.COM
I20251215 21:04:04.630293 8644 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9c00e5c4afd343249484cad9d76eee0d: No bootstrap required, opened a new log
I20251215 21:04:04.633417 8644 raft_consensus.cc:359] T 00000000000000000000000000000000 P 9c00e5c4afd343249484cad9d76eee0d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c00e5c4afd343249484cad9d76eee0d" member_type: VOTER last_known_addr { host: "127.28.113.62" port: 44055 } }
I20251215 21:04:04.633615 8644 raft_consensus.cc:385] T 00000000000000000000000000000000 P 9c00e5c4afd343249484cad9d76eee0d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251215 21:04:04.633677 8644 raft_consensus.cc:740] T 00000000000000000000000000000000 P 9c00e5c4afd343249484cad9d76eee0d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c00e5c4afd343249484cad9d76eee0d, State: Initialized, Role: FOLLOWER
I20251215 21:04:04.634038 8647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:04:04.621310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57228 (local address 127.28.113.62:44055)
1215 21:04:04.621736 (+ 426us) server_negotiation.cc:207] Beginning negotiation
1215 21:04:04.621748 (+ 12us) server_negotiation.cc:400] Waiting for connection header
1215 21:04:04.621787 (+ 39us) server_negotiation.cc:408] Connection header received
1215 21:04:04.622460 (+ 673us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1215 21:04:04.622493 (+ 33us) server_negotiation.cc:462] Received NEGOTIATE request from client
1215 21:04:04.622873 (+ 380us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1215 21:04:04.623225 (+ 352us) server_negotiation.cc:227] Negotiated authn=SASL
1215 21:04:04.624254 (+ 1029us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:04:04.625517 (+ 1263us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1215 21:04:04.626361 (+ 844us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:04:04.626648 (+ 287us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:04:04.629179 (+ 2531us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1215 21:04:04.629201 (+ 22us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1215 21:04:04.629216 (+ 15us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1215 21:04:04.629259 (+ 43us) server_negotiation.cc:923] calling sasl_server_start()
1215 21:04:04.631250 (+ 1991us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1215 21:04:04.631708 (+ 458us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1215 21:04:04.631714 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1215 21:04:04.631720 (+ 6us) server_negotiation.cc:968] calling sasl_server_step()
1215 21:04:04.631800 (+ 80us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1215 21:04:04.632049 (+ 249us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1215 21:04:04.632052 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1215 21:04:04.632054 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1215 21:04:04.632456 (+ 402us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1215 21:04:04.632673 (+ 217us) server_negotiation.cc:1036] Waiting for connection context
1215 21:04:04.632923 (+ 250us) server_negotiation.cc:300] Negotiation successful
1215 21:04:04.633287 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":277,"thread_start_us":130,"threads_started":1}
I20251215 21:04:04.634097 8644 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9c00e5c4afd343249484cad9d76eee0d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c00e5c4afd343249484cad9d76eee0d" member_type: VOTER last_known_addr { host: "127.28.113.62" port: 44055 } }
I20251215 21:04:04.634238 8644 raft_consensus.cc:399] T 00000000000000000000000000000000 P 9c00e5c4afd343249484cad9d76eee0d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251215 21:04:04.634286 8644 raft_consensus.cc:493] T 00000000000000000000000000000000 P 9c00e5c4afd343249484cad9d76eee0d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251215 21:04:04.634359 8644 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 9c00e5c4afd343249484cad9d76eee0d [term 0 FOLLOWER]: Advancing to term 1
I20251215 21:04:04.635159 8644 raft_consensus.cc:515] T 00000000000000000000000000000000 P 9c00e5c4afd343249484cad9d76eee0d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c00e5c4afd343249484cad9d76eee0d" member_type: VOTER last_known_addr { host: "127.28.113.62" port: 44055 } }
I20251215 21:04:04.635510 8644 leader_election.cc:304] T 00000000000000000000000000000000 P 9c00e5c4afd343249484cad9d76eee0d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9c00e5c4afd343249484cad9d76eee0d; no voters:
I20251215 21:04:04.635802 8644 leader_election.cc:290] T 00000000000000000000000000000000 P 9c00e5c4afd343249484cad9d76eee0d [CANDIDATE]: Term 1 election: Requested vote from peers
I20251215 21:04:04.635993 8649 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 9c00e5c4afd343249484cad9d76eee0d [term 1 FOLLOWER]: Leader election won for term 1
I20251215 21:04:04.636440 8649 raft_consensus.cc:697] T 00000000000000000000000000000000 P 9c00e5c4afd343249484cad9d76eee0d [term 1 LEADER]: Becoming Leader. State: Replica: 9c00e5c4afd343249484cad9d76eee0d, State: Running, Role: LEADER
I20251215 21:04:04.636781 8649 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9c00e5c4afd343249484cad9d76eee0d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c00e5c4afd343249484cad9d76eee0d" member_type: VOTER last_known_addr { host: "127.28.113.62" port: 44055 } }
I20251215 21:04:04.636879 8644 sys_catalog.cc:565] T 00000000000000000000000000000000 P 9c00e5c4afd343249484cad9d76eee0d [sys.catalog]: configured and running, proceeding with master startup.
I20251215 21:04:04.637897 8649 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9c00e5c4afd343249484cad9d76eee0d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9c00e5c4afd343249484cad9d76eee0d. Latest consensus state: current_term: 1 leader_uuid: "9c00e5c4afd343249484cad9d76eee0d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c00e5c4afd343249484cad9d76eee0d" member_type: VOTER last_known_addr { host: "127.28.113.62" port: 44055 } } }
I20251215 21:04:04.638074 8649 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9c00e5c4afd343249484cad9d76eee0d [sys.catalog]: This master's current role is: LEADER
I20251215 21:04:04.638238 8651 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9c00e5c4afd343249484cad9d76eee0d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9c00e5c4afd343249484cad9d76eee0d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c00e5c4afd343249484cad9d76eee0d" member_type: VOTER last_known_addr { host: "127.28.113.62" port: 44055 } } }
I20251215 21:04:04.638327 8651 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9c00e5c4afd343249484cad9d76eee0d [sys.catalog]: This master's current role is: LEADER
I20251215 21:04:04.638437 8658 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251215 21:04:04.641698 8658 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251215 21:04:04.642370 8644 ranger_client.cc:318] Using new properties file: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765832506145157-29124-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20251215 21:04:04.647763 8658 catalog_manager.cc:1357] Generated new cluster ID: 065d7f4eed1248e5821868c234688cd1
I20251215 21:04:04.647830 8658 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251215 21:04:04.670286 8658 catalog_manager.cc:1380] Generated new certificate authority record
I20251215 21:04:04.671664 8658 catalog_manager.cc:1514] Loading token signing keys...
I20251215 21:04:04.680859 8658 catalog_manager.cc:6027] T 00000000000000000000000000000000 P 9c00e5c4afd343249484cad9d76eee0d: Generated new TSK 0
I20251215 21:04:04.681824 8658 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.113.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.113.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.113.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765832506145157-29124-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.28.113.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765832506145157-29124-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.28.113.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.28.113.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.28.113.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765832506145157-29124-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.28.113.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765832506145157-29124-0/krb5kdc/kudu.keytab.
I20251215 21:04:05.416141 29124 external_mini_cluster.cc:1366] Running /tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu
/tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765832506145157-29124-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765832506145157-29124-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765832506145157-29124-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765832506145157-29124-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.28.113.20:46169/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.113.1:0
--local_ip_for_outbound_sockets=127.28.113.1
--webserver_interface=127.28.113.1
--webserver_port=0
--tserver_master_addrs=127.28.113.62:44055
--keytab_file=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765832506145157-29124-0/krb5kdc/kudu.keytab
--principal=kudu/127.28.113.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.28.113.20:46035
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765832506145157-29124-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765832506145157-29124-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765832506145157-29124-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251215 21:04:05.525801 8690 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251215 21:04:05.526021 8690 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251215 21:04:05.526096 8690 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251215 21:04:05.529420 8690 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251215 21:04:05.529476 8690 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251215 21:04:05.529570 8690 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.113.1
I20251215 21:04:05.533850 8690 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.113.20:46035
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765832506145157-29124-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765832506145157-29124-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.28.113.20:46169/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765832506145157-29124-0/krb5kdc/kudu.keytab
--principal=kudu/127.28.113.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.113.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765832506145157-29124-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.113.1
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.28.113.62:44055
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8690
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.113.1
--log_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765832506145157-29124-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision dc741732bb6a40213e11a13869fad06a49a209a3
build type DEBUG
built by None at 15 Dec 2025 20:43:14 UTC on 24a791456cd2
build id 9530
I20251215 21:04:05.534816 8690 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251215 21:04:05.535601 8690 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251215 21:04:05.541904 8698 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251215 21:04:05.541891 8697 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251215 21:04:05.541910 8700 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251215 21:04:05.542373 8690 server_base.cc:1047] running on GCE node
I20251215 21:04:05.542691 8690 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251215 21:04:05.543205 8690 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251215 21:04:05.544360 8690 hybrid_clock.cc:648] HybridClock initialized: now 1765832645544337 us; error 41 us; skew 500 ppm
Dec 15 21:04:05 dist-test-slave-bjsf krb5kdc[8105](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765832645, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.113.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251215 21:04:05.546802 8690 init.cc:377] Logged in from keytab as kudu/127.28.113.1@KRBTEST.COM (short username kudu)
I20251215 21:04:05.547859 8690 webserver.cc:492] Webserver started at http://127.28.113.1:42857/ using document root <none> and password file <none>
I20251215 21:04:05.548421 8690 fs_manager.cc:362] Metadata directory not provided
I20251215 21:04:05.548470 8690 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251215 21:04:05.548635 8690 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
Dec 15 21:04:05 dist-test-slave-bjsf krb5kdc[8105](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765832645, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.113.1@KRBTEST.COM for HTTP/127.28.113.20@KRBTEST.COM
I20251215 21:04:05.565218 8690 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765832506145157-29124-0/minicluster-data/ts-0/data/instance:
uuid: "89b2c368db8b4b51bf1733a3b934c011"
format_stamp: "Formatted at 2025-12-15 21:04:05 on dist-test-slave-bjsf"
server_key: "386da84805de3bb04ab0f9f06c32bab3"
server_key_iv: "c1b5e47f0ae68dad22436875dbfeec71"
server_key_version: "kuduclusterkey@0"
I20251215 21:04:05.566543 8690 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765832506145157-29124-0/minicluster-data/ts-0/wal/instance:
uuid: "89b2c368db8b4b51bf1733a3b934c011"
format_stamp: "Formatted at 2025-12-15 21:04:05 on dist-test-slave-bjsf"
server_key: "386da84805de3bb04ab0f9f06c32bab3"
server_key_iv: "c1b5e47f0ae68dad22436875dbfeec71"
server_key_version: "kuduclusterkey@0"
I20251215 21:04:05.571029 8690 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20251215 21:04:05.587280 8707 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251215 21:04:05.588656 8690 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20251215 21:04:05.588780 8690 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765832506145157-29124-0/minicluster-data/ts-0/data,/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765832506145157-29124-0/minicluster-data/ts-0/wal
uuid: "89b2c368db8b4b51bf1733a3b934c011"
format_stamp: "Formatted at 2025-12-15 21:04:05 on dist-test-slave-bjsf"
server_key: "386da84805de3bb04ab0f9f06c32bab3"
server_key_iv: "c1b5e47f0ae68dad22436875dbfeec71"
server_key_version: "kuduclusterkey@0"
I20251215 21:04:05.588893 8690 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765832506145157-29124-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765832506145157-29124-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765832506145157-29124-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251215 21:04:05.608533 8690 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251215 21:04:05.611912 8690 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251215 21:04:05.612176 8690 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251215 21:04:05.612825 8690 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251215 21:04:05.613925 8690 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251215 21:04:05.614030 8690 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251215 21:04:05.614147 8690 ts_tablet_manager.cc:616] Registered 0 tablets
I20251215 21:04:05.614219 8690 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251215 21:04:05.626078 8690 rpc_server.cc:307] RPC server started. Bound to: 127.28.113.1:35867
I20251215 21:04:05.627316 8690 server_base.cc:1179] Dumped server information to /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765832506145157-29124-0/minicluster-data/ts-0/data/info.pb
I20251215 21:04:05.627885 8822 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.28.113.1:35867 every 8 connection(s)
I20251215 21:04:05.630008 29124 external_mini_cluster.cc:1428] Started /tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu as pid 8690
I20251215 21:04:05.630154 29124 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765832506145157-29124-0/minicluster-data/ts-0/wal/instance
I20251215 21:04:05.630427 29124 external_mini_cluster.cc:1468] Setting key 124782622ff4119a609ad3da46189099
Dec 15 21:04:05 dist-test-slave-bjsf krb5kdc[8105](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765832645, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.113.1@KRBTEST.COM for kudu/127.28.113.62@KRBTEST.COM
I20251215 21:04:05.645774 8825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:04:05.629880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.113.1:43009 (local address 127.28.113.62:44055)
1215 21:04:05.630148 (+ 268us) server_negotiation.cc:207] Beginning negotiation
1215 21:04:05.630153 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1215 21:04:05.632211 (+ 2058us) server_negotiation.cc:408] Connection header received
1215 21:04:05.633263 (+ 1052us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1215 21:04:05.633267 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1215 21:04:05.633317 (+ 50us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1215 21:04:05.633369 (+ 52us) server_negotiation.cc:227] Negotiated authn=SASL
1215 21:04:05.634955 (+ 1586us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:04:05.635715 (+ 760us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1215 21:04:05.636624 (+ 909us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1215 21:04:05.636812 (+ 188us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:04:05.639477 (+ 2665us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1215 21:04:05.639498 (+ 21us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1215 21:04:05.639501 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1215 21:04:05.639528 (+ 27us) server_negotiation.cc:923] calling sasl_server_start()
1215 21:04:05.641329 (+ 1801us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1215 21:04:05.641892 (+ 563us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1215 21:04:05.641896 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1215 21:04:05.641898 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1215 21:04:05.641944 (+ 46us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1215 21:04:05.642265 (+ 321us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1215 21:04:05.642268 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1215 21:04:05.642270 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1215 21:04:05.642446 (+ 176us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1215 21:04:05.642512 (+ 66us) server_negotiation.cc:1036] Waiting for connection context
1215 21:04:05.645451 (+ 2939us) server_negotiation.cc:300] Negotiation successful
1215 21:04:05.645590 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":183,"thread_start_us":87,"threads_started":1}
I20251215 21:04:05.647481 8826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:04:05.630749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.113.62:44055 (local address 127.28.113.1:43009)
1215 21:04:05.632094 (+ 1345us) negotiation.cc:107] Waiting for socket to connect
1215 21:04:05.632137 (+ 43us) client_negotiation.cc:174] Beginning negotiation
1215 21:04:05.633046 (+ 909us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1215 21:04:05.633743 (+ 697us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1215 21:04:05.633753 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
1215 21:04:05.634080 (+ 327us) client_negotiation.cc:189] Negotiated authn=SASL
1215 21:04:05.634806 (+ 726us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:04:05.634820 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:04:05.635866 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1215 21:04:05.635869 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1215 21:04:05.636489 (+ 620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1215 21:04:05.636500 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1215 21:04:05.637046 (+ 546us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1215 21:04:05.637713 (+ 667us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1215 21:04:05.637746 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
1215 21:04:05.639309 (+ 1563us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1215 21:04:05.641466 (+ 2157us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1215 21:04:05.641471 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1215 21:04:05.641485 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
1215 21:04:05.641780 (+ 295us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1215 21:04:05.642038 (+ 258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1215 21:04:05.642041 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1215 21:04:05.642044 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
1215 21:04:05.642154 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1215 21:04:05.642589 (+ 435us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1215 21:04:05.642595 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1215 21:04:05.642876 (+ 281us) client_negotiation.cc:769] Sending connection context
1215 21:04:05.646342 (+ 3466us) client_negotiation.cc:240] Negotiation successful
1215 21:04:05.646619 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1112,"thread_start_us":108,"threads_started":1}
I20251215 21:04:05.649137 8823 heartbeater.cc:344] Connected to a master server at 127.28.113.62:44055
I20251215 21:04:05.649405 8823 heartbeater.cc:461] Registering TS with master...
I20251215 21:04:05.650094 8823 heartbeater.cc:507] Master 127.28.113.62:44055 requested a full tablet report, sending...
I20251215 21:04:05.651927 8601 ts_manager.cc:194] Registered new tserver with Master: 89b2c368db8b4b51bf1733a3b934c011 (127.28.113.1:35867)
WARNING: no policy specified for kudu/127.28.113.2@KRBTEST.COM; defaulting to no policy
I20251215 21:04:05.653625 8601 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.113.1@KRBTEST.COM'} at 127.28.113.1:43009
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.113.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.113.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765832506145157-29124-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.28.113.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765832506145157-29124-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.28.113.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.28.113.2@KRBTEST.COM" created.
Exception in thread "main" org.apache.kudu.subprocess.KuduSubprocessException: Failed to login with Kudu principal/keytab
at org.apache.kudu.subprocess.ranger.authorization.RangerKuduAuthorizer.init(RangerKuduAuthorizer.java:87)
at org.apache.kudu.subprocess.ranger.RangerProtocolHandler.<init>(RangerProtocolHandler.java:45)
at org.apache.kudu.subprocess.ranger.RangerSubprocessMain.main(RangerSubprocessMain.java:39)
Caused by: org.apache.hadoop.security.KerberosAuthException: failure to login: for principal: kudu/127.28.113.62 from keytab /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765832506145157-29124-0/krb5kdc/kudu.keytab javax.security.auth.login.LoginException: Unable to obtain password from user
at org.apache.hadoop.security.UserGroupInformation.doSubjectLogin(UserGroupInformation.java:2064)
at org.apache.hadoop.security.UserGroupInformation.loginUserFromKeytabAndReturnUGI(UserGroupInformation.java:1398)
at org.apache.hadoop.security.UserGroupInformation.loginUserFromKeytab(UserGroupInformation.java:1136)
at org.apache.kudu.subprocess.ranger.authorization.RangerKuduAuthorizer.init(RangerKuduAuthorizer.java:85)
... 2 more
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.28.113.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765832506145157-29124-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.28.113.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765832506145157-29124-0/krb5kdc/kudu.keytab.
Caused by: javax.security.auth.login.LoginException: Unable to obtain password from user
at com.sun.security.auth.module.Krb5LoginModule.promptForPass(Krb5LoginModule.java:901)
at com.sun.security.auth.module.Krb5LoginModule.attemptAuthentication(Krb5LoginModule.java:764)
at com.sun.security.auth.module.Krb5LoginModule.login(Krb5LoginModule.java:618)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at javax.security.auth.login.LoginContext.invoke(LoginContext.java:755)
at javax.security.auth.login.LoginContext.access$000(LoginContext.java:195)
at javax.security.auth.login.LoginContext$4.run(LoginContext.java:682)
at javax.security.auth.login.LoginContext$4.run(LoginContext.java:680)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.login.LoginContext.invokePriv(LoginContext.java:680)
at javax.security.auth.login.LoginContext.login(LoginContext.java:587)
at org.apache.hadoop.security.UserGroupInformation$HadoopLoginContext.login(UserGroupInformation.java:2148)
at org.apache.hadoop.security.UserGroupInformation.doSubjectLogin(UserGroupInformation.java:2053)
... 5 more
I20251215 21:04:05.696835 29124 external_mini_cluster.cc:1366] Running /tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu
/tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765832506145157-29124-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765832506145157-29124-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765832506145157-29124-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765832506145157-29124-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.28.113.20:46169/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.113.2:0
--local_ip_for_outbound_sockets=127.28.113.2
--webserver_interface=127.28.113.2
--webserver_port=0
--tserver_master_addrs=127.28.113.62:44055
--keytab_file=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765832506145157-29124-0/krb5kdc/kudu.keytab
--principal=kudu/127.28.113.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.28.113.20:46035
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765832506145157-29124-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765832506145157-29124-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765832506145157-29124-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20251215 21:04:05.702365 8682 server.cc:273] Received an EOF from the subprocess
F20251215 21:04:05.703073 8669 server.cc:406] The subprocess has exited with status 1
*** Check failure stack trace: ***
@ 0x7f543cfd2dcd google::LogMessage::Fail() at ??:0
@ 0x7f543cfd6b93 google::LogMessage::SendToLog() at ??:0
@ 0x7f543cfd27cc google::LogMessage::Flush() at ??:0
@ 0x7f543cfd3f59 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f543ec892d7 kudu::subprocess::SubprocessServer::ExitCheckerThread() at ??:0
@ 0x7f543ec860d9 _ZZN4kudu10subprocess16SubprocessServer4InitEvENKUlvE0_clEv at ??:0
@ 0x7f543ec8a005 _ZNSt17_Function_handlerIFvvEZN4kudu10subprocess16SubprocessServer4InitEvEUlvE0_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x56472029fa0a std::function<>::operator()() at ??:0
@ 0x7f543e42c7dc kudu::Thread::SuperviseThread() at ??:0
@ 0x7f543ecc56db start_thread at ??:0
@ 0x7f543c18971f clone at ??:0
W20251215 21:04:05.831036 8832 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251215 21:04:05.831264 8832 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251215 21:04:05.831317 8832 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251215 21:04:05.834585 8832 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251215 21:04:05.834648 8832 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251215 21:04:05.834734 8832 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.113.2
I20251215 21:04:05.838828 8832 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.113.20:46035
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765832506145157-29124-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765832506145157-29124-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.28.113.20:46169/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765832506145157-29124-0/krb5kdc/kudu.keytab
--principal=kudu/127.28.113.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.113.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765832506145157-29124-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.113.2
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.28.113.62:44055
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8832
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.113.2
--log_dir=/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765832506145157-29124-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision dc741732bb6a40213e11a13869fad06a49a209a3
build type DEBUG
built by None at 15 Dec 2025 20:43:14 UTC on 24a791456cd2
build id 9530
I20251215 21:04:05.839800 8832 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251215 21:04:05.840595 8832 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251215 21:04:05.846695 8848 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251215 21:04:05.846800 8849 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251215 21:04:05.846695 8851 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251215 21:04:05.847170 8832 server_base.cc:1047] running on GCE node
I20251215 21:04:05.847582 8832 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251215 21:04:05.848248 8832 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251215 21:04:05.849431 8832 hybrid_clock.cc:648] HybridClock initialized: now 1765832645849411 us; error 36 us; skew 500 ppm
Dec 15 21:04:05 dist-test-slave-bjsf krb5kdc[8105](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765832645, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.113.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251215 21:04:05.852456 8832 init.cc:377] Logged in from keytab as kudu/127.28.113.2@KRBTEST.COM (short username kudu)
I20251215 21:04:05.853708 8832 webserver.cc:492] Webserver started at http://127.28.113.2:33313/ using document root <none> and password file <none>
I20251215 21:04:05.854380 8832 fs_manager.cc:362] Metadata directory not provided
I20251215 21:04:05.854456 8832 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251215 21:04:05.854673 8832 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
Dec 15 21:04:05 dist-test-slave-bjsf krb5kdc[8105](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765832645, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.113.2@KRBTEST.COM for HTTP/127.28.113.20@KRBTEST.COM
I20251215 21:04:05.867715 8832 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765832506145157-29124-0/minicluster-data/ts-1/data/instance:
uuid: "4291f018dd9a437daecdc994bd4b6be3"
format_stamp: "Formatted at 2025-12-15 21:04:05 on dist-test-slave-bjsf"
server_key: "487a9240c81a21bbc02cd02238fc808f"
server_key_iv: "1f9dd74619bea71c05c0febf1d641551"
server_key_version: "kuduclusterkey@0"
I20251215 21:04:05.868367 8832 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765832506145157-29124-0/minicluster-data/ts-1/wal/instance:
uuid: "4291f018dd9a437daecdc994bd4b6be3"
format_stamp: "Formatted at 2025-12-15 21:04:05 on dist-test-slave-bjsf"
server_key: "487a9240c81a21bbc02cd02238fc808f"
server_key_iv: "1f9dd74619bea71c05c0febf1d641551"
server_key_version: "kuduclusterkey@0"
I20251215 21:04:05.872815 8832 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20251215 21:04:05.888358 8858 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251215 21:04:05.889714 8832 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.002s sys 0.000s
I20251215 21:04:05.889837 8832 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765832506145157-29124-0/minicluster-data/ts-1/data,/tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765832506145157-29124-0/minicluster-data/ts-1/wal
uuid: "4291f018dd9a437daecdc994bd4b6be3"
format_stamp: "Formatted at 2025-12-15 21:04:05 on dist-test-slave-bjsf"
server_key: "487a9240c81a21bbc02cd02238fc808f"
server_key_iv: "1f9dd74619bea71c05c0febf1d641551"
server_key_version: "kuduclusterkey@0"
I20251215 21:04:05.889959 8832 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765832506145157-29124-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765832506145157-29124-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765832506145157-29124-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251215 21:04:05.909607 8832 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251215 21:04:05.913014 8832 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251215 21:04:05.913203 8832 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251215 21:04:05.913859 8832 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251215 21:04:05.914899 8832 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251215 21:04:05.914960 8832 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251215 21:04:05.915009 8832 ts_tablet_manager.cc:616] Registered 0 tablets
I20251215 21:04:05.915038 8832 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251215 21:04:05.928099 8832 rpc_server.cc:307] RPC server started. Bound to: 127.28.113.2:38195
I20251215 21:04:05.928293 8971 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.28.113.2:38195 every 8 connection(s)
I20251215 21:04:05.929360 8832 server_base.cc:1179] Dumped server information to /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765832506145157-29124-0/minicluster-data/ts-1/data/info.pb
I20251215 21:04:05.932669 8974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:04:05.931352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.113.62:44055 (local address 127.28.113.2:35435)
1215 21:04:05.931819 (+ 467us) negotiation.cc:107] Waiting for socket to connect
1215 21:04:05.931935 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.113.62:44055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":93,"threads_started":1}
W20251215 21:04:05.933028 8972 heartbeater.cc:646] Failed to heartbeat to 127.28.113.62:44055 (0 consecutive failures): Network error: Failed to ping master at 127.28.113.62:44055: Client connection negotiation failed: client connection to 127.28.113.62:44055: connect: Connection refused (error 111)
I20251215 21:04:05.933178 29124 external_mini_cluster.cc:1428] Started /tmp/dist-test-task4Kz2CZ/build/debug/bin/kudu as pid 8832
I20251215 21:04:05.933291 29124 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task4Kz2CZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765832506145157-29124-0/minicluster-data/ts-1/wal/instance
I20251215 21:04:05.933543 29124 external_mini_cluster.cc:1468] Setting key 6250b86ae2300b91ea06fa0812d6aaa5
I20251215 21:04:05.933559 8974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:04:05.933368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.113.62:44055 (local address 127.28.113.2:36739)
1215 21:04:05.933470 (+ 102us) negotiation.cc:107] Waiting for socket to connect
1215 21:04:05.933511 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.113.62:44055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20251215 21:04:05.934358 8974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:04:05.933868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.113.62:44055 (local address 127.28.113.2:51157)
1215 21:04:05.934278 (+ 410us) negotiation.cc:107] Waiting for socket to connect
1215 21:04:05.934314 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.113.62:44055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":343}
W20251215 21:04:05.934557 8972 heartbeater.cc:412] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20251215 21:04:06.656687 8823 heartbeater.cc:499] Master 127.28.113.62:44055 was elected leader, sending a full tablet report...
I20251215 21:04:06.657604 8976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:04:06.657178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.113.62:44055 (local address 127.28.113.1:34401)
1215 21:04:06.657432 (+ 254us) negotiation.cc:107] Waiting for socket to connect
1215 21:04:06.657501 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.113.62:44055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":73,"threads_started":1}
W20251215 21:04:06.657843 8823 heartbeater.cc:646] Failed to heartbeat to 127.28.113.62:44055 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.28.113.62:44055: connect: Connection refused (error 111)
I20251215 21:04:06.935516 8977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:04:06.935067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.113.62:44055 (local address 127.28.113.2:38659)
1215 21:04:06.935354 (+ 287us) negotiation.cc:107] Waiting for socket to connect
1215 21:04:06.935415 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.113.62:44055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":69,"threads_started":1}
I20251215 21:04:07.658867 8978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:04:07.658423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.113.62:44055 (local address 127.28.113.1:55897)
1215 21:04:07.658703 (+ 280us) negotiation.cc:107] Waiting for socket to connect
1215 21:04:07.658765 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.113.62:44055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":69,"threads_started":1}
I20251215 21:04:07.936611 8979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:04:07.936146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.113.62:44055 (local address 127.28.113.2:56681)
1215 21:04:07.936442 (+ 296us) negotiation.cc:107] Waiting for socket to connect
1215 21:04:07.936499 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.113.62:44055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":88,"threads_started":1}
I20251215 21:04:08.660112 8980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:04:08.659671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.113.62:44055 (local address 127.28.113.1:57111)
1215 21:04:08.659938 (+ 267us) negotiation.cc:107] Waiting for socket to connect
1215 21:04:08.660013 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.113.62:44055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":86,"threads_started":1}
I20251215 21:04:08.937777 8981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:04:08.937273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.113.62:44055 (local address 127.28.113.2:38233)
1215 21:04:08.937583 (+ 310us) negotiation.cc:107] Waiting for socket to connect
1215 21:04:08.937653 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.113.62:44055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":84,"threads_started":1}
I20251215 21:04:09.661131 8982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:04:09.660742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.113.62:44055 (local address 127.28.113.1:44029)
1215 21:04:09.660979 (+ 237us) negotiation.cc:107] Waiting for socket to connect
1215 21:04:09.661031 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.113.62:44055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":74,"threads_started":1}
I20251215 21:04:09.938989 8983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:04:09.938492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.113.62:44055 (local address 127.28.113.2:58969)
1215 21:04:09.938795 (+ 303us) negotiation.cc:107] Waiting for socket to connect
1215 21:04:09.938865 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.113.62:44055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":70,"threads_started":1}
I20251215 21:04:10.662343 8984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:04:10.661925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.113.62:44055 (local address 127.28.113.1:33577)
1215 21:04:10.662180 (+ 255us) negotiation.cc:107] Waiting for socket to connect
1215 21:04:10.662240 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.113.62:44055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":78,"threads_started":1}
I20251215 21:04:10.940214 8985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1215 21:04:10.939771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.113.62:44055 (local address 127.28.113.2:46967)
1215 21:04:10.940049 (+ 278us) negotiation.cc:107] Waiting for socket to connect
1215 21:04:10.940119 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.113.62:44055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":84,"threads_started":1}
F20251215 21:04:10.960484 29124 external_mini_cluster.cc:650] Check failed: leader_master() != nullptr Must have started at least 1 master before adding tablet servers
*** Check failure stack trace: ***
*** Aborted at 1765832650 (unix time) try "date -d @1765832650" if you are using GNU date ***
PC: @ 0x0 (unknown)
*** SIGABRT (@0x3e8000071c4) received by PID 29124 (TID 0x7fc7881bad80) from PID 29124; stack trace: ***
@ 0x7fc793e60980 (unknown) at ??:0
@ 0x7fc791497fb7 gsignal at ??:0
@ 0x7fc791499921 abort at ??:0
@ 0x7fc792705dcd google::LogMessage::Fail() at ??:0
@ 0x7fc792709b93 google::LogMessage::SendToLog() at ??:0
@ 0x7fc7927057cc google::LogMessage::Flush() at ??:0
@ 0x7fc792706f59 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fc795491cab kudu::cluster::ExternalMiniCluster::AddTabletServer() at ??:0
@ 0x7fc79548e9c6 kudu::cluster::ExternalMiniCluster::Start() at ??:0
@ 0x55652239392f kudu::SecurityITest::StartCluster() at ??:0
@ 0x55652237b792 kudu::SecurityITest_TestEncryptionWithKMSIntegration_Test::TestBody() at ??:0
@ 0x7fc7937150ed testing::internal::HandleExceptionsInMethodIfSupported<>() at ??:0
@ 0x7fc793709bdb testing::Test::Run() at ??:0
@ 0x7fc793709d9d testing::TestInfo::Run() at ??:0
@ 0x7fc79370a377 testing::TestSuite::Run() at ??:0
@ 0x7fc79370a77c testing::internal::UnitTestImpl::RunAllTests() at ??:0
@ 0x7fc79371560d testing::internal::HandleExceptionsInMethodIfSupported<>() at ??:0
@ 0x7fc793709e63 testing::UnitTest::Run() at ??:0
@ 0x5565223cbc7e RUN_ALL_TESTS() at ??:0
@ 0x5565223cae6b main at ??:0
@ 0x7fc79147abf7 __libc_start_main at ??:0
@ 0x55652236306a _start at ??:0
2025-12-15 21:04:11.140 UTC [8554] FATAL: terminating connection due to unexpected postmaster exit
2025-12-15 21:04:11.141 UTC [8451] FATAL: terminating connection due to unexpected postmaster exit
2025-12-15 21:04:11.142 UTC [8447] FATAL: terminating connection due to unexpected postmaster exit
2025-12-15 21:04:11.143 UTC [8448] FATAL: terminating connection due to unexpected postmaster exit
2025-12-15 21:04:11.144 UTC [8450] FATAL: terminating connection due to unexpected postmaster exit
2025-12-15 21:04:11.145 UTC [8446] FATAL: terminating connection due to unexpected postmaster exit