[==========] Running 57 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 31 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765270668420570-20677-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Dec 09 08:57:48 dist-test-slave-8w6c krb5kdc[20684](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Dec 09 08:57:48 dist-test-slave-8w6c krb5kdc[20684](info): set up 2 sockets
Dec 09 08:57:48 dist-test-slave-8w6c krb5kdc[20684](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20251209 08:57:50.428615 20677 mini_kdc.cc:121] Time spent starting KDC: real 1.989s user 0.003s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765270668420570-20677-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765270668420570-20677-0/krb5kdc/test-user.keytab.
Dec 09 08:57:50 dist-test-slave-8w6c krb5kdc[20684](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765270670, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-12-09T08:57:50Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-12-09T08:57:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.49.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.49.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.49.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765270668420570-20677-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.20.49.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765270668420570-20677-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.20.49.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.20.49.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.20.49.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765270668420570-20677-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.20.49.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765270668420570-20677-0/krb5kdc/kudu.keytab.
I20251209 08:57:50.590492 20677 external_mini_cluster.cc:1366] Running /tmp/dist-test-task8XuDZV/build/debug/bin/kudu
/tmp/dist-test-task8XuDZV/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765270668420570-20677-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765270668420570-20677-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765270668420570-20677-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765270668420570-20677-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.49.126:45451
--webserver_interface=127.20.49.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765270668420570-20677-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.49.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.20.49.84:34569
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.49.126:45451
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765270668420570-20677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765270668420570-20677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765270668420570-20677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251209 08:57:50.707239 20700 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251209 08:57:50.707551 20700 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251209 08:57:50.707649 20700 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251209 08:57:50.711302 20700 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251209 08:57:50.711402 20700 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251209 08:57:50.711447 20700 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251209 08:57:50.711480 20700 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251209 08:57:50.711524 20700 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251209 08:57:50.716236 20700 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.49.84:34569
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765270668420570-20677-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765270668420570-20677-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.49.126:45451
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765270668420570-20677-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.49.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.49.126:45451
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765270668420570-20677-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.49.126
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20700
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765270668420570-20677-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 468897755969d686fb08386f42f7835685e7953a
build type DEBUG
built by None at 09 Dec 2025 08:43:15 UTC on 24a791456cd2
build id 9426
I20251209 08:57:50.717495 20700 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251209 08:57:50.718465 20700 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20251209 08:57:50.725030 20700 server_base.cc:1047] running on GCE node
W20251209 08:57:50.724989 20708 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251209 08:57:50.724990 20705 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251209 08:57:50.725365 20706 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251209 08:57:50.726063 20700 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251209 08:57:50.727205 20700 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251209 08:57:50.728379 20700 hybrid_clock.cc:648] HybridClock initialized: now 1765270670728366 us; error 35 us; skew 500 ppm
Dec 09 08:57:50 dist-test-slave-8w6c krb5kdc[20684](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765270670, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.49.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251209 08:57:50.731247 20700 init.cc:377] Logged in from keytab as kudu/127.20.49.126@KRBTEST.COM (short username kudu)
I20251209 08:57:50.732376 20700 webserver.cc:492] Webserver started at http://127.20.49.126:39367/ using document root <none> and password file <none>
I20251209 08:57:50.732954 20700 fs_manager.cc:362] Metadata directory not provided
I20251209 08:57:50.733006 20700 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251209 08:57:50.733222 20700 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251209 08:57:50.735131 20700 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765270668420570-20677-0/minicluster-data/master-0/data/instance:
uuid: "e49acee603f6439c8074c33f3b5ece05"
format_stamp: "Formatted at 2025-12-09 08:57:50 on dist-test-slave-8w6c"
server_key: "c94b5984eec2f0fd3acbbc0248be4128"
server_key_iv: "3f2bf434aebd125506893ee7b017325d"
server_key_version: "encryptionkey@0"
I20251209 08:57:50.735646 20700 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765270668420570-20677-0/minicluster-data/master-0/wal/instance:
uuid: "e49acee603f6439c8074c33f3b5ece05"
format_stamp: "Formatted at 2025-12-09 08:57:50 on dist-test-slave-8w6c"
server_key: "c94b5984eec2f0fd3acbbc0248be4128"
server_key_iv: "3f2bf434aebd125506893ee7b017325d"
server_key_version: "encryptionkey@0"
I20251209 08:57:50.739634 20700 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20251209 08:57:50.742348 20715 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251209 08:57:50.743579 20700 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20251209 08:57:50.743713 20700 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765270668420570-20677-0/minicluster-data/master-0/data,/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765270668420570-20677-0/minicluster-data/master-0/wal
uuid: "e49acee603f6439c8074c33f3b5ece05"
format_stamp: "Formatted at 2025-12-09 08:57:50 on dist-test-slave-8w6c"
server_key: "c94b5984eec2f0fd3acbbc0248be4128"
server_key_iv: "3f2bf434aebd125506893ee7b017325d"
server_key_version: "encryptionkey@0"
I20251209 08:57:50.743830 20700 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765270668420570-20677-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765270668420570-20677-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765270668420570-20677-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251209 08:57:50.764909 20700 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251209 08:57:50.768214 20700 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251209 08:57:50.768384 20700 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251209 08:57:50.776409 20700 rpc_server.cc:307] RPC server started. Bound to: 127.20.49.126:45451
I20251209 08:57:50.776470 20767 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.49.126:45451 every 8 connection(s)
I20251209 08:57:50.777516 20700 server_base.cc:1179] Dumped server information to /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765270668420570-20677-0/minicluster-data/master-0/data/info.pb
I20251209 08:57:50.780488 20768 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251209 08:57:50.786727 20768 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e49acee603f6439c8074c33f3b5ece05: Bootstrap starting.
I20251209 08:57:50.788058 20677 external_mini_cluster.cc:1428] Started /tmp/dist-test-task8XuDZV/build/debug/bin/kudu as pid 20700
I20251209 08:57:50.788182 20677 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765270668420570-20677-0/minicluster-data/master-0/wal/instance
I20251209 08:57:50.789206 20677 external_mini_cluster.cc:1468] Setting key e36173aec4e8dad710e1962862946b02
I20251209 08:57:50.789266 20768 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e49acee603f6439c8074c33f3b5ece05: Neither blocks nor log segments found. Creating new log.
I20251209 08:57:50.790139 20768 log.cc:826] T 00000000000000000000000000000000 P e49acee603f6439c8074c33f3b5ece05: Log is configured to *not* fsync() on all Append() calls
I20251209 08:57:50.792526 20768 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e49acee603f6439c8074c33f3b5ece05: No bootstrap required, opened a new log
I20251209 08:57:50.795387 20768 raft_consensus.cc:359] T 00000000000000000000000000000000 P e49acee603f6439c8074c33f3b5ece05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e49acee603f6439c8074c33f3b5ece05" member_type: VOTER last_known_addr { host: "127.20.49.126" port: 45451 } }
I20251209 08:57:50.795603 20768 raft_consensus.cc:385] T 00000000000000000000000000000000 P e49acee603f6439c8074c33f3b5ece05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251209 08:57:50.795680 20768 raft_consensus.cc:740] T 00000000000000000000000000000000 P e49acee603f6439c8074c33f3b5ece05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e49acee603f6439c8074c33f3b5ece05, State: Initialized, Role: FOLLOWER
I20251209 08:57:50.796164 20768 consensus_queue.cc:260] T 00000000000000000000000000000000 P e49acee603f6439c8074c33f3b5ece05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e49acee603f6439c8074c33f3b5ece05" member_type: VOTER last_known_addr { host: "127.20.49.126" port: 45451 } }
I20251209 08:57:50.796301 20768 raft_consensus.cc:399] T 00000000000000000000000000000000 P e49acee603f6439c8074c33f3b5ece05 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251209 08:57:50.796375 20768 raft_consensus.cc:493] T 00000000000000000000000000000000 P e49acee603f6439c8074c33f3b5ece05 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251209 08:57:50.796474 20768 raft_consensus.cc:3060] T 00000000000000000000000000000000 P e49acee603f6439c8074c33f3b5ece05 [term 0 FOLLOWER]: Advancing to term 1
I20251209 08:57:50.797389 20768 raft_consensus.cc:515] T 00000000000000000000000000000000 P e49acee603f6439c8074c33f3b5ece05 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e49acee603f6439c8074c33f3b5ece05" member_type: VOTER last_known_addr { host: "127.20.49.126" port: 45451 } }
I20251209 08:57:50.797791 20768 leader_election.cc:304] T 00000000000000000000000000000000 P e49acee603f6439c8074c33f3b5ece05 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e49acee603f6439c8074c33f3b5ece05; no voters:
I20251209 08:57:50.798175 20768 leader_election.cc:290] T 00000000000000000000000000000000 P e49acee603f6439c8074c33f3b5ece05 [CANDIDATE]: Term 1 election: Requested vote from peers
I20251209 08:57:50.798288 20773 raft_consensus.cc:2804] T 00000000000000000000000000000000 P e49acee603f6439c8074c33f3b5ece05 [term 1 FOLLOWER]: Leader election won for term 1
I20251209 08:57:50.798530 20773 raft_consensus.cc:697] T 00000000000000000000000000000000 P e49acee603f6439c8074c33f3b5ece05 [term 1 LEADER]: Becoming Leader. State: Replica: e49acee603f6439c8074c33f3b5ece05, State: Running, Role: LEADER
I20251209 08:57:50.798851 20773 consensus_queue.cc:237] T 00000000000000000000000000000000 P e49acee603f6439c8074c33f3b5ece05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e49acee603f6439c8074c33f3b5ece05" member_type: VOTER last_known_addr { host: "127.20.49.126" port: 45451 } }
I20251209 08:57:50.799412 20768 sys_catalog.cc:565] T 00000000000000000000000000000000 P e49acee603f6439c8074c33f3b5ece05 [sys.catalog]: configured and running, proceeding with master startup.
I20251209 08:57:50.800614 20774 sys_catalog.cc:455] T 00000000000000000000000000000000 P e49acee603f6439c8074c33f3b5ece05 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e49acee603f6439c8074c33f3b5ece05" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e49acee603f6439c8074c33f3b5ece05" member_type: VOTER last_known_addr { host: "127.20.49.126" port: 45451 } } }
I20251209 08:57:50.800736 20774 sys_catalog.cc:458] T 00000000000000000000000000000000 P e49acee603f6439c8074c33f3b5ece05 [sys.catalog]: This master's current role is: LEADER
I20251209 08:57:50.801050 20775 sys_catalog.cc:455] T 00000000000000000000000000000000 P e49acee603f6439c8074c33f3b5ece05 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e49acee603f6439c8074c33f3b5ece05. Latest consensus state: current_term: 1 leader_uuid: "e49acee603f6439c8074c33f3b5ece05" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e49acee603f6439c8074c33f3b5ece05" member_type: VOTER last_known_addr { host: "127.20.49.126" port: 45451 } } }
I20251209 08:57:50.801141 20775 sys_catalog.cc:458] T 00000000000000000000000000000000 P e49acee603f6439c8074c33f3b5ece05 [sys.catalog]: This master's current role is: LEADER
I20251209 08:57:50.801710 20782 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
Dec 09 08:57:50 dist-test-slave-8w6c krb5kdc[20684](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765270670, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.49.126@KRBTEST.COM
I20251209 08:57:50.805040 20782 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251209 08:57:50.810321 20771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:57:50.791605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56600 (local address 127.20.49.126:45451)
1209 08:57:50.792071 (+ 466us) server_negotiation.cc:207] Beginning negotiation
1209 08:57:50.792084 (+ 13us) server_negotiation.cc:400] Waiting for connection header
1209 08:57:50.792669 (+ 585us) server_negotiation.cc:408] Connection header received
1209 08:57:50.794981 (+ 2312us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1209 08:57:50.794999 (+ 18us) server_negotiation.cc:462] Received NEGOTIATE request from client
1209 08:57:50.795336 (+ 337us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1209 08:57:50.795675 (+ 339us) server_negotiation.cc:227] Negotiated authn=SASL
1209 08:57:50.797728 (+ 2053us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:57:50.798885 (+ 1157us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1209 08:57:50.799823 (+ 938us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:57:50.800122 (+ 299us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:57:50.804481 (+ 4359us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1209 08:57:50.804503 (+ 22us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1209 08:57:50.804518 (+ 15us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1209 08:57:50.804561 (+ 43us) server_negotiation.cc:923] calling sasl_server_start()
1209 08:57:50.807019 (+ 2458us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1209 08:57:50.807665 (+ 646us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1209 08:57:50.807671 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1209 08:57:50.807677 (+ 6us) server_negotiation.cc:968] calling sasl_server_step()
1209 08:57:50.807781 (+ 104us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1209 08:57:50.808130 (+ 349us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1209 08:57:50.808134 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1209 08:57:50.808136 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1209 08:57:50.808582 (+ 446us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1209 08:57:50.808803 (+ 221us) server_negotiation.cc:1036] Waiting for connection context
1209 08:57:50.809341 (+ 538us) server_negotiation.cc:300] Negotiation successful
1209 08:57:50.809594 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":290,"thread_start_us":149,"threads_started":1}
I20251209 08:57:50.811949 20782 catalog_manager.cc:1357] Generated new cluster ID: e816120fcfcb4df09822b8cf4bfe51c6
I20251209 08:57:50.812026 20782 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251209 08:57:50.821291 20782 catalog_manager.cc:1380] Generated new certificate authority record
I20251209 08:57:50.822678 20782 catalog_manager.cc:1514] Loading token signing keys...
I20251209 08:57:50.841231 20782 catalog_manager.cc:6027] T 00000000000000000000000000000000 P e49acee603f6439c8074c33f3b5ece05: Generated new TSK 0
I20251209 08:57:50.842229 20782 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.20.49.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.49.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.49.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765270668420570-20677-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.20.49.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765270668420570-20677-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.20.49.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.20.49.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.20.49.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765270668420570-20677-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.20.49.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765270668420570-20677-0/krb5kdc/kudu.keytab.
I20251209 08:57:50.904378 20677 external_mini_cluster.cc:1366] Running /tmp/dist-test-task8XuDZV/build/debug/bin/kudu
/tmp/dist-test-task8XuDZV/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765270668420570-20677-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765270668420570-20677-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765270668420570-20677-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765270668420570-20677-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.49.65:0
--local_ip_for_outbound_sockets=127.20.49.65
--webserver_interface=127.20.49.65
--webserver_port=0
--tserver_master_addrs=127.20.49.126:45451
--keytab_file=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765270668420570-20677-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.49.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.20.49.84:34569
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765270668420570-20677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765270668420570-20677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765270668420570-20677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251209 08:57:51.018257 20796 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251209 08:57:51.018496 20796 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251209 08:57:51.018558 20796 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251209 08:57:51.021998 20796 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251209 08:57:51.022075 20796 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251209 08:57:51.022161 20796 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.49.65
I20251209 08:57:51.026479 20796 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.49.84:34569
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765270668420570-20677-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765270668420570-20677-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765270668420570-20677-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.49.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.49.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765270668420570-20677-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.49.65
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.20.49.126:45451
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20796
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.49.65
--log_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765270668420570-20677-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 468897755969d686fb08386f42f7835685e7953a
build type DEBUG
built by None at 09 Dec 2025 08:43:15 UTC on 24a791456cd2
build id 9426
I20251209 08:57:51.027534 20796 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251209 08:57:51.028357 20796 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251209 08:57:51.035452 20801 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251209 08:57:51.035413 20802 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251209 08:57:51.035499 20804 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251209 08:57:51.036377 20796 server_base.cc:1047] running on GCE node
I20251209 08:57:51.036840 20796 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251209 08:57:51.037448 20796 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251209 08:57:51.038614 20796 hybrid_clock.cc:648] HybridClock initialized: now 1765270671038593 us; error 46 us; skew 500 ppm
Dec 09 08:57:51 dist-test-slave-8w6c krb5kdc[20684](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765270671, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.49.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251209 08:57:51.041522 20796 init.cc:377] Logged in from keytab as kudu/127.20.49.65@KRBTEST.COM (short username kudu)
I20251209 08:57:51.042739 20796 webserver.cc:492] Webserver started at http://127.20.49.65:45511/ using document root <none> and password file <none>
I20251209 08:57:51.043295 20796 fs_manager.cc:362] Metadata directory not provided
I20251209 08:57:51.043344 20796 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251209 08:57:51.043507 20796 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251209 08:57:51.045324 20796 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765270668420570-20677-0/minicluster-data/ts-0/data/instance:
uuid: "d6f7fecb192f4eb6a59005e0a2c46de8"
format_stamp: "Formatted at 2025-12-09 08:57:51 on dist-test-slave-8w6c"
server_key: "2615a6274ca10485bce1035221c54d8c"
server_key_iv: "fcf531ac61290b64518470ffa14ce687"
server_key_version: "encryptionkey@0"
I20251209 08:57:51.045853 20796 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765270668420570-20677-0/minicluster-data/ts-0/wal/instance:
uuid: "d6f7fecb192f4eb6a59005e0a2c46de8"
format_stamp: "Formatted at 2025-12-09 08:57:51 on dist-test-slave-8w6c"
server_key: "2615a6274ca10485bce1035221c54d8c"
server_key_iv: "fcf531ac61290b64518470ffa14ce687"
server_key_version: "encryptionkey@0"
I20251209 08:57:51.049762 20796 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20251209 08:57:51.052327 20811 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251209 08:57:51.053450 20796 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20251209 08:57:51.053562 20796 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765270668420570-20677-0/minicluster-data/ts-0/data,/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765270668420570-20677-0/minicluster-data/ts-0/wal
uuid: "d6f7fecb192f4eb6a59005e0a2c46de8"
format_stamp: "Formatted at 2025-12-09 08:57:51 on dist-test-slave-8w6c"
server_key: "2615a6274ca10485bce1035221c54d8c"
server_key_iv: "fcf531ac61290b64518470ffa14ce687"
server_key_version: "encryptionkey@0"
I20251209 08:57:51.053812 20796 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765270668420570-20677-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765270668420570-20677-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765270668420570-20677-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251209 08:57:51.076733 20796 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251209 08:57:51.080477 20796 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251209 08:57:51.080797 20796 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251209 08:57:51.081490 20796 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251209 08:57:51.082489 20796 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251209 08:57:51.082541 20796 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251209 08:57:51.082584 20796 ts_tablet_manager.cc:616] Registered 0 tablets
I20251209 08:57:51.082616 20796 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251209 08:57:51.091997 20796 rpc_server.cc:307] RPC server started. Bound to: 127.20.49.65:46781
I20251209 08:57:51.092064 20924 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.49.65:46781 every 8 connection(s)
I20251209 08:57:51.093014 20796 server_base.cc:1179] Dumped server information to /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765270668420570-20677-0/minicluster-data/ts-0/data/info.pb
I20251209 08:57:51.101295 20677 external_mini_cluster.cc:1428] Started /tmp/dist-test-task8XuDZV/build/debug/bin/kudu as pid 20796
I20251209 08:57:51.101433 20677 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765270668420570-20677-0/minicluster-data/ts-0/wal/instance
I20251209 08:57:51.101790 20677 external_mini_cluster.cc:1468] Setting key 0c3f8c0d668b2eaf96cb29780bef67a6
Dec 09 08:57:51 dist-test-slave-8w6c krb5kdc[20684](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765270671, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.49.65@KRBTEST.COM for kudu/127.20.49.126@KRBTEST.COM
I20251209 08:57:51.107214 20771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:57:51.094972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.49.65:44925 (local address 127.20.49.126:45451)
1209 08:57:51.095135 (+ 163us) server_negotiation.cc:207] Beginning negotiation
1209 08:57:51.095139 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1209 08:57:51.095870 (+ 731us) server_negotiation.cc:408] Connection header received
1209 08:57:51.096766 (+ 896us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1209 08:57:51.096769 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1209 08:57:51.096819 (+ 50us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1209 08:57:51.096914 (+ 95us) server_negotiation.cc:227] Negotiated authn=SASL
1209 08:57:51.098695 (+ 1781us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:57:51.099252 (+ 557us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1209 08:57:51.100038 (+ 786us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:57:51.100186 (+ 148us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:57:51.103021 (+ 2835us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1209 08:57:51.103039 (+ 18us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1209 08:57:51.103041 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1209 08:57:51.103068 (+ 27us) server_negotiation.cc:923] calling sasl_server_start()
1209 08:57:51.104680 (+ 1612us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1209 08:57:51.105381 (+ 701us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1209 08:57:51.105388 (+ 7us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1209 08:57:51.105391 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
1209 08:57:51.105455 (+ 64us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1209 08:57:51.105809 (+ 354us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1209 08:57:51.105814 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1209 08:57:51.105818 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
1209 08:57:51.106076 (+ 258us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1209 08:57:51.106217 (+ 141us) server_negotiation.cc:1036] Waiting for connection context
1209 08:57:51.106771 (+ 554us) server_negotiation.cc:300] Negotiation successful
1209 08:57:51.106956 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20251209 08:57:51.107689 20927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:57:51.095218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.49.126:45451 (local address 127.20.49.65:44925)
1209 08:57:51.095683 (+ 465us) negotiation.cc:107] Waiting for socket to connect
1209 08:57:51.095752 (+ 69us) client_negotiation.cc:174] Beginning negotiation
1209 08:57:51.096509 (+ 757us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1209 08:57:51.097084 (+ 575us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1209 08:57:51.097093 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
1209 08:57:51.097495 (+ 402us) client_negotiation.cc:189] Negotiated authn=SASL
1209 08:57:51.098507 (+ 1012us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:57:51.098522 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:57:51.099399 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1209 08:57:51.099403 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1209 08:57:51.099867 (+ 464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:57:51.099875 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:57:51.100107 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:57:51.100804 (+ 697us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1209 08:57:51.100845 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
1209 08:57:51.102790 (+ 1945us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1209 08:57:51.104822 (+ 2032us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1209 08:57:51.104833 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1209 08:57:51.104851 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
1209 08:57:51.105253 (+ 402us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1209 08:57:51.105570 (+ 317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1209 08:57:51.105572 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1209 08:57:51.105574 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1209 08:57:51.105700 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1209 08:57:51.106245 (+ 545us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1209 08:57:51.106251 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1209 08:57:51.106500 (+ 249us) client_negotiation.cc:769] Sending connection context
1209 08:57:51.106694 (+ 194us) client_negotiation.cc:240] Negotiation successful
1209 08:57:51.106940 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":122,"threads_started":1}
I20251209 08:57:51.109067 20925 heartbeater.cc:344] Connected to a master server at 127.20.49.126:45451
I20251209 08:57:51.109321 20925 heartbeater.cc:461] Registering TS with master...
I20251209 08:57:51.110059 20925 heartbeater.cc:507] Master 127.20.49.126:45451 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.20.49.66@KRBTEST.COM; defaulting to no policy
I20251209 08:57:51.111676 20732 ts_manager.cc:194] Registered new tserver with Master: d6f7fecb192f4eb6a59005e0a2c46de8 (127.20.49.65:46781)
I20251209 08:57:51.112875 20732 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.49.65@KRBTEST.COM'} at 127.20.49.65:44925
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.49.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.49.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765270668420570-20677-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.20.49.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765270668420570-20677-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.20.49.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.20.49.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.20.49.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765270668420570-20677-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.20.49.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765270668420570-20677-0/krb5kdc/kudu.keytab.
I20251209 08:57:51.152925 20677 external_mini_cluster.cc:1366] Running /tmp/dist-test-task8XuDZV/build/debug/bin/kudu
/tmp/dist-test-task8XuDZV/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765270668420570-20677-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765270668420570-20677-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765270668420570-20677-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765270668420570-20677-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.49.66:0
--local_ip_for_outbound_sockets=127.20.49.66
--webserver_interface=127.20.49.66
--webserver_port=0
--tserver_master_addrs=127.20.49.126:45451
--keytab_file=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765270668420570-20677-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.49.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.20.49.84:34569
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765270668420570-20677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765270668420570-20677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765270668420570-20677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251209 08:57:51.264142 20932 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251209 08:57:51.264396 20932 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251209 08:57:51.264490 20932 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251209 08:57:51.267932 20932 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251209 08:57:51.268029 20932 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251209 08:57:51.268149 20932 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.49.66
I20251209 08:57:51.272851 20932 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.49.84:34569
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765270668420570-20677-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765270668420570-20677-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765270668420570-20677-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.49.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.49.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765270668420570-20677-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.49.66
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.20.49.126:45451
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20932
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.49.66
--log_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765270668420570-20677-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 468897755969d686fb08386f42f7835685e7953a
build type DEBUG
built by None at 09 Dec 2025 08:43:15 UTC on 24a791456cd2
build id 9426
I20251209 08:57:51.274116 20932 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251209 08:57:51.275002 20932 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251209 08:57:51.281968 20938 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251209 08:57:51.282044 20940 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251209 08:57:51.282068 20937 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251209 08:57:51.282214 20932 server_base.cc:1047] running on GCE node
I20251209 08:57:51.282673 20932 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251209 08:57:51.283253 20932 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251209 08:57:51.284421 20932 hybrid_clock.cc:648] HybridClock initialized: now 1765270671284406 us; error 34 us; skew 500 ppm
Dec 09 08:57:51 dist-test-slave-8w6c krb5kdc[20684](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765270671, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.49.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251209 08:57:51.287274 20932 init.cc:377] Logged in from keytab as kudu/127.20.49.66@KRBTEST.COM (short username kudu)
I20251209 08:57:51.288342 20932 webserver.cc:492] Webserver started at http://127.20.49.66:44463/ using document root <none> and password file <none>
I20251209 08:57:51.288933 20932 fs_manager.cc:362] Metadata directory not provided
I20251209 08:57:51.288985 20932 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251209 08:57:51.289176 20932 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251209 08:57:51.291409 20932 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765270668420570-20677-0/minicluster-data/ts-1/data/instance:
uuid: "30e0e03871e9497f8450fd5c7c87f716"
format_stamp: "Formatted at 2025-12-09 08:57:51 on dist-test-slave-8w6c"
server_key: "1ffd73a40f7bba6ae489c9824b24892e"
server_key_iv: "011e7cbe4545fc9a867b49d50c471471"
server_key_version: "encryptionkey@0"
I20251209 08:57:51.291940 20932 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765270668420570-20677-0/minicluster-data/ts-1/wal/instance:
uuid: "30e0e03871e9497f8450fd5c7c87f716"
format_stamp: "Formatted at 2025-12-09 08:57:51 on dist-test-slave-8w6c"
server_key: "1ffd73a40f7bba6ae489c9824b24892e"
server_key_iv: "011e7cbe4545fc9a867b49d50c471471"
server_key_version: "encryptionkey@0"
I20251209 08:57:51.295580 20932 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.001s
I20251209 08:57:51.298023 20947 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251209 08:57:51.299137 20932 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20251209 08:57:51.299250 20932 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765270668420570-20677-0/minicluster-data/ts-1/data,/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765270668420570-20677-0/minicluster-data/ts-1/wal
uuid: "30e0e03871e9497f8450fd5c7c87f716"
format_stamp: "Formatted at 2025-12-09 08:57:51 on dist-test-slave-8w6c"
server_key: "1ffd73a40f7bba6ae489c9824b24892e"
server_key_iv: "011e7cbe4545fc9a867b49d50c471471"
server_key_version: "encryptionkey@0"
I20251209 08:57:51.299345 20932 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765270668420570-20677-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765270668420570-20677-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765270668420570-20677-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251209 08:57:51.312059 20932 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251209 08:57:51.315199 20932 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251209 08:57:51.315399 20932 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251209 08:57:51.315963 20932 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251209 08:57:51.316825 20932 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251209 08:57:51.316875 20932 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251209 08:57:51.316942 20932 ts_tablet_manager.cc:616] Registered 0 tablets
I20251209 08:57:51.316970 20932 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251209 08:57:51.327126 20932 rpc_server.cc:307] RPC server started. Bound to: 127.20.49.66:38361
I20251209 08:57:51.327142 21060 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.49.66:38361 every 8 connection(s)
I20251209 08:57:51.328207 20932 server_base.cc:1179] Dumped server information to /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765270668420570-20677-0/minicluster-data/ts-1/data/info.pb
I20251209 08:57:51.329420 20677 external_mini_cluster.cc:1428] Started /tmp/dist-test-task8XuDZV/build/debug/bin/kudu as pid 20932
I20251209 08:57:51.329516 20677 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765270668420570-20677-0/minicluster-data/ts-1/wal/instance
I20251209 08:57:51.329789 20677 external_mini_cluster.cc:1468] Setting key 35d7598e25519040cea3e3a8610ea304
WARNING: no policy specified for kudu/127.20.49.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.49.67@KRBTEST.COM" created.
Dec 09 08:57:51 dist-test-slave-8w6c krb5kdc[20684](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765270671, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.49.66@KRBTEST.COM for kudu/127.20.49.126@KRBTEST.COM
I20251209 08:57:51.348362 20771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:57:51.330380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.49.66:60365 (local address 127.20.49.126:45451)
1209 08:57:51.330546 (+ 166us) server_negotiation.cc:207] Beginning negotiation
1209 08:57:51.330551 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1209 08:57:51.331485 (+ 934us) server_negotiation.cc:408] Connection header received
1209 08:57:51.332784 (+ 1299us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1209 08:57:51.332788 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1209 08:57:51.332843 (+ 55us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1209 08:57:51.332926 (+ 83us) server_negotiation.cc:227] Negotiated authn=SASL
1209 08:57:51.335156 (+ 2230us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:57:51.335836 (+ 680us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1209 08:57:51.336882 (+ 1046us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:57:51.337038 (+ 156us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:57:51.344188 (+ 7150us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1209 08:57:51.344209 (+ 21us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1209 08:57:51.344212 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1209 08:57:51.344245 (+ 33us) server_negotiation.cc:923] calling sasl_server_start()
1209 08:57:51.346162 (+ 1917us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1209 08:57:51.346752 (+ 590us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1209 08:57:51.346755 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1209 08:57:51.346756 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1209 08:57:51.346807 (+ 51us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1209 08:57:51.347142 (+ 335us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1209 08:57:51.347148 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1209 08:57:51.347150 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1209 08:57:51.347323 (+ 173us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1209 08:57:51.347425 (+ 102us) server_negotiation.cc:1036] Waiting for connection context
1209 08:57:51.348027 (+ 602us) server_negotiation.cc:300] Negotiation successful
1209 08:57:51.348179 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20251209 08:57:51.349076 21063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:57:51.330672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.49.126:45451 (local address 127.20.49.66:60365)
1209 08:57:51.331306 (+ 634us) negotiation.cc:107] Waiting for socket to connect
1209 08:57:51.331357 (+ 51us) client_negotiation.cc:174] Beginning negotiation
1209 08:57:51.332507 (+ 1150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1209 08:57:51.333243 (+ 736us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1209 08:57:51.333257 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
1209 08:57:51.333903 (+ 646us) client_negotiation.cc:189] Negotiated authn=SASL
1209 08:57:51.334962 (+ 1059us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:57:51.334980 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:57:51.336003 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1209 08:57:51.336018 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1209 08:57:51.336735 (+ 717us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:57:51.336751 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:57:51.337046 (+ 295us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:57:51.341236 (+ 4190us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1209 08:57:51.341281 (+ 45us) client_negotiation.cc:656] calling sasl_client_start()
1209 08:57:51.343971 (+ 2690us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1209 08:57:51.346332 (+ 2361us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1209 08:57:51.346341 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1209 08:57:51.346353 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
1209 08:57:51.346630 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1209 08:57:51.346906 (+ 276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1209 08:57:51.346910 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1209 08:57:51.346911 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1209 08:57:51.347039 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1209 08:57:51.347439 (+ 400us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1209 08:57:51.347450 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1209 08:57:51.347736 (+ 286us) client_negotiation.cc:769] Sending connection context
1209 08:57:51.347971 (+ 235us) client_negotiation.cc:240] Negotiation successful
1209 08:57:51.348285 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":410,"thread_start_us":120,"threads_started":1}
I20251209 08:57:51.350611 21061 heartbeater.cc:344] Connected to a master server at 127.20.49.126:45451
I20251209 08:57:51.350945 21061 heartbeater.cc:461] Registering TS with master...
I20251209 08:57:51.351549 21061 heartbeater.cc:507] Master 127.20.49.126:45451 requested a full tablet report, sending...
I20251209 08:57:51.352851 20732 ts_manager.cc:194] Registered new tserver with Master: 30e0e03871e9497f8450fd5c7c87f716 (127.20.49.66:38361)
I20251209 08:57:51.353461 20732 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.49.66@KRBTEST.COM'} at 127.20.49.66:60365
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.49.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765270668420570-20677-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.20.49.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765270668420570-20677-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.20.49.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.20.49.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.20.49.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765270668420570-20677-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.20.49.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765270668420570-20677-0/krb5kdc/kudu.keytab.
I20251209 08:57:51.389518 20677 external_mini_cluster.cc:1366] Running /tmp/dist-test-task8XuDZV/build/debug/bin/kudu
/tmp/dist-test-task8XuDZV/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765270668420570-20677-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765270668420570-20677-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765270668420570-20677-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765270668420570-20677-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.49.67:0
--local_ip_for_outbound_sockets=127.20.49.67
--webserver_interface=127.20.49.67
--webserver_port=0
--tserver_master_addrs=127.20.49.126:45451
--keytab_file=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765270668420570-20677-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.49.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.20.49.84:34569
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765270668420570-20677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765270668420570-20677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765270668420570-20677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251209 08:57:51.501521 21068 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251209 08:57:51.501806 21068 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251209 08:57:51.501868 21068 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251209 08:57:51.505308 21068 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251209 08:57:51.505381 21068 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251209 08:57:51.505470 21068 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.49.67
I20251209 08:57:51.509861 21068 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.49.84:34569
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765270668420570-20677-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765270668420570-20677-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765270668420570-20677-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.49.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.49.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765270668420570-20677-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.49.67
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.20.49.126:45451
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21068
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.49.67
--log_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765270668420570-20677-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 468897755969d686fb08386f42f7835685e7953a
build type DEBUG
built by None at 09 Dec 2025 08:43:15 UTC on 24a791456cd2
build id 9426
I20251209 08:57:51.510941 21068 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251209 08:57:51.511869 21068 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251209 08:57:51.519953 21074 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251209 08:57:51.519960 21073 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251209 08:57:51.520280 21068 server_base.cc:1047] running on GCE node
W20251209 08:57:51.519959 21076 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251209 08:57:51.520716 21068 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251209 08:57:51.521335 21068 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251209 08:57:51.522509 21068 hybrid_clock.cc:648] HybridClock initialized: now 1765270671522453 us; error 77 us; skew 500 ppm
Dec 09 08:57:51 dist-test-slave-8w6c krb5kdc[20684](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765270671, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.49.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251209 08:57:51.525501 21068 init.cc:377] Logged in from keytab as kudu/127.20.49.67@KRBTEST.COM (short username kudu)
I20251209 08:57:51.526858 21068 webserver.cc:492] Webserver started at http://127.20.49.67:39695/ using document root <none> and password file <none>
I20251209 08:57:51.527446 21068 fs_manager.cc:362] Metadata directory not provided
I20251209 08:57:51.527504 21068 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251209 08:57:51.527697 21068 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251209 08:57:51.529649 21068 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765270668420570-20677-0/minicluster-data/ts-2/data/instance:
uuid: "db3a09e2122042b693a589b8a8137ae7"
format_stamp: "Formatted at 2025-12-09 08:57:51 on dist-test-slave-8w6c"
server_key: "3b36a1425691ca20510e746d611f8c1e"
server_key_iv: "98ce555b9690881d52f6c4953fbca4e3"
server_key_version: "encryptionkey@0"
I20251209 08:57:51.530239 21068 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765270668420570-20677-0/minicluster-data/ts-2/wal/instance:
uuid: "db3a09e2122042b693a589b8a8137ae7"
format_stamp: "Formatted at 2025-12-09 08:57:51 on dist-test-slave-8w6c"
server_key: "3b36a1425691ca20510e746d611f8c1e"
server_key_iv: "98ce555b9690881d52f6c4953fbca4e3"
server_key_version: "encryptionkey@0"
I20251209 08:57:51.533924 21068 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20251209 08:57:51.536418 21083 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251209 08:57:51.537658 21068 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20251209 08:57:51.537859 21068 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765270668420570-20677-0/minicluster-data/ts-2/data,/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765270668420570-20677-0/minicluster-data/ts-2/wal
uuid: "db3a09e2122042b693a589b8a8137ae7"
format_stamp: "Formatted at 2025-12-09 08:57:51 on dist-test-slave-8w6c"
server_key: "3b36a1425691ca20510e746d611f8c1e"
server_key_iv: "98ce555b9690881d52f6c4953fbca4e3"
server_key_version: "encryptionkey@0"
I20251209 08:57:51.537977 21068 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765270668420570-20677-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765270668420570-20677-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765270668420570-20677-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251209 08:57:51.568416 21068 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251209 08:57:51.571846 21068 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251209 08:57:51.572072 21068 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251209 08:57:51.572656 21068 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251209 08:57:51.573614 21068 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251209 08:57:51.573668 21068 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251209 08:57:51.573765 21068 ts_tablet_manager.cc:616] Registered 0 tablets
I20251209 08:57:51.573799 21068 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251209 08:57:51.583925 21068 rpc_server.cc:307] RPC server started. Bound to: 127.20.49.67:41429
I20251209 08:57:51.583966 21196 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.49.67:41429 every 8 connection(s)
I20251209 08:57:51.584949 21068 server_base.cc:1179] Dumped server information to /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765270668420570-20677-0/minicluster-data/ts-2/data/info.pb
I20251209 08:57:51.586578 20677 external_mini_cluster.cc:1428] Started /tmp/dist-test-task8XuDZV/build/debug/bin/kudu as pid 21068
I20251209 08:57:51.586676 20677 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1765270668420570-20677-0/minicluster-data/ts-2/wal/instance
I20251209 08:57:51.586907 20677 external_mini_cluster.cc:1468] Setting key 111c8b687cbbe00a7b245e474b35a634
Dec 09 08:57:51 dist-test-slave-8w6c krb5kdc[20684](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765270671, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.49.67@KRBTEST.COM for kudu/127.20.49.126@KRBTEST.COM
I20251209 08:57:51.598126 20771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:57:51.587010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.49.67:57817 (local address 127.20.49.126:45451)
1209 08:57:51.587175 (+ 165us) server_negotiation.cc:207] Beginning negotiation
1209 08:57:51.587180 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1209 08:57:51.588116 (+ 936us) server_negotiation.cc:408] Connection header received
1209 08:57:51.589008 (+ 892us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1209 08:57:51.589012 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1209 08:57:51.589074 (+ 62us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1209 08:57:51.589161 (+ 87us) server_negotiation.cc:227] Negotiated authn=SASL
1209 08:57:51.591118 (+ 1957us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:57:51.591679 (+ 561us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1209 08:57:51.592408 (+ 729us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:57:51.592557 (+ 149us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:57:51.594894 (+ 2337us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1209 08:57:51.594914 (+ 20us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1209 08:57:51.594916 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1209 08:57:51.594943 (+ 27us) server_negotiation.cc:923] calling sasl_server_start()
1209 08:57:51.596270 (+ 1327us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1209 08:57:51.596800 (+ 530us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1209 08:57:51.596802 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1209 08:57:51.596804 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1209 08:57:51.596847 (+ 43us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1209 08:57:51.597079 (+ 232us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1209 08:57:51.597080 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1209 08:57:51.597082 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1209 08:57:51.597226 (+ 144us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1209 08:57:51.597297 (+ 71us) server_negotiation.cc:1036] Waiting for connection context
1209 08:57:51.597886 (+ 589us) server_negotiation.cc:300] Negotiation successful
1209 08:57:51.597996 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20251209 08:57:51.598834 21199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:57:51.587331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.49.126:45451 (local address 127.20.49.67:57817)
1209 08:57:51.587953 (+ 622us) negotiation.cc:107] Waiting for socket to connect
1209 08:57:51.587999 (+ 46us) client_negotiation.cc:174] Beginning negotiation
1209 08:57:51.588820 (+ 821us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1209 08:57:51.589457 (+ 637us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1209 08:57:51.589469 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
1209 08:57:51.589942 (+ 473us) client_negotiation.cc:189] Negotiated authn=SASL
1209 08:57:51.590899 (+ 957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:57:51.590914 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:57:51.591827 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1209 08:57:51.591830 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1209 08:57:51.592267 (+ 437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:57:51.592275 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:57:51.592475 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:57:51.593100 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1209 08:57:51.593125 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
1209 08:57:51.594686 (+ 1561us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1209 08:57:51.596387 (+ 1701us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1209 08:57:51.596392 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1209 08:57:51.596403 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
1209 08:57:51.596642 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1209 08:57:51.596913 (+ 271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1209 08:57:51.596916 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1209 08:57:51.596917 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1209 08:57:51.597011 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1209 08:57:51.597308 (+ 297us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1209 08:57:51.597313 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1209 08:57:51.597548 (+ 235us) client_negotiation.cc:769] Sending connection context
1209 08:57:51.597877 (+ 329us) client_negotiation.cc:240] Negotiation successful
1209 08:57:51.598124 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":415,"thread_start_us":250,"threads_started":1}
I20251209 08:57:51.600122 21197 heartbeater.cc:344] Connected to a master server at 127.20.49.126:45451
I20251209 08:57:51.600428 21197 heartbeater.cc:461] Registering TS with master...
I20251209 08:57:51.601061 21197 heartbeater.cc:507] Master 127.20.49.126:45451 requested a full tablet report, sending...
I20251209 08:57:51.602203 20732 ts_manager.cc:194] Registered new tserver with Master: db3a09e2122042b693a589b8a8137ae7 (127.20.49.67:41429)
I20251209 08:57:51.602764 20732 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.49.67@KRBTEST.COM'} at 127.20.49.67:57817
I20251209 08:57:51.613839 20677 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
Dec 09 08:57:51 dist-test-slave-8w6c krb5kdc[20684](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765270671, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Dec 09 08:57:51 dist-test-slave-8w6c krb5kdc[20684](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765270671, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.20.49.65@KRBTEST.COM
I20251209 08:57:51.644292 21204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:57:51.634563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60220 (local address 127.20.49.65:46781)
1209 08:57:51.634852 (+ 289us) server_negotiation.cc:207] Beginning negotiation
1209 08:57:51.634858 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1209 08:57:51.634885 (+ 27us) server_negotiation.cc:408] Connection header received
1209 08:57:51.635063 (+ 178us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1209 08:57:51.635068 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1209 08:57:51.635243 (+ 175us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1209 08:57:51.635368 (+ 125us) server_negotiation.cc:227] Negotiated authn=SASL
1209 08:57:51.636193 (+ 825us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:57:51.636917 (+ 724us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1209 08:57:51.637667 (+ 750us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:57:51.637972 (+ 305us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:57:51.640257 (+ 2285us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1209 08:57:51.640306 (+ 49us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1209 08:57:51.640318 (+ 12us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1209 08:57:51.640349 (+ 31us) server_negotiation.cc:923] calling sasl_server_start()
1209 08:57:51.642440 (+ 2091us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1209 08:57:51.642967 (+ 527us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1209 08:57:51.642972 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1209 08:57:51.642974 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1209 08:57:51.643034 (+ 60us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1209 08:57:51.643270 (+ 236us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1209 08:57:51.643274 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1209 08:57:51.643276 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1209 08:57:51.643497 (+ 221us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1209 08:57:51.643737 (+ 240us) server_negotiation.cc:1036] Waiting for connection context
1209 08:57:51.643928 (+ 191us) server_negotiation.cc:300] Negotiation successful
1209 08:57:51.644122 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":181,"thread_start_us":105,"threads_started":1}
W20251209 08:57:51.645244 20839 server_base.cc:1129] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60220
Dec 09 08:57:51 dist-test-slave-8w6c krb5kdc[20684](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765270671, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Dec 09 08:57:51 dist-test-slave-8w6c krb5kdc[20684](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765270671, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.49.65@KRBTEST.COM
I20251209 08:57:51.671689 21204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:57:51.663336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60236 (local address 127.20.49.65:46781)
1209 08:57:51.663524 (+ 188us) server_negotiation.cc:207] Beginning negotiation
1209 08:57:51.663528 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1209 08:57:51.663545 (+ 17us) server_negotiation.cc:408] Connection header received
1209 08:57:51.663689 (+ 144us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1209 08:57:51.663691 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1209 08:57:51.663744 (+ 53us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1209 08:57:51.663832 (+ 88us) server_negotiation.cc:227] Negotiated authn=SASL
1209 08:57:51.664714 (+ 882us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:57:51.665253 (+ 539us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1209 08:57:51.666004 (+ 751us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:57:51.666168 (+ 164us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:57:51.668586 (+ 2418us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1209 08:57:51.668607 (+ 21us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1209 08:57:51.668609 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1209 08:57:51.668639 (+ 30us) server_negotiation.cc:923] calling sasl_server_start()
1209 08:57:51.670209 (+ 1570us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1209 08:57:51.670620 (+ 411us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1209 08:57:51.670623 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1209 08:57:51.670624 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1209 08:57:51.670672 (+ 48us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1209 08:57:51.670888 (+ 216us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1209 08:57:51.670891 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1209 08:57:51.670893 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1209 08:57:51.671044 (+ 151us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1209 08:57:51.671166 (+ 122us) server_negotiation.cc:1036] Waiting for connection context
1209 08:57:51.671390 (+ 224us) server_negotiation.cc:300] Negotiation successful
1209 08:57:51.671512 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20251209 08:57:51.672958 20677 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task8XuDZV/build/debug/bin/kudu with pid 20796
I20251209 08:57:51.679563 20677 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task8XuDZV/build/debug/bin/kudu with pid 20932
I20251209 08:57:51.687500 20677 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task8XuDZV/build/debug/bin/kudu with pid 21068
I20251209 08:57:51.693653 20677 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task8XuDZV/build/debug/bin/kudu with pid 20700
2025-12-09T08:57:51Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (3285 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765270668420570-20677-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Dec 09 08:57:51 dist-test-slave-8w6c krb5kdc[21211](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Dec 09 08:57:51 dist-test-slave-8w6c krb5kdc[21211](info): set up 2 sockets
Dec 09 08:57:51 dist-test-slave-8w6c krb5kdc[21211](info): commencing operation
krb5kdc: starting...
W20251209 08:57:53.714169 20677 mini_kdc.cc:121] Time spent starting KDC: real 1.996s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765270668420570-20677-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765270668420570-20677-0/krb5kdc/test-user.keytab.
Dec 09 08:57:53 dist-test-slave-8w6c krb5kdc[21211](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765270673, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-12-09T08:57:53Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-12-09T08:57:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.49.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.49.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.49.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765270668420570-20677-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.20.49.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765270668420570-20677-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.20.49.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.20.49.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.20.49.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765270668420570-20677-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.20.49.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765270668420570-20677-0/krb5kdc/kudu.keytab.
I20251209 08:57:53.861214 20677 external_mini_cluster.cc:1366] Running /tmp/dist-test-task8XuDZV/build/debug/bin/kudu
/tmp/dist-test-task8XuDZV/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765270668420570-20677-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765270668420570-20677-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765270668420570-20677-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765270668420570-20677-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.49.126:36881
--webserver_interface=127.20.49.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765270668420570-20677-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.49.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.20.49.84:46307
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.49.126:36881
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765270668420570-20677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765270668420570-20677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765270668420570-20677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251209 08:57:53.975428 21227 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251209 08:57:53.975711 21227 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251209 08:57:53.975769 21227 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251209 08:57:53.979306 21227 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251209 08:57:53.979396 21227 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251209 08:57:53.979420 21227 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251209 08:57:53.979442 21227 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251209 08:57:53.979460 21227 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251209 08:57:53.984015 21227 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.49.84:46307
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765270668420570-20677-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765270668420570-20677-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.49.126:36881
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765270668420570-20677-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.49.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.49.126:36881
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765270668420570-20677-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.49.126
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21227
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765270668420570-20677-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 468897755969d686fb08386f42f7835685e7953a
build type DEBUG
built by None at 09 Dec 2025 08:43:15 UTC on 24a791456cd2
build id 9426
I20251209 08:57:53.985170 21227 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251209 08:57:53.986080 21227 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251209 08:57:53.991792 21235 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251209 08:57:53.991940 21232 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251209 08:57:53.992005 21233 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251209 08:57:53.992178 21227 server_base.cc:1047] running on GCE node
I20251209 08:57:53.992668 21227 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251209 08:57:53.993607 21227 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251209 08:57:53.994771 21227 hybrid_clock.cc:648] HybridClock initialized: now 1765270673994737 us; error 55 us; skew 500 ppm
Dec 09 08:57:53 dist-test-slave-8w6c krb5kdc[21211](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765270673, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.49.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251209 08:57:53.997782 21227 init.cc:377] Logged in from keytab as kudu/127.20.49.126@KRBTEST.COM (short username kudu)
I20251209 08:57:53.998894 21227 webserver.cc:492] Webserver started at http://127.20.49.126:34957/ using document root <none> and password file <none>
I20251209 08:57:53.999454 21227 fs_manager.cc:362] Metadata directory not provided
I20251209 08:57:53.999511 21227 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251209 08:57:53.999698 21227 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251209 08:57:54.001498 21227 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765270668420570-20677-0/minicluster-data/master-0/data/instance:
uuid: "8dab6b8209b64a20a2fb576a3f240631"
format_stamp: "Formatted at 2025-12-09 08:57:53 on dist-test-slave-8w6c"
server_key: "7dce2e0a88dbbe18a3f4201f01fd01b2"
server_key_iv: "edf4810b89862f18ae8b905474e67390"
server_key_version: "encryptionkey@0"
I20251209 08:57:54.002066 21227 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765270668420570-20677-0/minicluster-data/master-0/wal/instance:
uuid: "8dab6b8209b64a20a2fb576a3f240631"
format_stamp: "Formatted at 2025-12-09 08:57:53 on dist-test-slave-8w6c"
server_key: "7dce2e0a88dbbe18a3f4201f01fd01b2"
server_key_iv: "edf4810b89862f18ae8b905474e67390"
server_key_version: "encryptionkey@0"
I20251209 08:57:54.005648 21227 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20251209 08:57:54.008052 21242 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251209 08:57:54.009227 21227 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20251209 08:57:54.009367 21227 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765270668420570-20677-0/minicluster-data/master-0/data,/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765270668420570-20677-0/minicluster-data/master-0/wal
uuid: "8dab6b8209b64a20a2fb576a3f240631"
format_stamp: "Formatted at 2025-12-09 08:57:53 on dist-test-slave-8w6c"
server_key: "7dce2e0a88dbbe18a3f4201f01fd01b2"
server_key_iv: "edf4810b89862f18ae8b905474e67390"
server_key_version: "encryptionkey@0"
I20251209 08:57:54.009467 21227 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765270668420570-20677-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765270668420570-20677-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765270668420570-20677-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251209 08:57:54.026652 21227 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251209 08:57:54.030287 21227 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251209 08:57:54.030496 21227 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251209 08:57:54.038537 21227 rpc_server.cc:307] RPC server started. Bound to: 127.20.49.126:36881
I20251209 08:57:54.038543 21294 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.49.126:36881 every 8 connection(s)
I20251209 08:57:54.039664 21227 server_base.cc:1179] Dumped server information to /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765270668420570-20677-0/minicluster-data/master-0/data/info.pb
I20251209 08:57:54.042819 21295 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251209 08:57:54.048468 20677 external_mini_cluster.cc:1428] Started /tmp/dist-test-task8XuDZV/build/debug/bin/kudu as pid 21227
I20251209 08:57:54.048604 20677 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765270668420570-20677-0/minicluster-data/master-0/wal/instance
I20251209 08:57:54.048841 20677 external_mini_cluster.cc:1468] Setting key 57e40420a2f1943289de0a352bd72b98
I20251209 08:57:54.048914 21295 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8dab6b8209b64a20a2fb576a3f240631: Bootstrap starting.
I20251209 08:57:54.051687 21295 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8dab6b8209b64a20a2fb576a3f240631: Neither blocks nor log segments found. Creating new log.
I20251209 08:57:54.052600 21295 log.cc:826] T 00000000000000000000000000000000 P 8dab6b8209b64a20a2fb576a3f240631: Log is configured to *not* fsync() on all Append() calls
I20251209 08:57:54.054576 21295 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8dab6b8209b64a20a2fb576a3f240631: No bootstrap required, opened a new log
Dec 09 08:57:54 dist-test-slave-8w6c krb5kdc[21211](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765270673, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.49.126@KRBTEST.COM
I20251209 08:57:54.057744 21295 raft_consensus.cc:359] T 00000000000000000000000000000000 P 8dab6b8209b64a20a2fb576a3f240631 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8dab6b8209b64a20a2fb576a3f240631" member_type: VOTER last_known_addr { host: "127.20.49.126" port: 36881 } }
I20251209 08:57:54.058002 21295 raft_consensus.cc:385] T 00000000000000000000000000000000 P 8dab6b8209b64a20a2fb576a3f240631 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251209 08:57:54.058075 21295 raft_consensus.cc:740] T 00000000000000000000000000000000 P 8dab6b8209b64a20a2fb576a3f240631 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8dab6b8209b64a20a2fb576a3f240631, State: Initialized, Role: FOLLOWER
I20251209 08:57:54.058612 21295 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8dab6b8209b64a20a2fb576a3f240631 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8dab6b8209b64a20a2fb576a3f240631" member_type: VOTER last_known_addr { host: "127.20.49.126" port: 36881 } }
I20251209 08:57:54.058750 21295 raft_consensus.cc:399] T 00000000000000000000000000000000 P 8dab6b8209b64a20a2fb576a3f240631 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251209 08:57:54.058828 21295 raft_consensus.cc:493] T 00000000000000000000000000000000 P 8dab6b8209b64a20a2fb576a3f240631 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251209 08:57:54.058940 21295 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 8dab6b8209b64a20a2fb576a3f240631 [term 0 FOLLOWER]: Advancing to term 1
I20251209 08:57:54.060159 21295 raft_consensus.cc:515] T 00000000000000000000000000000000 P 8dab6b8209b64a20a2fb576a3f240631 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8dab6b8209b64a20a2fb576a3f240631" member_type: VOTER last_known_addr { host: "127.20.49.126" port: 36881 } }
I20251209 08:57:54.060555 21295 leader_election.cc:304] T 00000000000000000000000000000000 P 8dab6b8209b64a20a2fb576a3f240631 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8dab6b8209b64a20a2fb576a3f240631; no voters:
I20251209 08:57:54.060920 21295 leader_election.cc:290] T 00000000000000000000000000000000 P 8dab6b8209b64a20a2fb576a3f240631 [CANDIDATE]: Term 1 election: Requested vote from peers
I20251209 08:57:54.061376 21300 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 8dab6b8209b64a20a2fb576a3f240631 [term 1 FOLLOWER]: Leader election won for term 1
I20251209 08:57:54.061607 21300 raft_consensus.cc:697] T 00000000000000000000000000000000 P 8dab6b8209b64a20a2fb576a3f240631 [term 1 LEADER]: Becoming Leader. State: Replica: 8dab6b8209b64a20a2fb576a3f240631, State: Running, Role: LEADER
I20251209 08:57:54.061975 21300 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8dab6b8209b64a20a2fb576a3f240631 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8dab6b8209b64a20a2fb576a3f240631" member_type: VOTER last_known_addr { host: "127.20.49.126" port: 36881 } }
I20251209 08:57:54.062270 21295 sys_catalog.cc:565] T 00000000000000000000000000000000 P 8dab6b8209b64a20a2fb576a3f240631 [sys.catalog]: configured and running, proceeding with master startup.
I20251209 08:57:54.063339 21302 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8dab6b8209b64a20a2fb576a3f240631 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8dab6b8209b64a20a2fb576a3f240631" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8dab6b8209b64a20a2fb576a3f240631" member_type: VOTER last_known_addr { host: "127.20.49.126" port: 36881 } } }
I20251209 08:57:54.063338 21300 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8dab6b8209b64a20a2fb576a3f240631 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8dab6b8209b64a20a2fb576a3f240631. Latest consensus state: current_term: 1 leader_uuid: "8dab6b8209b64a20a2fb576a3f240631" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8dab6b8209b64a20a2fb576a3f240631" member_type: VOTER last_known_addr { host: "127.20.49.126" port: 36881 } } }
I20251209 08:57:54.063427 21298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:57:54.050170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55744 (local address 127.20.49.126:36881)
1209 08:57:54.050615 (+ 445us) server_negotiation.cc:207] Beginning negotiation
1209 08:57:54.050627 (+ 12us) server_negotiation.cc:400] Waiting for connection header
1209 08:57:54.050684 (+ 57us) server_negotiation.cc:408] Connection header received
1209 08:57:54.051443 (+ 759us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1209 08:57:54.051470 (+ 27us) server_negotiation.cc:462] Received NEGOTIATE request from client
1209 08:57:54.051869 (+ 399us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1209 08:57:54.052269 (+ 400us) server_negotiation.cc:227] Negotiated authn=SASL
1209 08:57:54.053194 (+ 925us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:57:54.054007 (+ 813us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1209 08:57:54.054700 (+ 693us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:57:54.055009 (+ 309us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:57:54.057480 (+ 2471us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1209 08:57:54.057504 (+ 24us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1209 08:57:54.057518 (+ 14us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1209 08:57:54.057564 (+ 46us) server_negotiation.cc:923] calling sasl_server_start()
1209 08:57:54.060085 (+ 2521us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1209 08:57:54.060652 (+ 567us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1209 08:57:54.060660 (+ 8us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1209 08:57:54.060666 (+ 6us) server_negotiation.cc:968] calling sasl_server_step()
1209 08:57:54.060774 (+ 108us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1209 08:57:54.061167 (+ 393us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1209 08:57:54.061170 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1209 08:57:54.061172 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1209 08:57:54.061632 (+ 460us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1209 08:57:54.061885 (+ 253us) server_negotiation.cc:1036] Waiting for connection context
1209 08:57:54.062253 (+ 368us) server_negotiation.cc:300] Negotiation successful
1209 08:57:54.062551 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":296,"thread_start_us":129,"threads_started":1}
I20251209 08:57:54.063483 21302 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8dab6b8209b64a20a2fb576a3f240631 [sys.catalog]: This master's current role is: LEADER
I20251209 08:57:54.063999 21309 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251209 08:57:54.063483 21300 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8dab6b8209b64a20a2fb576a3f240631 [sys.catalog]: This master's current role is: LEADER
I20251209 08:57:54.067122 21309 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251209 08:57:54.072885 21309 catalog_manager.cc:1357] Generated new cluster ID: fac9c9d5097248e8a36c8fe630d22af7
I20251209 08:57:54.072969 21309 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251209 08:57:54.084097 21309 catalog_manager.cc:1380] Generated new certificate authority record
I20251209 08:57:54.085057 21309 catalog_manager.cc:1514] Loading token signing keys...
I20251209 08:57:54.096498 21309 catalog_manager.cc:6027] T 00000000000000000000000000000000 P 8dab6b8209b64a20a2fb576a3f240631: Generated new TSK 0
I20251209 08:57:54.097409 21309 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.20.49.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.49.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.49.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765270668420570-20677-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.20.49.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765270668420570-20677-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.20.49.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.20.49.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.20.49.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765270668420570-20677-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.20.49.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765270668420570-20677-0/krb5kdc/kudu.keytab.
I20251209 08:57:54.166249 20677 external_mini_cluster.cc:1366] Running /tmp/dist-test-task8XuDZV/build/debug/bin/kudu
/tmp/dist-test-task8XuDZV/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765270668420570-20677-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765270668420570-20677-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765270668420570-20677-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765270668420570-20677-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.49.65:0
--local_ip_for_outbound_sockets=127.20.49.65
--webserver_interface=127.20.49.65
--webserver_port=0
--tserver_master_addrs=127.20.49.126:36881
--keytab_file=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765270668420570-20677-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.49.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.20.49.84:46307
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765270668420570-20677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765270668420570-20677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765270668420570-20677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251209 08:57:54.288448 21323 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251209 08:57:54.288702 21323 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251209 08:57:54.288766 21323 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251209 08:57:54.292105 21323 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251209 08:57:54.292179 21323 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251209 08:57:54.292268 21323 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.49.65
I20251209 08:57:54.296720 21323 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.49.84:46307
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765270668420570-20677-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765270668420570-20677-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765270668420570-20677-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.49.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.49.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765270668420570-20677-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.49.65
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.20.49.126:36881
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21323
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.49.65
--log_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765270668420570-20677-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 468897755969d686fb08386f42f7835685e7953a
build type DEBUG
built by None at 09 Dec 2025 08:43:15 UTC on 24a791456cd2
build id 9426
I20251209 08:57:54.297881 21323 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251209 08:57:54.298732 21323 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251209 08:57:54.305253 21329 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251209 08:57:54.305249 21331 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251209 08:57:54.305246 21328 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251209 08:57:54.305369 21323 server_base.cc:1047] running on GCE node
I20251209 08:57:54.305970 21323 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251209 08:57:54.306520 21323 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251209 08:57:54.307674 21323 hybrid_clock.cc:648] HybridClock initialized: now 1765270674307662 us; error 33 us; skew 500 ppm
Dec 09 08:57:54 dist-test-slave-8w6c krb5kdc[21211](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765270674, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.49.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251209 08:57:54.310336 21323 init.cc:377] Logged in from keytab as kudu/127.20.49.65@KRBTEST.COM (short username kudu)
I20251209 08:57:54.311390 21323 webserver.cc:492] Webserver started at http://127.20.49.65:38293/ using document root <none> and password file <none>
I20251209 08:57:54.311959 21323 fs_manager.cc:362] Metadata directory not provided
I20251209 08:57:54.312008 21323 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251209 08:57:54.312175 21323 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251209 08:57:54.314201 21323 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765270668420570-20677-0/minicluster-data/ts-0/data/instance:
uuid: "3f47b562197247999dbf34e5fe9f59f4"
format_stamp: "Formatted at 2025-12-09 08:57:54 on dist-test-slave-8w6c"
server_key: "3862368b3cb9676febba387c033ffa35"
server_key_iv: "62d3ba1a9889725ea2c71d435271c851"
server_key_version: "encryptionkey@0"
I20251209 08:57:54.314687 21323 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765270668420570-20677-0/minicluster-data/ts-0/wal/instance:
uuid: "3f47b562197247999dbf34e5fe9f59f4"
format_stamp: "Formatted at 2025-12-09 08:57:54 on dist-test-slave-8w6c"
server_key: "3862368b3cb9676febba387c033ffa35"
server_key_iv: "62d3ba1a9889725ea2c71d435271c851"
server_key_version: "encryptionkey@0"
I20251209 08:57:54.318336 21323 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20251209 08:57:54.320720 21338 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251209 08:57:54.321949 21323 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20251209 08:57:54.322060 21323 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765270668420570-20677-0/minicluster-data/ts-0/data,/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765270668420570-20677-0/minicluster-data/ts-0/wal
uuid: "3f47b562197247999dbf34e5fe9f59f4"
format_stamp: "Formatted at 2025-12-09 08:57:54 on dist-test-slave-8w6c"
server_key: "3862368b3cb9676febba387c033ffa35"
server_key_iv: "62d3ba1a9889725ea2c71d435271c851"
server_key_version: "encryptionkey@0"
I20251209 08:57:54.322173 21323 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765270668420570-20677-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765270668420570-20677-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765270668420570-20677-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251209 08:57:54.342929 21323 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251209 08:57:54.346280 21323 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251209 08:57:54.346483 21323 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251209 08:57:54.347059 21323 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251209 08:57:54.347978 21323 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251209 08:57:54.348030 21323 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251209 08:57:54.348090 21323 ts_tablet_manager.cc:616] Registered 0 tablets
I20251209 08:57:54.348121 21323 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251209 08:57:54.357626 21323 rpc_server.cc:307] RPC server started. Bound to: 127.20.49.65:34415
I20251209 08:57:54.357712 21451 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.49.65:34415 every 8 connection(s)
I20251209 08:57:54.358688 21323 server_base.cc:1179] Dumped server information to /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765270668420570-20677-0/minicluster-data/ts-0/data/info.pb
I20251209 08:57:54.363930 20677 external_mini_cluster.cc:1428] Started /tmp/dist-test-task8XuDZV/build/debug/bin/kudu as pid 21323
I20251209 08:57:54.364051 20677 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765270668420570-20677-0/minicluster-data/ts-0/wal/instance
I20251209 08:57:54.364313 20677 external_mini_cluster.cc:1468] Setting key 12481ca116934d45c19012562915d01f
Dec 09 08:57:54 dist-test-slave-8w6c krb5kdc[21211](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765270674, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.49.65@KRBTEST.COM for kudu/127.20.49.126@KRBTEST.COM
I20251209 08:57:54.372531 21298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:57:54.360542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.49.65:43075 (local address 127.20.49.126:36881)
1209 08:57:54.360682 (+ 140us) server_negotiation.cc:207] Beginning negotiation
1209 08:57:54.360686 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1209 08:57:54.361412 (+ 726us) server_negotiation.cc:408] Connection header received
1209 08:57:54.362281 (+ 869us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1209 08:57:54.362284 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1209 08:57:54.362333 (+ 49us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1209 08:57:54.362415 (+ 82us) server_negotiation.cc:227] Negotiated authn=SASL
1209 08:57:54.364312 (+ 1897us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:57:54.365043 (+ 731us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1209 08:57:54.365802 (+ 759us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:57:54.365999 (+ 197us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:57:54.368707 (+ 2708us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1209 08:57:54.368727 (+ 20us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1209 08:57:54.368729 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1209 08:57:54.368755 (+ 26us) server_negotiation.cc:923] calling sasl_server_start()
1209 08:57:54.370516 (+ 1761us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1209 08:57:54.371056 (+ 540us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1209 08:57:54.371060 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1209 08:57:54.371061 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1209 08:57:54.371110 (+ 49us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1209 08:57:54.371393 (+ 283us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1209 08:57:54.371396 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1209 08:57:54.371398 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1209 08:57:54.371563 (+ 165us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1209 08:57:54.371655 (+ 92us) server_negotiation.cc:1036] Waiting for connection context
1209 08:57:54.372202 (+ 547us) server_negotiation.cc:300] Negotiation successful
1209 08:57:54.372353 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20251209 08:57:54.373209 21454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:57:54.360804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.49.126:36881 (local address 127.20.49.65:43075)
1209 08:57:54.361261 (+ 457us) negotiation.cc:107] Waiting for socket to connect
1209 08:57:54.361298 (+ 37us) client_negotiation.cc:174] Beginning negotiation
1209 08:57:54.362073 (+ 775us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1209 08:57:54.362560 (+ 487us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1209 08:57:54.362568 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
1209 08:57:54.362984 (+ 416us) client_negotiation.cc:189] Negotiated authn=SASL
1209 08:57:54.364139 (+ 1155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:57:54.364158 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:57:54.365176 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1209 08:57:54.365180 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1209 08:57:54.365661 (+ 481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:57:54.365674 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:57:54.365947 (+ 273us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:57:54.366704 (+ 757us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1209 08:57:54.366733 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
1209 08:57:54.368536 (+ 1803us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1209 08:57:54.370659 (+ 2123us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1209 08:57:54.370666 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1209 08:57:54.370680 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
1209 08:57:54.370951 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1209 08:57:54.371198 (+ 247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1209 08:57:54.371202 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1209 08:57:54.371206 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
1209 08:57:54.371303 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1209 08:57:54.371668 (+ 365us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1209 08:57:54.371678 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1209 08:57:54.371941 (+ 263us) client_negotiation.cc:769] Sending connection context
1209 08:57:54.372149 (+ 208us) client_negotiation.cc:240] Negotiation successful
1209 08:57:54.372452 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":113,"threads_started":1}
WARNING: no policy specified for kudu/127.20.49.66@KRBTEST.COM; defaulting to no policy
I20251209 08:57:54.374531 21452 heartbeater.cc:344] Connected to a master server at 127.20.49.126:36881
I20251209 08:57:54.374783 21452 heartbeater.cc:461] Registering TS with master...
I20251209 08:57:54.375375 21452 heartbeater.cc:507] Master 127.20.49.126:36881 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.49.66@KRBTEST.COM" created.
I20251209 08:57:54.376972 21259 ts_manager.cc:194] Registered new tserver with Master: 3f47b562197247999dbf34e5fe9f59f4 (127.20.49.65:34415)
I20251209 08:57:54.378576 21259 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.49.65@KRBTEST.COM'} at 127.20.49.65:43075
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.49.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765270668420570-20677-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.20.49.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765270668420570-20677-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.20.49.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.20.49.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.20.49.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765270668420570-20677-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.20.49.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765270668420570-20677-0/krb5kdc/kudu.keytab.
I20251209 08:57:54.417814 20677 external_mini_cluster.cc:1366] Running /tmp/dist-test-task8XuDZV/build/debug/bin/kudu
/tmp/dist-test-task8XuDZV/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765270668420570-20677-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765270668420570-20677-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765270668420570-20677-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765270668420570-20677-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.49.66:0
--local_ip_for_outbound_sockets=127.20.49.66
--webserver_interface=127.20.49.66
--webserver_port=0
--tserver_master_addrs=127.20.49.126:36881
--keytab_file=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765270668420570-20677-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.49.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.20.49.84:46307
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765270668420570-20677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765270668420570-20677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765270668420570-20677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251209 08:57:54.528497 21459 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251209 08:57:54.528728 21459 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251209 08:57:54.528790 21459 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251209 08:57:54.532214 21459 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251209 08:57:54.532289 21459 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251209 08:57:54.532400 21459 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.49.66
I20251209 08:57:54.536844 21459 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.49.84:46307
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765270668420570-20677-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765270668420570-20677-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765270668420570-20677-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.49.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.49.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765270668420570-20677-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.49.66
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.20.49.126:36881
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21459
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.49.66
--log_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765270668420570-20677-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 468897755969d686fb08386f42f7835685e7953a
build type DEBUG
built by None at 09 Dec 2025 08:43:15 UTC on 24a791456cd2
build id 9426
I20251209 08:57:54.538055 21459 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251209 08:57:54.538906 21459 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251209 08:57:54.545372 21465 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251209 08:57:54.545423 21464 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251209 08:57:54.545560 21459 server_base.cc:1047] running on GCE node
W20251209 08:57:54.545351 21467 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251209 08:57:54.546061 21459 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251209 08:57:54.546597 21459 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251209 08:57:54.547778 21459 hybrid_clock.cc:648] HybridClock initialized: now 1765270674547762 us; error 28 us; skew 500 ppm
Dec 09 08:57:54 dist-test-slave-8w6c krb5kdc[21211](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765270674, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.49.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251209 08:57:54.550522 21459 init.cc:377] Logged in from keytab as kudu/127.20.49.66@KRBTEST.COM (short username kudu)
I20251209 08:57:54.551586 21459 webserver.cc:492] Webserver started at http://127.20.49.66:38077/ using document root <none> and password file <none>
I20251209 08:57:54.552148 21459 fs_manager.cc:362] Metadata directory not provided
I20251209 08:57:54.552204 21459 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251209 08:57:54.552387 21459 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251209 08:57:54.554176 21459 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765270668420570-20677-0/minicluster-data/ts-1/data/instance:
uuid: "4e7232bd790c4224828b7b0b4afa5343"
format_stamp: "Formatted at 2025-12-09 08:57:54 on dist-test-slave-8w6c"
server_key: "66eb32f00d886f6cbb283466363ffb81"
server_key_iv: "e07e8c65f4d2e987d5d4cabf1e2438ff"
server_key_version: "encryptionkey@0"
I20251209 08:57:54.554648 21459 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765270668420570-20677-0/minicluster-data/ts-1/wal/instance:
uuid: "4e7232bd790c4224828b7b0b4afa5343"
format_stamp: "Formatted at 2025-12-09 08:57:54 on dist-test-slave-8w6c"
server_key: "66eb32f00d886f6cbb283466363ffb81"
server_key_iv: "e07e8c65f4d2e987d5d4cabf1e2438ff"
server_key_version: "encryptionkey@0"
I20251209 08:57:54.558009 21459 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20251209 08:57:54.560204 21474 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251209 08:57:54.561204 21459 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20251209 08:57:54.561311 21459 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765270668420570-20677-0/minicluster-data/ts-1/data,/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765270668420570-20677-0/minicluster-data/ts-1/wal
uuid: "4e7232bd790c4224828b7b0b4afa5343"
format_stamp: "Formatted at 2025-12-09 08:57:54 on dist-test-slave-8w6c"
server_key: "66eb32f00d886f6cbb283466363ffb81"
server_key_iv: "e07e8c65f4d2e987d5d4cabf1e2438ff"
server_key_version: "encryptionkey@0"
I20251209 08:57:54.561399 21459 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765270668420570-20677-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765270668420570-20677-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765270668420570-20677-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251209 08:57:54.580860 21459 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251209 08:57:54.584082 21459 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251209 08:57:54.584290 21459 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251209 08:57:54.584879 21459 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251209 08:57:54.585901 21459 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251209 08:57:54.585958 21459 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251209 08:57:54.586022 21459 ts_tablet_manager.cc:616] Registered 0 tablets
I20251209 08:57:54.586068 21459 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251209 08:57:54.598390 21459 rpc_server.cc:307] RPC server started. Bound to: 127.20.49.66:39157
I20251209 08:57:54.598412 21587 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.49.66:39157 every 8 connection(s)
I20251209 08:57:54.599434 21459 server_base.cc:1179] Dumped server information to /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765270668420570-20677-0/minicluster-data/ts-1/data/info.pb
I20251209 08:57:54.604560 20677 external_mini_cluster.cc:1428] Started /tmp/dist-test-task8XuDZV/build/debug/bin/kudu as pid 21459
I20251209 08:57:54.604655 20677 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765270668420570-20677-0/minicluster-data/ts-1/wal/instance
I20251209 08:57:54.604935 20677 external_mini_cluster.cc:1468] Setting key 4cc118da27a2454691021e4c1c15d1ab
Dec 09 08:57:54 dist-test-slave-8w6c krb5kdc[21211](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765270674, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.49.66@KRBTEST.COM for kudu/127.20.49.126@KRBTEST.COM
I20251209 08:57:54.612986 21298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:57:54.601343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.49.66:34543 (local address 127.20.49.126:36881)
1209 08:57:54.601535 (+ 192us) server_negotiation.cc:207] Beginning negotiation
1209 08:57:54.601540 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1209 08:57:54.602299 (+ 759us) server_negotiation.cc:408] Connection header received
1209 08:57:54.603189 (+ 890us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1209 08:57:54.603192 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1209 08:57:54.603248 (+ 56us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1209 08:57:54.603341 (+ 93us) server_negotiation.cc:227] Negotiated authn=SASL
1209 08:57:54.605222 (+ 1881us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:57:54.605884 (+ 662us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1209 08:57:54.606610 (+ 726us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:57:54.606796 (+ 186us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:57:54.609089 (+ 2293us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1209 08:57:54.609118 (+ 29us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1209 08:57:54.609125 (+ 7us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1209 08:57:54.609160 (+ 35us) server_negotiation.cc:923] calling sasl_server_start()
1209 08:57:54.610869 (+ 1709us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1209 08:57:54.611388 (+ 519us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1209 08:57:54.611391 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1209 08:57:54.611393 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1209 08:57:54.611437 (+ 44us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1209 08:57:54.611726 (+ 289us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1209 08:57:54.611729 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1209 08:57:54.611731 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1209 08:57:54.611993 (+ 262us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1209 08:57:54.612118 (+ 125us) server_negotiation.cc:1036] Waiting for connection context
1209 08:57:54.612663 (+ 545us) server_negotiation.cc:300] Negotiation successful
1209 08:57:54.612801 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20251209 08:57:54.613660 21590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:57:54.601623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.49.126:36881 (local address 127.20.49.66:34543)
1209 08:57:54.602152 (+ 529us) negotiation.cc:107] Waiting for socket to connect
1209 08:57:54.602192 (+ 40us) client_negotiation.cc:174] Beginning negotiation
1209 08:57:54.602969 (+ 777us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1209 08:57:54.603543 (+ 574us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1209 08:57:54.603552 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
1209 08:57:54.603995 (+ 443us) client_negotiation.cc:189] Negotiated authn=SASL
1209 08:57:54.605049 (+ 1054us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:57:54.605065 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:57:54.606006 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1209 08:57:54.606011 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1209 08:57:54.606489 (+ 478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:57:54.606498 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:57:54.606713 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:57:54.607370 (+ 657us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1209 08:57:54.607394 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
1209 08:57:54.608933 (+ 1539us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1209 08:57:54.611010 (+ 2077us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1209 08:57:54.611018 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1209 08:57:54.611028 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
1209 08:57:54.611269 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1209 08:57:54.611527 (+ 258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1209 08:57:54.611532 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1209 08:57:54.611534 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1209 08:57:54.611627 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1209 08:57:54.612131 (+ 504us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1209 08:57:54.612162 (+ 31us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1209 08:57:54.612404 (+ 242us) client_negotiation.cc:769] Sending connection context
1209 08:57:54.612609 (+ 205us) client_negotiation.cc:240] Negotiation successful
1209 08:57:54.612895 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":342,"thread_start_us":132,"threads_started":1}
WARNING: no policy specified for kudu/127.20.49.67@KRBTEST.COM; defaulting to no policy
I20251209 08:57:54.614943 21588 heartbeater.cc:344] Connected to a master server at 127.20.49.126:36881
I20251209 08:57:54.615226 21588 heartbeater.cc:461] Registering TS with master...
I20251209 08:57:54.615754 21588 heartbeater.cc:507] Master 127.20.49.126:36881 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.49.67@KRBTEST.COM" created.
I20251209 08:57:54.616904 21259 ts_manager.cc:194] Registered new tserver with Master: 4e7232bd790c4224828b7b0b4afa5343 (127.20.49.66:39157)
I20251209 08:57:54.617736 21259 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.49.66@KRBTEST.COM'} at 127.20.49.66:34543
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.49.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765270668420570-20677-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.20.49.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765270668420570-20677-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.20.49.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.20.49.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.20.49.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765270668420570-20677-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.20.49.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765270668420570-20677-0/krb5kdc/kudu.keytab.
I20251209 08:57:54.656562 20677 external_mini_cluster.cc:1366] Running /tmp/dist-test-task8XuDZV/build/debug/bin/kudu
/tmp/dist-test-task8XuDZV/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765270668420570-20677-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765270668420570-20677-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765270668420570-20677-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765270668420570-20677-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.49.67:0
--local_ip_for_outbound_sockets=127.20.49.67
--webserver_interface=127.20.49.67
--webserver_port=0
--tserver_master_addrs=127.20.49.126:36881
--keytab_file=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765270668420570-20677-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.49.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.20.49.84:46307
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765270668420570-20677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765270668420570-20677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765270668420570-20677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251209 08:57:54.767581 21595 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251209 08:57:54.767831 21595 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251209 08:57:54.767901 21595 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251209 08:57:54.771170 21595 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251209 08:57:54.771234 21595 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251209 08:57:54.771317 21595 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.49.67
I20251209 08:57:54.775547 21595 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.49.84:46307
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765270668420570-20677-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765270668420570-20677-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765270668420570-20677-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.49.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.49.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765270668420570-20677-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.49.67
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.20.49.126:36881
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21595
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.49.67
--log_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765270668420570-20677-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 468897755969d686fb08386f42f7835685e7953a
build type DEBUG
built by None at 09 Dec 2025 08:43:15 UTC on 24a791456cd2
build id 9426
I20251209 08:57:54.776580 21595 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251209 08:57:54.777422 21595 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251209 08:57:54.783633 21603 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251209 08:57:54.783674 21601 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251209 08:57:54.783677 21600 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251209 08:57:54.784081 21595 server_base.cc:1047] running on GCE node
I20251209 08:57:54.784390 21595 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251209 08:57:54.784899 21595 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251209 08:57:54.786046 21595 hybrid_clock.cc:648] HybridClock initialized: now 1765270674786038 us; error 32 us; skew 500 ppm
Dec 09 08:57:54 dist-test-slave-8w6c krb5kdc[21211](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765270674, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.49.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251209 08:57:54.788676 21595 init.cc:377] Logged in from keytab as kudu/127.20.49.67@KRBTEST.COM (short username kudu)
I20251209 08:57:54.789726 21595 webserver.cc:492] Webserver started at http://127.20.49.67:44919/ using document root <none> and password file <none>
I20251209 08:57:54.790287 21595 fs_manager.cc:362] Metadata directory not provided
I20251209 08:57:54.790342 21595 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251209 08:57:54.790521 21595 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251209 08:57:54.792268 21595 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765270668420570-20677-0/minicluster-data/ts-2/data/instance:
uuid: "3fa3633f63ba443d832f1dfee97d6eb9"
format_stamp: "Formatted at 2025-12-09 08:57:54 on dist-test-slave-8w6c"
server_key: "b55b77ea4b4fd95a8184aaae60fb4105"
server_key_iv: "7ea841148d47cf50db4df8f077ac1718"
server_key_version: "encryptionkey@0"
I20251209 08:57:54.792739 21595 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765270668420570-20677-0/minicluster-data/ts-2/wal/instance:
uuid: "3fa3633f63ba443d832f1dfee97d6eb9"
format_stamp: "Formatted at 2025-12-09 08:57:54 on dist-test-slave-8w6c"
server_key: "b55b77ea4b4fd95a8184aaae60fb4105"
server_key_iv: "7ea841148d47cf50db4df8f077ac1718"
server_key_version: "encryptionkey@0"
I20251209 08:57:54.796310 21595 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20251209 08:57:54.798660 21610 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251209 08:57:54.799721 21595 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20251209 08:57:54.799853 21595 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765270668420570-20677-0/minicluster-data/ts-2/data,/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765270668420570-20677-0/minicluster-data/ts-2/wal
uuid: "3fa3633f63ba443d832f1dfee97d6eb9"
format_stamp: "Formatted at 2025-12-09 08:57:54 on dist-test-slave-8w6c"
server_key: "b55b77ea4b4fd95a8184aaae60fb4105"
server_key_iv: "7ea841148d47cf50db4df8f077ac1718"
server_key_version: "encryptionkey@0"
I20251209 08:57:54.799943 21595 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765270668420570-20677-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765270668420570-20677-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765270668420570-20677-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251209 08:57:54.812865 21595 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251209 08:57:54.815976 21595 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251209 08:57:54.816170 21595 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251209 08:57:54.816725 21595 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251209 08:57:54.817608 21595 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251209 08:57:54.817663 21595 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251209 08:57:54.817765 21595 ts_tablet_manager.cc:616] Registered 0 tablets
I20251209 08:57:54.817803 21595 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251209 08:57:54.827416 21595 rpc_server.cc:307] RPC server started. Bound to: 127.20.49.67:36801
I20251209 08:57:54.827436 21723 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.49.67:36801 every 8 connection(s)
I20251209 08:57:54.828406 21595 server_base.cc:1179] Dumped server information to /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765270668420570-20677-0/minicluster-data/ts-2/data/info.pb
I20251209 08:57:54.833184 20677 external_mini_cluster.cc:1428] Started /tmp/dist-test-task8XuDZV/build/debug/bin/kudu as pid 21595
I20251209 08:57:54.833300 20677 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1765270668420570-20677-0/minicluster-data/ts-2/wal/instance
I20251209 08:57:54.833521 20677 external_mini_cluster.cc:1468] Setting key 9f715dc06165f370abae80844ad16b2f
Dec 09 08:57:54 dist-test-slave-8w6c krb5kdc[21211](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765270674, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.49.67@KRBTEST.COM for kudu/127.20.49.126@KRBTEST.COM
I20251209 08:57:54.841653 21298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:57:54.830263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.49.67:49269 (local address 127.20.49.126:36881)
1209 08:57:54.830397 (+ 134us) server_negotiation.cc:207] Beginning negotiation
1209 08:57:54.830401 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1209 08:57:54.831095 (+ 694us) server_negotiation.cc:408] Connection header received
1209 08:57:54.832068 (+ 973us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1209 08:57:54.832072 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1209 08:57:54.832137 (+ 65us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1209 08:57:54.832221 (+ 84us) server_negotiation.cc:227] Negotiated authn=SASL
1209 08:57:54.834120 (+ 1899us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:57:54.834811 (+ 691us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1209 08:57:54.835679 (+ 868us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:57:54.835860 (+ 181us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:57:54.838227 (+ 2367us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1209 08:57:54.838242 (+ 15us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1209 08:57:54.838244 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1209 08:57:54.838270 (+ 26us) server_negotiation.cc:923] calling sasl_server_start()
1209 08:57:54.839739 (+ 1469us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1209 08:57:54.840287 (+ 548us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1209 08:57:54.840291 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1209 08:57:54.840292 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1209 08:57:54.840335 (+ 43us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1209 08:57:54.840601 (+ 266us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1209 08:57:54.840605 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1209 08:57:54.840607 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1209 08:57:54.840761 (+ 154us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1209 08:57:54.840852 (+ 91us) server_negotiation.cc:1036] Waiting for connection context
1209 08:57:54.841355 (+ 503us) server_negotiation.cc:300] Negotiation successful
1209 08:57:54.841479 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20251209 08:57:54.842336 21726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:57:54.830504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.49.126:36881 (local address 127.20.49.67:49269)
1209 08:57:54.830943 (+ 439us) negotiation.cc:107] Waiting for socket to connect
1209 08:57:54.830980 (+ 37us) client_negotiation.cc:174] Beginning negotiation
1209 08:57:54.831832 (+ 852us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1209 08:57:54.832397 (+ 565us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1209 08:57:54.832405 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
1209 08:57:54.832807 (+ 402us) client_negotiation.cc:189] Negotiated authn=SASL
1209 08:57:54.833894 (+ 1087us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:57:54.833957 (+ 63us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:57:54.834975 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1209 08:57:54.834979 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1209 08:57:54.835557 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:57:54.835566 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:57:54.835792 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:57:54.836455 (+ 663us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1209 08:57:54.836479 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
1209 08:57:54.838056 (+ 1577us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1209 08:57:54.839879 (+ 1823us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1209 08:57:54.839886 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1209 08:57:54.839933 (+ 47us) client_negotiation.cc:760] calling sasl_client_step()
1209 08:57:54.840182 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1209 08:57:54.840420 (+ 238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1209 08:57:54.840424 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1209 08:57:54.840425 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1209 08:57:54.840514 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1209 08:57:54.840865 (+ 351us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1209 08:57:54.840873 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1209 08:57:54.841112 (+ 239us) client_negotiation.cc:769] Sending connection context
1209 08:57:54.841311 (+ 199us) client_negotiation.cc:240] Negotiation successful
1209 08:57:54.841591 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":108,"threads_started":1}
I20251209 08:57:54.843461 21724 heartbeater.cc:344] Connected to a master server at 127.20.49.126:36881
I20251209 08:57:54.843716 21724 heartbeater.cc:461] Registering TS with master...
I20251209 08:57:54.844254 21724 heartbeater.cc:507] Master 127.20.49.126:36881 requested a full tablet report, sending...
I20251209 08:57:54.845261 21259 ts_manager.cc:194] Registered new tserver with Master: 3fa3633f63ba443d832f1dfee97d6eb9 (127.20.49.67:36801)
I20251209 08:57:54.845857 21259 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.49.67@KRBTEST.COM'} at 127.20.49.67:49269
I20251209 08:57:54.847420 20677 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20251209 08:57:54.857448 21298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:57:54.850328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55750 (local address 127.20.49.126:36881)
1209 08:57:54.850469 (+ 141us) server_negotiation.cc:207] Beginning negotiation
1209 08:57:54.850474 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1209 08:57:54.850559 (+ 85us) server_negotiation.cc:408] Connection header received
1209 08:57:54.850697 (+ 138us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1209 08:57:54.850701 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1209 08:57:54.850748 (+ 47us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1209 08:57:54.850820 (+ 72us) server_negotiation.cc:227] Negotiated authn=SASL
1209 08:57:54.851714 (+ 894us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:57:54.852214 (+ 500us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1209 08:57:54.852872 (+ 658us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:57:54.853086 (+ 214us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:57:54.854281 (+ 1195us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1209 08:57:54.854298 (+ 17us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1209 08:57:54.854300 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1209 08:57:54.854325 (+ 25us) server_negotiation.cc:923] calling sasl_server_start()
1209 08:57:54.855795 (+ 1470us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1209 08:57:54.856244 (+ 449us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1209 08:57:54.856247 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1209 08:57:54.856249 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1209 08:57:54.856293 (+ 44us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1209 08:57:54.856504 (+ 211us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1209 08:57:54.856507 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1209 08:57:54.856509 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1209 08:57:54.856751 (+ 242us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1209 08:57:54.856955 (+ 204us) server_negotiation.cc:1036] Waiting for connection context
1209 08:57:54.857157 (+ 202us) server_negotiation.cc:300] Negotiation successful
1209 08:57:54.857278 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20251209 08:57:54.861972 21259 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55750:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251209 08:57:54.864943 21259 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251209 08:57:54.878202 21737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:57:54.873753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.49.65:34415 (local address 127.0.0.1:54782)
1209 08:57:54.874529 (+ 776us) negotiation.cc:107] Waiting for socket to connect
1209 08:57:54.874569 (+ 40us) client_negotiation.cc:174] Beginning negotiation
1209 08:57:54.874719 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1209 08:57:54.875305 (+ 586us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1209 08:57:54.875311 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
1209 08:57:54.875333 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1209 08:57:54.875623 (+ 290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:57:54.875631 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:57:54.876667 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1209 08:57:54.876671 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1209 08:57:54.877658 (+ 987us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:57:54.877666 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:57:54.877795 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:57:54.877867 (+ 72us) client_negotiation.cc:769] Sending connection context
1209 08:57:54.877962 (+ 95us) client_negotiation.cc:240] Negotiation successful
1209 08:57:54.878048 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":619,"thread_start_us":95,"threads_started":1}
I20251209 08:57:54.878623 21735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:57:54.873938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.49.67:36801 (local address 127.0.0.1:59902)
1209 08:57:54.874945 (+ 1007us) negotiation.cc:107] Waiting for socket to connect
1209 08:57:54.874961 (+ 16us) client_negotiation.cc:174] Beginning negotiation
1209 08:57:54.875042 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1209 08:57:54.875725 (+ 683us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1209 08:57:54.875731 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
1209 08:57:54.875750 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1209 08:57:54.876022 (+ 272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:57:54.876031 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:57:54.877375 (+ 1344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1209 08:57:54.877379 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1209 08:57:54.878270 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:57:54.878283 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:57:54.878393 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:57:54.878409 (+ 16us) client_negotiation.cc:769] Sending connection context
1209 08:57:54.878461 (+ 52us) client_negotiation.cc:240] Negotiation successful
1209 08:57:54.878515 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":929,"thread_start_us":91,"threads_started":1}
I20251209 08:57:54.878979 21739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:57:54.874837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.49.66:39157 (local address 127.0.0.1:40046)
1209 08:57:54.875608 (+ 771us) negotiation.cc:107] Waiting for socket to connect
1209 08:57:54.875623 (+ 15us) client_negotiation.cc:174] Beginning negotiation
1209 08:57:54.875698 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1209 08:57:54.876388 (+ 690us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1209 08:57:54.876392 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
1209 08:57:54.876411 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1209 08:57:54.876641 (+ 230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:57:54.876651 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:57:54.877747 (+ 1096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1209 08:57:54.877751 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1209 08:57:54.878605 (+ 854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:57:54.878614 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:57:54.878726 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:57:54.878749 (+ 23us) client_negotiation.cc:769] Sending connection context
1209 08:57:54.878799 (+ 50us) client_negotiation.cc:240] Negotiation successful
1209 08:57:54.878854 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":697,"thread_start_us":100,"threads_started":1}
I20251209 08:57:54.879655 21736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:57:54.873919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54782 (local address 127.20.49.65:34415)
1209 08:57:54.874347 (+ 428us) server_negotiation.cc:207] Beginning negotiation
1209 08:57:54.874356 (+ 9us) server_negotiation.cc:400] Waiting for connection header
1209 08:57:54.874789 (+ 433us) server_negotiation.cc:408] Connection header received
1209 08:57:54.874890 (+ 101us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1209 08:57:54.874896 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
1209 08:57:54.875108 (+ 212us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1209 08:57:54.875303 (+ 195us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1209 08:57:54.875747 (+ 444us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:57:54.876545 (+ 798us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1209 08:57:54.878580 (+ 2035us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:57:54.879262 (+ 682us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:57:54.879379 (+ 117us) server_negotiation.cc:1036] Waiting for connection context
1209 08:57:54.879445 (+ 66us) server_negotiation.cc:300] Negotiation successful
1209 08:57:54.879530 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":317,"thread_start_us":93,"threads_started":1}
I20251209 08:57:54.879665 21738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:57:54.874237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59902 (local address 127.20.49.67:36801)
1209 08:57:54.874670 (+ 433us) server_negotiation.cc:207] Beginning negotiation
1209 08:57:54.874677 (+ 7us) server_negotiation.cc:400] Waiting for connection header
1209 08:57:54.875152 (+ 475us) server_negotiation.cc:408] Connection header received
1209 08:57:54.875246 (+ 94us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1209 08:57:54.875252 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
1209 08:57:54.875425 (+ 173us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1209 08:57:54.875564 (+ 139us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1209 08:57:54.876166 (+ 602us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:57:54.877241 (+ 1075us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1209 08:57:54.878803 (+ 1562us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:57:54.879300 (+ 497us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:57:54.879378 (+ 78us) server_negotiation.cc:1036] Waiting for connection context
1209 08:57:54.879445 (+ 67us) server_negotiation.cc:300] Negotiation successful
1209 08:57:54.879529 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":330,"thread_start_us":159,"threads_started":1}
I20251209 08:57:54.879789 21740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:57:54.875251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40046 (local address 127.20.49.66:39157)
1209 08:57:54.875967 (+ 716us) server_negotiation.cc:207] Beginning negotiation
1209 08:57:54.875973 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1209 08:57:54.875994 (+ 21us) server_negotiation.cc:408] Connection header received
1209 08:57:54.876076 (+ 82us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1209 08:57:54.876083 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
1209 08:57:54.876252 (+ 169us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1209 08:57:54.876384 (+ 132us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1209 08:57:54.876764 (+ 380us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:57:54.877606 (+ 842us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1209 08:57:54.878889 (+ 1283us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:57:54.879469 (+ 580us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:57:54.879548 (+ 79us) server_negotiation.cc:1036] Waiting for connection context
1209 08:57:54.879597 (+ 49us) server_negotiation.cc:300] Negotiation successful
1209 08:57:54.879656 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":613,"thread_start_us":118,"threads_started":1}
I20251209 08:57:54.882408 21386 tablet_service.cc:1505] Processing CreateTablet for tablet 2300e8a9f70c43e480f6fbdf53e35e3e (DEFAULT_TABLE table=test-table [id=e0e466f2fc4140069cafa4d152436046]), partition=RANGE (key) PARTITION UNBOUNDED
I20251209 08:57:54.882459 21522 tablet_service.cc:1505] Processing CreateTablet for tablet 2300e8a9f70c43e480f6fbdf53e35e3e (DEFAULT_TABLE table=test-table [id=e0e466f2fc4140069cafa4d152436046]), partition=RANGE (key) PARTITION UNBOUNDED
I20251209 08:57:54.883280 21386 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2300e8a9f70c43e480f6fbdf53e35e3e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251209 08:57:54.883350 21522 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2300e8a9f70c43e480f6fbdf53e35e3e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251209 08:57:54.883433 21658 tablet_service.cc:1505] Processing CreateTablet for tablet 2300e8a9f70c43e480f6fbdf53e35e3e (DEFAULT_TABLE table=test-table [id=e0e466f2fc4140069cafa4d152436046]), partition=RANGE (key) PARTITION UNBOUNDED
I20251209 08:57:54.884231 21658 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2300e8a9f70c43e480f6fbdf53e35e3e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251209 08:57:54.888724 21741 tablet_bootstrap.cc:492] T 2300e8a9f70c43e480f6fbdf53e35e3e P 3f47b562197247999dbf34e5fe9f59f4: Bootstrap starting.
I20251209 08:57:54.890662 21742 tablet_bootstrap.cc:492] T 2300e8a9f70c43e480f6fbdf53e35e3e P 4e7232bd790c4224828b7b0b4afa5343: Bootstrap starting.
I20251209 08:57:54.890662 21743 tablet_bootstrap.cc:492] T 2300e8a9f70c43e480f6fbdf53e35e3e P 3fa3633f63ba443d832f1dfee97d6eb9: Bootstrap starting.
I20251209 08:57:54.891252 21741 tablet_bootstrap.cc:654] T 2300e8a9f70c43e480f6fbdf53e35e3e P 3f47b562197247999dbf34e5fe9f59f4: Neither blocks nor log segments found. Creating new log.
I20251209 08:57:54.891921 21741 log.cc:826] T 2300e8a9f70c43e480f6fbdf53e35e3e P 3f47b562197247999dbf34e5fe9f59f4: Log is configured to *not* fsync() on all Append() calls
I20251209 08:57:54.892680 21743 tablet_bootstrap.cc:654] T 2300e8a9f70c43e480f6fbdf53e35e3e P 3fa3633f63ba443d832f1dfee97d6eb9: Neither blocks nor log segments found. Creating new log.
I20251209 08:57:54.892680 21742 tablet_bootstrap.cc:654] T 2300e8a9f70c43e480f6fbdf53e35e3e P 4e7232bd790c4224828b7b0b4afa5343: Neither blocks nor log segments found. Creating new log.
I20251209 08:57:54.893493 21742 log.cc:826] T 2300e8a9f70c43e480f6fbdf53e35e3e P 4e7232bd790c4224828b7b0b4afa5343: Log is configured to *not* fsync() on all Append() calls
I20251209 08:57:54.893493 21743 log.cc:826] T 2300e8a9f70c43e480f6fbdf53e35e3e P 3fa3633f63ba443d832f1dfee97d6eb9: Log is configured to *not* fsync() on all Append() calls
I20251209 08:57:54.893545 21741 tablet_bootstrap.cc:492] T 2300e8a9f70c43e480f6fbdf53e35e3e P 3f47b562197247999dbf34e5fe9f59f4: No bootstrap required, opened a new log
I20251209 08:57:54.893785 21741 ts_tablet_manager.cc:1403] T 2300e8a9f70c43e480f6fbdf53e35e3e P 3f47b562197247999dbf34e5fe9f59f4: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20251209 08:57:54.895171 21742 tablet_bootstrap.cc:492] T 2300e8a9f70c43e480f6fbdf53e35e3e P 4e7232bd790c4224828b7b0b4afa5343: No bootstrap required, opened a new log
I20251209 08:57:54.895346 21742 ts_tablet_manager.cc:1403] T 2300e8a9f70c43e480f6fbdf53e35e3e P 4e7232bd790c4224828b7b0b4afa5343: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20251209 08:57:54.895612 21743 tablet_bootstrap.cc:492] T 2300e8a9f70c43e480f6fbdf53e35e3e P 3fa3633f63ba443d832f1dfee97d6eb9: No bootstrap required, opened a new log
I20251209 08:57:54.895799 21743 ts_tablet_manager.cc:1403] T 2300e8a9f70c43e480f6fbdf53e35e3e P 3fa3633f63ba443d832f1dfee97d6eb9: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20251209 08:57:54.896776 21741 raft_consensus.cc:359] T 2300e8a9f70c43e480f6fbdf53e35e3e P 3f47b562197247999dbf34e5fe9f59f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f47b562197247999dbf34e5fe9f59f4" member_type: VOTER last_known_addr { host: "127.20.49.65" port: 34415 } } peers { permanent_uuid: "3fa3633f63ba443d832f1dfee97d6eb9" member_type: VOTER last_known_addr { host: "127.20.49.67" port: 36801 } } peers { permanent_uuid: "4e7232bd790c4224828b7b0b4afa5343" member_type: VOTER last_known_addr { host: "127.20.49.66" port: 39157 } }
I20251209 08:57:54.897015 21741 raft_consensus.cc:385] T 2300e8a9f70c43e480f6fbdf53e35e3e P 3f47b562197247999dbf34e5fe9f59f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251209 08:57:54.897081 21741 raft_consensus.cc:740] T 2300e8a9f70c43e480f6fbdf53e35e3e P 3f47b562197247999dbf34e5fe9f59f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f47b562197247999dbf34e5fe9f59f4, State: Initialized, Role: FOLLOWER
I20251209 08:57:54.897524 21741 consensus_queue.cc:260] T 2300e8a9f70c43e480f6fbdf53e35e3e P 3f47b562197247999dbf34e5fe9f59f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f47b562197247999dbf34e5fe9f59f4" member_type: VOTER last_known_addr { host: "127.20.49.65" port: 34415 } } peers { permanent_uuid: "3fa3633f63ba443d832f1dfee97d6eb9" member_type: VOTER last_known_addr { host: "127.20.49.67" port: 36801 } } peers { permanent_uuid: "4e7232bd790c4224828b7b0b4afa5343" member_type: VOTER last_known_addr { host: "127.20.49.66" port: 39157 } }
I20251209 08:57:54.898326 21452 heartbeater.cc:499] Master 127.20.49.126:36881 was elected leader, sending a full tablet report...
I20251209 08:57:54.898577 21742 raft_consensus.cc:359] T 2300e8a9f70c43e480f6fbdf53e35e3e P 4e7232bd790c4224828b7b0b4afa5343 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f47b562197247999dbf34e5fe9f59f4" member_type: VOTER last_known_addr { host: "127.20.49.65" port: 34415 } } peers { permanent_uuid: "3fa3633f63ba443d832f1dfee97d6eb9" member_type: VOTER last_known_addr { host: "127.20.49.67" port: 36801 } } peers { permanent_uuid: "4e7232bd790c4224828b7b0b4afa5343" member_type: VOTER last_known_addr { host: "127.20.49.66" port: 39157 } }
I20251209 08:57:54.898741 21742 raft_consensus.cc:385] T 2300e8a9f70c43e480f6fbdf53e35e3e P 4e7232bd790c4224828b7b0b4afa5343 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251209 08:57:54.898794 21742 raft_consensus.cc:740] T 2300e8a9f70c43e480f6fbdf53e35e3e P 4e7232bd790c4224828b7b0b4afa5343 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e7232bd790c4224828b7b0b4afa5343, State: Initialized, Role: FOLLOWER
I20251209 08:57:54.899281 21741 ts_tablet_manager.cc:1434] T 2300e8a9f70c43e480f6fbdf53e35e3e P 3f47b562197247999dbf34e5fe9f59f4: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20251209 08:57:54.899271 21742 consensus_queue.cc:260] T 2300e8a9f70c43e480f6fbdf53e35e3e P 4e7232bd790c4224828b7b0b4afa5343 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f47b562197247999dbf34e5fe9f59f4" member_type: VOTER last_known_addr { host: "127.20.49.65" port: 34415 } } peers { permanent_uuid: "3fa3633f63ba443d832f1dfee97d6eb9" member_type: VOTER last_known_addr { host: "127.20.49.67" port: 36801 } } peers { permanent_uuid: "4e7232bd790c4224828b7b0b4afa5343" member_type: VOTER last_known_addr { host: "127.20.49.66" port: 39157 } }
I20251209 08:57:54.899410 21743 raft_consensus.cc:359] T 2300e8a9f70c43e480f6fbdf53e35e3e P 3fa3633f63ba443d832f1dfee97d6eb9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f47b562197247999dbf34e5fe9f59f4" member_type: VOTER last_known_addr { host: "127.20.49.65" port: 34415 } } peers { permanent_uuid: "3fa3633f63ba443d832f1dfee97d6eb9" member_type: VOTER last_known_addr { host: "127.20.49.67" port: 36801 } } peers { permanent_uuid: "4e7232bd790c4224828b7b0b4afa5343" member_type: VOTER last_known_addr { host: "127.20.49.66" port: 39157 } }
I20251209 08:57:54.899619 21743 raft_consensus.cc:385] T 2300e8a9f70c43e480f6fbdf53e35e3e P 3fa3633f63ba443d832f1dfee97d6eb9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251209 08:57:54.899682 21743 raft_consensus.cc:740] T 2300e8a9f70c43e480f6fbdf53e35e3e P 3fa3633f63ba443d832f1dfee97d6eb9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fa3633f63ba443d832f1dfee97d6eb9, State: Initialized, Role: FOLLOWER
I20251209 08:57:54.900051 21588 heartbeater.cc:499] Master 127.20.49.126:36881 was elected leader, sending a full tablet report...
I20251209 08:57:54.900095 21743 consensus_queue.cc:260] T 2300e8a9f70c43e480f6fbdf53e35e3e P 3fa3633f63ba443d832f1dfee97d6eb9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f47b562197247999dbf34e5fe9f59f4" member_type: VOTER last_known_addr { host: "127.20.49.65" port: 34415 } } peers { permanent_uuid: "3fa3633f63ba443d832f1dfee97d6eb9" member_type: VOTER last_known_addr { host: "127.20.49.67" port: 36801 } } peers { permanent_uuid: "4e7232bd790c4224828b7b0b4afa5343" member_type: VOTER last_known_addr { host: "127.20.49.66" port: 39157 } }
I20251209 08:57:54.900233 21742 ts_tablet_manager.cc:1434] T 2300e8a9f70c43e480f6fbdf53e35e3e P 4e7232bd790c4224828b7b0b4afa5343: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20251209 08:57:54.900897 21743 ts_tablet_manager.cc:1434] T 2300e8a9f70c43e480f6fbdf53e35e3e P 3fa3633f63ba443d832f1dfee97d6eb9: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20251209 08:57:54.900982 21724 heartbeater.cc:499] Master 127.20.49.126:36881 was elected leader, sending a full tablet report...
I20251209 08:57:55.074180 21747 raft_consensus.cc:493] T 2300e8a9f70c43e480f6fbdf53e35e3e P 3f47b562197247999dbf34e5fe9f59f4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251209 08:57:55.074380 21747 raft_consensus.cc:515] T 2300e8a9f70c43e480f6fbdf53e35e3e P 3f47b562197247999dbf34e5fe9f59f4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f47b562197247999dbf34e5fe9f59f4" member_type: VOTER last_known_addr { host: "127.20.49.65" port: 34415 } } peers { permanent_uuid: "3fa3633f63ba443d832f1dfee97d6eb9" member_type: VOTER last_known_addr { host: "127.20.49.67" port: 36801 } } peers { permanent_uuid: "4e7232bd790c4224828b7b0b4afa5343" member_type: VOTER last_known_addr { host: "127.20.49.66" port: 39157 } }
I20251209 08:57:55.075606 21747 leader_election.cc:290] T 2300e8a9f70c43e480f6fbdf53e35e3e P 3f47b562197247999dbf34e5fe9f59f4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3fa3633f63ba443d832f1dfee97d6eb9 (127.20.49.67:36801), 4e7232bd790c4224828b7b0b4afa5343 (127.20.49.66:39157)
I20251209 08:57:55.078833 21751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:57:55.075842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.49.67:36801 (local address 127.20.49.65:44559)
1209 08:57:55.076212 (+ 370us) negotiation.cc:107] Waiting for socket to connect
1209 08:57:55.076225 (+ 13us) client_negotiation.cc:174] Beginning negotiation
1209 08:57:55.076332 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1209 08:57:55.076597 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1209 08:57:55.076602 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
1209 08:57:55.076625 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1209 08:57:55.076927 (+ 302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:57:55.076934 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:57:55.077797 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1209 08:57:55.077801 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1209 08:57:55.078466 (+ 665us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:57:55.078475 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:57:55.078571 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:57:55.078588 (+ 17us) client_negotiation.cc:769] Sending connection context
1209 08:57:55.078627 (+ 39us) client_negotiation.cc:240] Negotiation successful
1209 08:57:55.078670 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":296,"spinlock_wait_cycles":41344,"thread_start_us":160,"threads_started":1}
W20251209 08:57:55.079263 21725 tablet.cc:2378] T 2300e8a9f70c43e480f6fbdf53e35e3e P 3fa3633f63ba443d832f1dfee97d6eb9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20251209 08:57:55.079413 21738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:57:55.075957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.49.65:44559 (local address 127.20.49.67:36801)
1209 08:57:55.076099 (+ 142us) server_negotiation.cc:207] Beginning negotiation
1209 08:57:55.076104 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1209 08:57:55.076324 (+ 220us) server_negotiation.cc:408] Connection header received
1209 08:57:55.076425 (+ 101us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1209 08:57:55.076431 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
1209 08:57:55.076495 (+ 64us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1209 08:57:55.076559 (+ 64us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1209 08:57:55.077076 (+ 517us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:57:55.077652 (+ 576us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1209 08:57:55.078598 (+ 946us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:57:55.079123 (+ 525us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:57:55.079163 (+ 40us) server_negotiation.cc:1036] Waiting for connection context
1209 08:57:55.079216 (+ 53us) server_negotiation.cc:300] Negotiation successful
1209 08:57:55.079272 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20251209 08:57:55.079540 21740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:57:55.076011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.49.65:59507 (local address 127.20.49.66:39157)
1209 08:57:55.076224 (+ 213us) server_negotiation.cc:207] Beginning negotiation
1209 08:57:55.076229 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1209 08:57:55.076386 (+ 157us) server_negotiation.cc:408] Connection header received
1209 08:57:55.076500 (+ 114us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1209 08:57:55.076503 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1209 08:57:55.076549 (+ 46us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1209 08:57:55.076631 (+ 82us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1209 08:57:55.077336 (+ 705us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:57:55.077868 (+ 532us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1209 08:57:55.078731 (+ 863us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:57:55.079245 (+ 514us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:57:55.079288 (+ 43us) server_negotiation.cc:1036] Waiting for connection context
1209 08:57:55.079340 (+ 52us) server_negotiation.cc:300] Negotiation successful
1209 08:57:55.079407 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115}
I20251209 08:57:55.079025 21750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:57:55.075872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.49.66:39157 (local address 127.20.49.65:59507)
1209 08:57:55.076272 (+ 400us) negotiation.cc:107] Waiting for socket to connect
1209 08:57:55.076289 (+ 17us) client_negotiation.cc:174] Beginning negotiation
1209 08:57:55.076405 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1209 08:57:55.076654 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1209 08:57:55.076657 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1209 08:57:55.076700 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1209 08:57:55.077217 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:57:55.077225 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:57:55.077989 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1209 08:57:55.077992 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1209 08:57:55.078606 (+ 614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:57:55.078615 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:57:55.078729 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:57:55.078747 (+ 18us) client_negotiation.cc:769] Sending connection context
1209 08:57:55.078797 (+ 50us) client_negotiation.cc:240] Negotiation successful
1209 08:57:55.078851 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":320,"spinlock_wait_cycles":896,"thread_start_us":124,"threads_started":1}
I20251209 08:57:55.079981 21678 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2300e8a9f70c43e480f6fbdf53e35e3e" candidate_uuid: "3f47b562197247999dbf34e5fe9f59f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fa3633f63ba443d832f1dfee97d6eb9" is_pre_election: true
I20251209 08:57:55.080272 21678 raft_consensus.cc:2468] T 2300e8a9f70c43e480f6fbdf53e35e3e P 3fa3633f63ba443d832f1dfee97d6eb9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f47b562197247999dbf34e5fe9f59f4 in term 0.
I20251209 08:57:55.080444 21524 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2300e8a9f70c43e480f6fbdf53e35e3e" candidate_uuid: "3f47b562197247999dbf34e5fe9f59f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e7232bd790c4224828b7b0b4afa5343" is_pre_election: true
I20251209 08:57:55.080708 21524 raft_consensus.cc:2468] T 2300e8a9f70c43e480f6fbdf53e35e3e P 4e7232bd790c4224828b7b0b4afa5343 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f47b562197247999dbf34e5fe9f59f4 in term 0.
I20251209 08:57:55.080785 21339 leader_election.cc:304] T 2300e8a9f70c43e480f6fbdf53e35e3e P 3f47b562197247999dbf34e5fe9f59f4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f47b562197247999dbf34e5fe9f59f4, 3fa3633f63ba443d832f1dfee97d6eb9; no voters:
I20251209 08:57:55.081089 21747 raft_consensus.cc:2804] T 2300e8a9f70c43e480f6fbdf53e35e3e P 3f47b562197247999dbf34e5fe9f59f4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251209 08:57:55.081179 21747 raft_consensus.cc:493] T 2300e8a9f70c43e480f6fbdf53e35e3e P 3f47b562197247999dbf34e5fe9f59f4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251209 08:57:55.081231 21747 raft_consensus.cc:3060] T 2300e8a9f70c43e480f6fbdf53e35e3e P 3f47b562197247999dbf34e5fe9f59f4 [term 0 FOLLOWER]: Advancing to term 1
I20251209 08:57:55.082193 21747 raft_consensus.cc:515] T 2300e8a9f70c43e480f6fbdf53e35e3e P 3f47b562197247999dbf34e5fe9f59f4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f47b562197247999dbf34e5fe9f59f4" member_type: VOTER last_known_addr { host: "127.20.49.65" port: 34415 } } peers { permanent_uuid: "3fa3633f63ba443d832f1dfee97d6eb9" member_type: VOTER last_known_addr { host: "127.20.49.67" port: 36801 } } peers { permanent_uuid: "4e7232bd790c4224828b7b0b4afa5343" member_type: VOTER last_known_addr { host: "127.20.49.66" port: 39157 } }
I20251209 08:57:55.082525 21747 leader_election.cc:290] T 2300e8a9f70c43e480f6fbdf53e35e3e P 3f47b562197247999dbf34e5fe9f59f4 [CANDIDATE]: Term 1 election: Requested vote from peers 3fa3633f63ba443d832f1dfee97d6eb9 (127.20.49.67:36801), 4e7232bd790c4224828b7b0b4afa5343 (127.20.49.66:39157)
I20251209 08:57:55.082896 21524 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2300e8a9f70c43e480f6fbdf53e35e3e" candidate_uuid: "3f47b562197247999dbf34e5fe9f59f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e7232bd790c4224828b7b0b4afa5343"
I20251209 08:57:55.082942 21678 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2300e8a9f70c43e480f6fbdf53e35e3e" candidate_uuid: "3f47b562197247999dbf34e5fe9f59f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fa3633f63ba443d832f1dfee97d6eb9"
I20251209 08:57:55.083030 21524 raft_consensus.cc:3060] T 2300e8a9f70c43e480f6fbdf53e35e3e P 4e7232bd790c4224828b7b0b4afa5343 [term 0 FOLLOWER]: Advancing to term 1
I20251209 08:57:55.083047 21678 raft_consensus.cc:3060] T 2300e8a9f70c43e480f6fbdf53e35e3e P 3fa3633f63ba443d832f1dfee97d6eb9 [term 0 FOLLOWER]: Advancing to term 1
I20251209 08:57:55.084254 21678 raft_consensus.cc:2468] T 2300e8a9f70c43e480f6fbdf53e35e3e P 3fa3633f63ba443d832f1dfee97d6eb9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f47b562197247999dbf34e5fe9f59f4 in term 1.
I20251209 08:57:55.084254 21524 raft_consensus.cc:2468] T 2300e8a9f70c43e480f6fbdf53e35e3e P 4e7232bd790c4224828b7b0b4afa5343 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f47b562197247999dbf34e5fe9f59f4 in term 1.
I20251209 08:57:55.084569 21339 leader_election.cc:304] T 2300e8a9f70c43e480f6fbdf53e35e3e P 3f47b562197247999dbf34e5fe9f59f4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f47b562197247999dbf34e5fe9f59f4, 3fa3633f63ba443d832f1dfee97d6eb9; no voters:
I20251209 08:57:55.084730 21747 raft_consensus.cc:2804] T 2300e8a9f70c43e480f6fbdf53e35e3e P 3f47b562197247999dbf34e5fe9f59f4 [term 1 FOLLOWER]: Leader election won for term 1
I20251209 08:57:55.084952 21747 raft_consensus.cc:697] T 2300e8a9f70c43e480f6fbdf53e35e3e P 3f47b562197247999dbf34e5fe9f59f4 [term 1 LEADER]: Becoming Leader. State: Replica: 3f47b562197247999dbf34e5fe9f59f4, State: Running, Role: LEADER
I20251209 08:57:55.085320 21747 consensus_queue.cc:237] T 2300e8a9f70c43e480f6fbdf53e35e3e P 3f47b562197247999dbf34e5fe9f59f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f47b562197247999dbf34e5fe9f59f4" member_type: VOTER last_known_addr { host: "127.20.49.65" port: 34415 } } peers { permanent_uuid: "3fa3633f63ba443d832f1dfee97d6eb9" member_type: VOTER last_known_addr { host: "127.20.49.67" port: 36801 } } peers { permanent_uuid: "4e7232bd790c4224828b7b0b4afa5343" member_type: VOTER last_known_addr { host: "127.20.49.66" port: 39157 } }
I20251209 08:57:55.088610 21259 catalog_manager.cc:5654] T 2300e8a9f70c43e480f6fbdf53e35e3e P 3f47b562197247999dbf34e5fe9f59f4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3f47b562197247999dbf34e5fe9f59f4 (127.20.49.65). New cstate: current_term: 1 leader_uuid: "3f47b562197247999dbf34e5fe9f59f4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f47b562197247999dbf34e5fe9f59f4" member_type: VOTER last_known_addr { host: "127.20.49.65" port: 34415 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3fa3633f63ba443d832f1dfee97d6eb9" member_type: VOTER last_known_addr { host: "127.20.49.67" port: 36801 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4e7232bd790c4224828b7b0b4afa5343" member_type: VOTER last_known_addr { host: "127.20.49.66" port: 39157 } health_report { overall_health: UNKNOWN } } }
W20251209 08:57:55.100507 21589 tablet.cc:2378] T 2300e8a9f70c43e480f6fbdf53e35e3e P 4e7232bd790c4224828b7b0b4afa5343: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20251209 08:57:55.109943 21453 tablet.cc:2378] T 2300e8a9f70c43e480f6fbdf53e35e3e P 3f47b562197247999dbf34e5fe9f59f4: Can't schedule compaction. Clean time has not been advanced past its initial value.
Dec 09 08:57:55 dist-test-slave-8w6c krb5kdc[21211](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765270673, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.49.65@KRBTEST.COM
I20251209 08:57:55.117666 21736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:57:55.109138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54794 (local address 127.20.49.65:34415)
1209 08:57:55.109298 (+ 160us) server_negotiation.cc:207] Beginning negotiation
1209 08:57:55.109303 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1209 08:57:55.109416 (+ 113us) server_negotiation.cc:408] Connection header received
1209 08:57:55.109545 (+ 129us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1209 08:57:55.109550 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1209 08:57:55.109595 (+ 45us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1209 08:57:55.109667 (+ 72us) server_negotiation.cc:227] Negotiated authn=SASL
1209 08:57:55.110565 (+ 898us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:57:55.111071 (+ 506us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1209 08:57:55.111700 (+ 629us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:57:55.111934 (+ 234us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:57:55.114291 (+ 2357us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1209 08:57:55.114313 (+ 22us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1209 08:57:55.114324 (+ 11us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1209 08:57:55.114349 (+ 25us) server_negotiation.cc:923] calling sasl_server_start()
1209 08:57:55.116060 (+ 1711us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1209 08:57:55.116525 (+ 465us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1209 08:57:55.116530 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1209 08:57:55.116531 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1209 08:57:55.116586 (+ 55us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1209 08:57:55.116803 (+ 217us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1209 08:57:55.116807 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1209 08:57:55.116809 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1209 08:57:55.117008 (+ 199us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1209 08:57:55.117203 (+ 195us) server_negotiation.cc:1036] Waiting for connection context
1209 08:57:55.117381 (+ 178us) server_negotiation.cc:300] Negotiation successful
1209 08:57:55.117504 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
Dec 09 08:57:55 dist-test-slave-8w6c krb5kdc[21211](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765270675, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Dec 09 08:57:55 dist-test-slave-8w6c krb5kdc[21211](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765270675, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.20.49.65@KRBTEST.COM
I20251209 08:57:55.142859 21736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:57:55.135428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54808 (local address 127.20.49.65:34415)
1209 08:57:55.135585 (+ 157us) server_negotiation.cc:207] Beginning negotiation
1209 08:57:55.135591 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1209 08:57:55.135675 (+ 84us) server_negotiation.cc:408] Connection header received
1209 08:57:55.135821 (+ 146us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1209 08:57:55.135825 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1209 08:57:55.135875 (+ 50us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1209 08:57:55.135941 (+ 66us) server_negotiation.cc:227] Negotiated authn=SASL
1209 08:57:55.136750 (+ 809us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:57:55.137243 (+ 493us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1209 08:57:55.137927 (+ 684us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:57:55.138154 (+ 227us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:57:55.140194 (+ 2040us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1209 08:57:55.140217 (+ 23us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1209 08:57:55.140219 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1209 08:57:55.140246 (+ 27us) server_negotiation.cc:923] calling sasl_server_start()
1209 08:57:55.141513 (+ 1267us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1209 08:57:55.141975 (+ 462us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1209 08:57:55.141978 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1209 08:57:55.141980 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1209 08:57:55.142024 (+ 44us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1209 08:57:55.142241 (+ 217us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1209 08:57:55.142243 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1209 08:57:55.142244 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1209 08:57:55.142370 (+ 126us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1209 08:57:55.142463 (+ 93us) server_negotiation.cc:1036] Waiting for connection context
1209 08:57:55.142637 (+ 174us) server_negotiation.cc:300] Negotiation successful
1209 08:57:55.142734 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
Dec 09 08:57:55 dist-test-slave-8w6c krb5kdc[21211](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765270675, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Dec 09 08:57:55 dist-test-slave-8w6c krb5kdc[21211](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765270675, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.49.65@KRBTEST.COM
I20251209 08:57:55.167260 21736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:57:55.159496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54822 (local address 127.20.49.65:34415)
1209 08:57:55.159646 (+ 150us) server_negotiation.cc:207] Beginning negotiation
1209 08:57:55.159652 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1209 08:57:55.159729 (+ 77us) server_negotiation.cc:408] Connection header received
1209 08:57:55.159876 (+ 147us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1209 08:57:55.159880 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1209 08:57:55.159927 (+ 47us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1209 08:57:55.160020 (+ 93us) server_negotiation.cc:227] Negotiated authn=SASL
1209 08:57:55.160825 (+ 805us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:57:55.161297 (+ 472us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1209 08:57:55.161963 (+ 666us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:57:55.162173 (+ 210us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:57:55.164351 (+ 2178us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1209 08:57:55.164369 (+ 18us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1209 08:57:55.164372 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1209 08:57:55.164398 (+ 26us) server_negotiation.cc:923] calling sasl_server_start()
1209 08:57:55.165802 (+ 1404us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1209 08:57:55.166233 (+ 431us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1209 08:57:55.166238 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1209 08:57:55.166240 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1209 08:57:55.166290 (+ 50us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1209 08:57:55.166503 (+ 213us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1209 08:57:55.166506 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1209 08:57:55.166508 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1209 08:57:55.166646 (+ 138us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1209 08:57:55.166766 (+ 120us) server_negotiation.cc:1036] Waiting for connection context
1209 08:57:55.166975 (+ 209us) server_negotiation.cc:300] Negotiation successful
1209 08:57:55.167095 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
W20251209 08:57:55.168542 21366 tablet_service.cc:3012] Rejecting scan request for tablet 2300e8a9f70c43e480f6fbdf53e35e3e: Uninitialized: clean time has not yet been initialized
I20251209 08:57:55.170236 20677 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task8XuDZV/build/debug/bin/kudu with pid 21323
I20251209 08:57:55.177433 20677 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task8XuDZV/build/debug/bin/kudu with pid 21459
I20251209 08:57:55.183353 20677 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task8XuDZV/build/debug/bin/kudu with pid 21595
I20251209 08:57:55.189570 20677 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task8XuDZV/build/debug/bin/kudu with pid 21227
2025-12-09T08:57:55Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (3495 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765270668420570-20677-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Dec 09 08:57:55 dist-test-slave-8w6c krb5kdc[21766](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Dec 09 08:57:55 dist-test-slave-8w6c krb5kdc[21766](info): set up 2 sockets
Dec 09 08:57:55 dist-test-slave-8w6c krb5kdc[21766](info): commencing operation
krb5kdc: starting...
W20251209 08:57:57.198168 20677 mini_kdc.cc:121] Time spent starting KDC: real 1.984s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765270668420570-20677-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765270668420570-20677-0/krb5kdc/test-user.keytab.
Dec 09 08:57:57 dist-test-slave-8w6c krb5kdc[21766](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765270677, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-12-09T08:57:57Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-12-09T08:57:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.49.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.49.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.49.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765270668420570-20677-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.20.49.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765270668420570-20677-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.20.49.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.20.49.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.20.49.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765270668420570-20677-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.20.49.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765270668420570-20677-0/krb5kdc/kudu.keytab.
I20251209 08:57:57.338091 20677 external_mini_cluster.cc:1366] Running /tmp/dist-test-task8XuDZV/build/debug/bin/kudu
/tmp/dist-test-task8XuDZV/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765270668420570-20677-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765270668420570-20677-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765270668420570-20677-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765270668420570-20677-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.49.126:35129
--webserver_interface=127.20.49.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765270668420570-20677-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.49.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.20.49.84:37945
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.49.126:35129
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765270668420570-20677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765270668420570-20677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765270668420570-20677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251209 08:57:57.445247 21782 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251209 08:57:57.445489 21782 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251209 08:57:57.445580 21782 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251209 08:57:57.448841 21782 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251209 08:57:57.448906 21782 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20251209 08:57:57.448937 21782 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251209 08:57:57.448980 21782 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251209 08:57:57.449020 21782 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251209 08:57:57.449065 21782 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251209 08:57:57.453300 21782 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.49.84:37945
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765270668420570-20677-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765270668420570-20677-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.49.126:35129
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765270668420570-20677-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.49.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.49.126:35129
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765270668420570-20677-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.49.126
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21782
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765270668420570-20677-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 468897755969d686fb08386f42f7835685e7953a
build type DEBUG
built by None at 09 Dec 2025 08:43:15 UTC on 24a791456cd2
build id 9426
I20251209 08:57:57.454360 21782 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251209 08:57:57.455205 21782 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251209 08:57:57.460681 21787 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251209 08:57:57.460672 21788 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251209 08:57:57.460672 21790 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251209 08:57:57.461176 21782 server_base.cc:1047] running on GCE node
I20251209 08:57:57.461630 21782 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251209 08:57:57.462550 21782 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251209 08:57:57.463701 21782 hybrid_clock.cc:648] HybridClock initialized: now 1765270677463692 us; error 29 us; skew 500 ppm
Dec 09 08:57:57 dist-test-slave-8w6c krb5kdc[21766](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765270677, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.49.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251209 08:57:57.466248 21782 init.cc:377] Logged in from keytab as kudu/127.20.49.126@KRBTEST.COM (short username kudu)
I20251209 08:57:57.467291 21782 webserver.cc:492] Webserver started at http://127.20.49.126:41295/ using document root <none> and password file <none>
I20251209 08:57:57.467818 21782 fs_manager.cc:362] Metadata directory not provided
I20251209 08:57:57.467873 21782 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251209 08:57:57.468052 21782 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251209 08:57:57.469784 21782 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765270668420570-20677-0/minicluster-data/master-0/data/instance:
uuid: "fc08086ef48c4a79a51eac09446fa095"
format_stamp: "Formatted at 2025-12-09 08:57:57 on dist-test-slave-8w6c"
server_key: "1be56262cd95a702a9b9b384e604221d"
server_key_iv: "d65542b8dc485a8a029e9b9033f77b22"
server_key_version: "encryptionkey@0"
I20251209 08:57:57.470271 21782 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765270668420570-20677-0/minicluster-data/master-0/wal/instance:
uuid: "fc08086ef48c4a79a51eac09446fa095"
format_stamp: "Formatted at 2025-12-09 08:57:57 on dist-test-slave-8w6c"
server_key: "1be56262cd95a702a9b9b384e604221d"
server_key_iv: "d65542b8dc485a8a029e9b9033f77b22"
server_key_version: "encryptionkey@0"
I20251209 08:57:57.473608 21782 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20251209 08:57:57.475919 21797 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251209 08:57:57.476992 21782 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20251209 08:57:57.477106 21782 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765270668420570-20677-0/minicluster-data/master-0/data,/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765270668420570-20677-0/minicluster-data/master-0/wal
uuid: "fc08086ef48c4a79a51eac09446fa095"
format_stamp: "Formatted at 2025-12-09 08:57:57 on dist-test-slave-8w6c"
server_key: "1be56262cd95a702a9b9b384e604221d"
server_key_iv: "d65542b8dc485a8a029e9b9033f77b22"
server_key_version: "encryptionkey@0"
I20251209 08:57:57.477195 21782 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765270668420570-20677-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765270668420570-20677-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765270668420570-20677-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251209 08:57:57.490144 21782 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251209 08:57:57.492786 21782 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251209 08:57:57.492970 21782 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251209 08:57:57.501039 21782 rpc_server.cc:307] RPC server started. Bound to: 127.20.49.126:35129
I20251209 08:57:57.501349 21859 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.49.126:35129 every 8 connection(s)
I20251209 08:57:57.502079 21782 server_base.cc:1179] Dumped server information to /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765270668420570-20677-0/minicluster-data/master-0/data/info.pb
I20251209 08:57:57.504168 20677 external_mini_cluster.cc:1428] Started /tmp/dist-test-task8XuDZV/build/debug/bin/kudu as pid 21782
I20251209 08:57:57.504284 20677 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765270668420570-20677-0/minicluster-data/master-0/wal/instance
I20251209 08:57:57.504519 20677 external_mini_cluster.cc:1468] Setting key 31cf4848e7bf8d28839399aecc2e0837
I20251209 08:57:57.505039 21860 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251209 08:57:57.510294 21860 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc08086ef48c4a79a51eac09446fa095: Bootstrap starting.
Dec 09 08:57:57 dist-test-slave-8w6c krb5kdc[21766](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765270677, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.49.126@KRBTEST.COM
I20251209 08:57:57.512580 21860 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fc08086ef48c4a79a51eac09446fa095: Neither blocks nor log segments found. Creating new log.
I20251209 08:57:57.513394 21860 log.cc:826] T 00000000000000000000000000000000 P fc08086ef48c4a79a51eac09446fa095: Log is configured to *not* fsync() on all Append() calls
I20251209 08:57:57.515232 21860 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc08086ef48c4a79a51eac09446fa095: No bootstrap required, opened a new log
I20251209 08:57:57.517339 21863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:57:57.505775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43270 (local address 127.20.49.126:35129)
1209 08:57:57.506188 (+ 413us) server_negotiation.cc:207] Beginning negotiation
1209 08:57:57.506198 (+ 10us) server_negotiation.cc:400] Waiting for connection header
1209 08:57:57.506231 (+ 33us) server_negotiation.cc:408] Connection header received
1209 08:57:57.506867 (+ 636us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1209 08:57:57.506884 (+ 17us) server_negotiation.cc:462] Received NEGOTIATE request from client
1209 08:57:57.507168 (+ 284us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1209 08:57:57.507464 (+ 296us) server_negotiation.cc:227] Negotiated authn=SASL
1209 08:57:57.508368 (+ 904us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:57:57.509132 (+ 764us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1209 08:57:57.509780 (+ 648us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:57:57.510075 (+ 295us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:57:57.512506 (+ 2431us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1209 08:57:57.512523 (+ 17us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1209 08:57:57.512533 (+ 10us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1209 08:57:57.512564 (+ 31us) server_negotiation.cc:923] calling sasl_server_start()
1209 08:57:57.514683 (+ 2119us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1209 08:57:57.515079 (+ 396us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1209 08:57:57.515083 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1209 08:57:57.515088 (+ 5us) server_negotiation.cc:968] calling sasl_server_step()
1209 08:57:57.515165 (+ 77us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1209 08:57:57.515405 (+ 240us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1209 08:57:57.515409 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1209 08:57:57.515411 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1209 08:57:57.515810 (+ 399us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1209 08:57:57.516005 (+ 195us) server_negotiation.cc:1036] Waiting for connection context
1209 08:57:57.516308 (+ 303us) server_negotiation.cc:300] Negotiation successful
1209 08:57:57.516569 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":285,"thread_start_us":135,"threads_started":1}
I20251209 08:57:57.518920 21860 raft_consensus.cc:359] T 00000000000000000000000000000000 P fc08086ef48c4a79a51eac09446fa095 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc08086ef48c4a79a51eac09446fa095" member_type: VOTER last_known_addr { host: "127.20.49.126" port: 35129 } }
I20251209 08:57:57.519104 21860 raft_consensus.cc:385] T 00000000000000000000000000000000 P fc08086ef48c4a79a51eac09446fa095 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251209 08:57:57.519143 21860 raft_consensus.cc:740] T 00000000000000000000000000000000 P fc08086ef48c4a79a51eac09446fa095 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc08086ef48c4a79a51eac09446fa095, State: Initialized, Role: FOLLOWER
I20251209 08:57:57.519518 21860 consensus_queue.cc:260] T 00000000000000000000000000000000 P fc08086ef48c4a79a51eac09446fa095 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc08086ef48c4a79a51eac09446fa095" member_type: VOTER last_known_addr { host: "127.20.49.126" port: 35129 } }
I20251209 08:57:57.519631 21860 raft_consensus.cc:399] T 00000000000000000000000000000000 P fc08086ef48c4a79a51eac09446fa095 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251209 08:57:57.519693 21860 raft_consensus.cc:493] T 00000000000000000000000000000000 P fc08086ef48c4a79a51eac09446fa095 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251209 08:57:57.519783 21860 raft_consensus.cc:3060] T 00000000000000000000000000000000 P fc08086ef48c4a79a51eac09446fa095 [term 0 FOLLOWER]: Advancing to term 1
I20251209 08:57:57.520700 21860 raft_consensus.cc:515] T 00000000000000000000000000000000 P fc08086ef48c4a79a51eac09446fa095 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc08086ef48c4a79a51eac09446fa095" member_type: VOTER last_known_addr { host: "127.20.49.126" port: 35129 } }
I20251209 08:57:57.521070 21860 leader_election.cc:304] T 00000000000000000000000000000000 P fc08086ef48c4a79a51eac09446fa095 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fc08086ef48c4a79a51eac09446fa095; no voters:
I20251209 08:57:57.521373 21860 leader_election.cc:290] T 00000000000000000000000000000000 P fc08086ef48c4a79a51eac09446fa095 [CANDIDATE]: Term 1 election: Requested vote from peers
I20251209 08:57:57.521490 21865 raft_consensus.cc:2804] T 00000000000000000000000000000000 P fc08086ef48c4a79a51eac09446fa095 [term 1 FOLLOWER]: Leader election won for term 1
I20251209 08:57:57.521744 21865 raft_consensus.cc:697] T 00000000000000000000000000000000 P fc08086ef48c4a79a51eac09446fa095 [term 1 LEADER]: Becoming Leader. State: Replica: fc08086ef48c4a79a51eac09446fa095, State: Running, Role: LEADER
I20251209 08:57:57.522089 21865 consensus_queue.cc:237] T 00000000000000000000000000000000 P fc08086ef48c4a79a51eac09446fa095 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc08086ef48c4a79a51eac09446fa095" member_type: VOTER last_known_addr { host: "127.20.49.126" port: 35129 } }
I20251209 08:57:57.522475 21860 sys_catalog.cc:565] T 00000000000000000000000000000000 P fc08086ef48c4a79a51eac09446fa095 [sys.catalog]: configured and running, proceeding with master startup.
I20251209 08:57:57.523922 21867 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc08086ef48c4a79a51eac09446fa095 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fc08086ef48c4a79a51eac09446fa095. Latest consensus state: current_term: 1 leader_uuid: "fc08086ef48c4a79a51eac09446fa095" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc08086ef48c4a79a51eac09446fa095" member_type: VOTER last_known_addr { host: "127.20.49.126" port: 35129 } } }
I20251209 08:57:57.523927 21866 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc08086ef48c4a79a51eac09446fa095 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fc08086ef48c4a79a51eac09446fa095" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc08086ef48c4a79a51eac09446fa095" member_type: VOTER last_known_addr { host: "127.20.49.126" port: 35129 } } }
I20251209 08:57:57.524055 21866 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc08086ef48c4a79a51eac09446fa095 [sys.catalog]: This master's current role is: LEADER
I20251209 08:57:57.524055 21867 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc08086ef48c4a79a51eac09446fa095 [sys.catalog]: This master's current role is: LEADER
I20251209 08:57:57.524747 21874 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251209 08:57:57.527410 21874 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251209 08:57:57.532611 21874 catalog_manager.cc:1357] Generated new cluster ID: 704e9f0391e440d695094437653e9464
I20251209 08:57:57.532684 21874 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251209 08:57:57.562546 21874 catalog_manager.cc:1380] Generated new certificate authority record
I20251209 08:57:57.563531 21874 catalog_manager.cc:1514] Loading token signing keys...
I20251209 08:57:57.571071 21874 catalog_manager.cc:6027] T 00000000000000000000000000000000 P fc08086ef48c4a79a51eac09446fa095: Generated new TSK 0
I20251209 08:57:57.571782 21874 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.20.49.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.49.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.49.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765270668420570-20677-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.20.49.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765270668420570-20677-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.20.49.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.20.49.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.20.49.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765270668420570-20677-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.20.49.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765270668420570-20677-0/krb5kdc/kudu.keytab.
I20251209 08:57:57.625121 20677 external_mini_cluster.cc:1366] Running /tmp/dist-test-task8XuDZV/build/debug/bin/kudu
/tmp/dist-test-task8XuDZV/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765270668420570-20677-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765270668420570-20677-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765270668420570-20677-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765270668420570-20677-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.49.65:0
--local_ip_for_outbound_sockets=127.20.49.65
--webserver_interface=127.20.49.65
--webserver_port=0
--tserver_master_addrs=127.20.49.126:35129
--keytab_file=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765270668420570-20677-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.49.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.20.49.84:37945
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765270668420570-20677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765270668420570-20677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765270668420570-20677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251209 08:57:57.734391 21888 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251209 08:57:57.734627 21888 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251209 08:57:57.734694 21888 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251209 08:57:57.737986 21888 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251209 08:57:57.738061 21888 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251209 08:57:57.738114 21888 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20251209 08:57:57.738194 21888 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.49.65
I20251209 08:57:57.742384 21888 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.49.84:37945
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765270668420570-20677-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765270668420570-20677-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765270668420570-20677-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.49.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.49.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765270668420570-20677-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.49.65
--webserver_port=0
--webserver_require_spnego=true
--enable_txn_system_client_init=true
--tserver_master_addrs=127.20.49.126:35129
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21888
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.49.65
--log_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765270668420570-20677-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 468897755969d686fb08386f42f7835685e7953a
build type DEBUG
built by None at 09 Dec 2025 08:43:15 UTC on 24a791456cd2
build id 9426
I20251209 08:57:57.743495 21888 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251209 08:57:57.744372 21888 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251209 08:57:57.750865 21893 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251209 08:57:57.750875 21894 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251209 08:57:57.750874 21896 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251209 08:57:57.750931 21888 server_base.cc:1047] running on GCE node
I20251209 08:57:57.751443 21888 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251209 08:57:57.751963 21888 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251209 08:57:57.753139 21888 hybrid_clock.cc:648] HybridClock initialized: now 1765270677753112 us; error 39 us; skew 500 ppm
Dec 09 08:57:57 dist-test-slave-8w6c krb5kdc[21766](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765270677, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.49.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251209 08:57:57.755738 21888 init.cc:377] Logged in from keytab as kudu/127.20.49.65@KRBTEST.COM (short username kudu)
I20251209 08:57:57.756778 21888 webserver.cc:492] Webserver started at http://127.20.49.65:43811/ using document root <none> and password file <none>
I20251209 08:57:57.757311 21888 fs_manager.cc:362] Metadata directory not provided
I20251209 08:57:57.757361 21888 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251209 08:57:57.757519 21888 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251209 08:57:57.759225 21888 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765270668420570-20677-0/minicluster-data/ts-0/data/instance:
uuid: "4764a44e2a814f6db9d0185ea0bc2425"
format_stamp: "Formatted at 2025-12-09 08:57:57 on dist-test-slave-8w6c"
server_key: "5f5595df2f660f418a5e7d4207001efa"
server_key_iv: "4901fa4032c1b1a2cc3cd7987b109b5b"
server_key_version: "encryptionkey@0"
I20251209 08:57:57.759660 21888 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765270668420570-20677-0/minicluster-data/ts-0/wal/instance:
uuid: "4764a44e2a814f6db9d0185ea0bc2425"
format_stamp: "Formatted at 2025-12-09 08:57:57 on dist-test-slave-8w6c"
server_key: "5f5595df2f660f418a5e7d4207001efa"
server_key_iv: "4901fa4032c1b1a2cc3cd7987b109b5b"
server_key_version: "encryptionkey@0"
I20251209 08:57:57.763104 21888 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20251209 08:57:57.765353 21903 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251209 08:57:57.766548 21888 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20251209 08:57:57.766650 21888 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765270668420570-20677-0/minicluster-data/ts-0/data,/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765270668420570-20677-0/minicluster-data/ts-0/wal
uuid: "4764a44e2a814f6db9d0185ea0bc2425"
format_stamp: "Formatted at 2025-12-09 08:57:57 on dist-test-slave-8w6c"
server_key: "5f5595df2f660f418a5e7d4207001efa"
server_key_iv: "4901fa4032c1b1a2cc3cd7987b109b5b"
server_key_version: "encryptionkey@0"
I20251209 08:57:57.766759 21888 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765270668420570-20677-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765270668420570-20677-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765270668420570-20677-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251209 08:57:57.785080 21888 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251209 08:57:57.788383 21888 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251209 08:57:57.788576 21888 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251209 08:57:57.790129 21888 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251209 08:57:57.790182 21888 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251209 08:57:57.790235 21888 ts_tablet_manager.cc:616] Registered 0 tablets
I20251209 08:57:57.790270 21888 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Dec 09 08:57:57 dist-test-slave-8w6c krb5kdc[21766](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765270677, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.49.65@KRBTEST.COM for kudu/127.20.49.126@KRBTEST.COM
I20251209 08:57:57.801559 21888 rpc_server.cc:307] RPC server started. Bound to: 127.20.49.65:43253
I20251209 08:57:57.801625 22018 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.49.65:43253 every 8 connection(s)
I20251209 08:57:57.802570 21863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:57:57.790727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.49.65:54559 (local address 127.20.49.126:35129)
1209 08:57:57.790854 (+ 127us) server_negotiation.cc:207] Beginning negotiation
1209 08:57:57.790858 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1209 08:57:57.791709 (+ 851us) server_negotiation.cc:408] Connection header received
1209 08:57:57.792565 (+ 856us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1209 08:57:57.792569 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1209 08:57:57.792636 (+ 67us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1209 08:57:57.792719 (+ 83us) server_negotiation.cc:227] Negotiated authn=SASL
1209 08:57:57.794372 (+ 1653us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:57:57.794991 (+ 619us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1209 08:57:57.795639 (+ 648us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:57:57.795822 (+ 183us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:57:57.798662 (+ 2840us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1209 08:57:57.798691 (+ 29us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1209 08:57:57.798695 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1209 08:57:57.798730 (+ 35us) server_negotiation.cc:923] calling sasl_server_start()
1209 08:57:57.800494 (+ 1764us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1209 08:57:57.800971 (+ 477us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1209 08:57:57.800975 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1209 08:57:57.800976 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1209 08:57:57.801042 (+ 66us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1209 08:57:57.801380 (+ 338us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1209 08:57:57.801383 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1209 08:57:57.801385 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1209 08:57:57.801584 (+ 199us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1209 08:57:57.801716 (+ 132us) server_negotiation.cc:1036] Waiting for connection context
1209 08:57:57.802258 (+ 542us) server_negotiation.cc:300] Negotiation successful
1209 08:57:57.802409 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20251209 08:57:57.802660 21888 server_base.cc:1179] Dumped server information to /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765270668420570-20677-0/minicluster-data/ts-0/data/info.pb
I20251209 08:57:57.803050 21912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:57:57.791085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.49.126:35129 (local address 127.20.49.65:54559)
1209 08:57:57.791564 (+ 479us) negotiation.cc:107] Waiting for socket to connect
1209 08:57:57.791599 (+ 35us) client_negotiation.cc:174] Beginning negotiation
1209 08:57:57.792363 (+ 764us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1209 08:57:57.792900 (+ 537us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1209 08:57:57.792913 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
1209 08:57:57.793465 (+ 552us) client_negotiation.cc:189] Negotiated authn=SASL
1209 08:57:57.794214 (+ 749us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:57:57.794229 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:57:57.795111 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1209 08:57:57.795117 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1209 08:57:57.795524 (+ 407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:57:57.795531 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:57:57.795735 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:57:57.796434 (+ 699us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1209 08:57:57.796471 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
1209 08:57:57.798493 (+ 2022us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1209 08:57:57.800616 (+ 2123us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1209 08:57:57.800622 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1209 08:57:57.800633 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
1209 08:57:57.800864 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1209 08:57:57.801155 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1209 08:57:57.801158 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1209 08:57:57.801160 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1209 08:57:57.801261 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1209 08:57:57.801773 (+ 512us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1209 08:57:57.801780 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1209 08:57:57.802000 (+ 220us) client_negotiation.cc:769] Sending connection context
1209 08:57:57.802171 (+ 171us) client_negotiation.cc:240] Negotiation successful
1209 08:57:57.802386 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":301,"thread_start_us":136,"threads_started":1}
I20251209 08:57:57.804495 22019 heartbeater.cc:344] Connected to a master server at 127.20.49.126:35129
I20251209 08:57:57.804649 22019 heartbeater.cc:461] Registering TS with master...
I20251209 08:57:57.805353 22019 heartbeater.cc:507] Master 127.20.49.126:35129 requested a full tablet report, sending...
I20251209 08:57:57.806912 21813 ts_manager.cc:194] Registered new tserver with Master: 4764a44e2a814f6db9d0185ea0bc2425 (127.20.49.65:43253)
I20251209 08:57:57.808084 21813 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.49.65@KRBTEST.COM'} at 127.20.49.65:54559
I20251209 08:57:57.811391 20677 external_mini_cluster.cc:1428] Started /tmp/dist-test-task8XuDZV/build/debug/bin/kudu as pid 21888
I20251209 08:57:57.811483 20677 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765270668420570-20677-0/minicluster-data/ts-0/wal/instance
I20251209 08:57:57.811720 20677 external_mini_cluster.cc:1468] Setting key 757fbff5054c256ba07457682d2a34d0
I20251209 08:57:57.813197 22025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:57:57.806754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.49.126:35129 (local address 127.20.49.65:44441)
1209 08:57:57.807124 (+ 370us) negotiation.cc:107] Waiting for socket to connect
1209 08:57:57.807137 (+ 13us) client_negotiation.cc:174] Beginning negotiation
1209 08:57:57.807238 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1209 08:57:57.807507 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1209 08:57:57.807510 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1209 08:57:57.807679 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
1209 08:57:57.807881 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:57:57.807887 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:57:57.808626 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1209 08:57:57.808629 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1209 08:57:57.809084 (+ 455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:57:57.809094 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:57:57.809212 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:57:57.809788 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1209 08:57:57.809803 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
1209 08:57:57.810163 (+ 360us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1209 08:57:57.811927 (+ 1764us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1209 08:57:57.811931 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1209 08:57:57.811933 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1209 08:57:57.812140 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1209 08:57:57.812368 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1209 08:57:57.812371 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1209 08:57:57.812373 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1209 08:57:57.812427 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1209 08:57:57.812752 (+ 325us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1209 08:57:57.812754 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1209 08:57:57.812823 (+ 69us) client_negotiation.cc:769] Sending connection context
1209 08:57:57.812897 (+ 74us) client_negotiation.cc:240] Negotiation successful
1209 08:57:57.813005 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":101,"threads_started":1}
I20251209 08:57:57.813210 21863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:57:57.806842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.49.65:44441 (local address 127.20.49.126:35129)
1209 08:57:57.806955 (+ 113us) server_negotiation.cc:207] Beginning negotiation
1209 08:57:57.806959 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1209 08:57:57.807207 (+ 248us) server_negotiation.cc:408] Connection header received
1209 08:57:57.807353 (+ 146us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1209 08:57:57.807357 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1209 08:57:57.807403 (+ 46us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1209 08:57:57.807470 (+ 67us) server_negotiation.cc:227] Negotiated authn=SASL
1209 08:57:57.807993 (+ 523us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:57:57.808517 (+ 524us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1209 08:57:57.809196 (+ 679us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:57:57.809347 (+ 151us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:57:57.810293 (+ 946us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1209 08:57:57.810309 (+ 16us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1209 08:57:57.810311 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1209 08:57:57.810340 (+ 29us) server_negotiation.cc:923] calling sasl_server_start()
1209 08:57:57.811824 (+ 1484us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1209 08:57:57.812237 (+ 413us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1209 08:57:57.812240 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1209 08:57:57.812242 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1209 08:57:57.812284 (+ 42us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1209 08:57:57.812507 (+ 223us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1209 08:57:57.812510 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1209 08:57:57.812512 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1209 08:57:57.812672 (+ 160us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1209 08:57:57.812767 (+ 95us) server_negotiation.cc:1036] Waiting for connection context
1209 08:57:57.812957 (+ 190us) server_negotiation.cc:300] Negotiation successful
1209 08:57:57.813064 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
WARNING: no policy specified for kudu/127.20.49.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.49.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.49.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765270668420570-20677-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.20.49.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765270668420570-20677-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.20.49.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.20.49.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.20.49.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765270668420570-20677-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.20.49.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765270668420570-20677-0/krb5kdc/kudu.keytab.
I20251209 08:57:57.858539 20677 external_mini_cluster.cc:1366] Running /tmp/dist-test-task8XuDZV/build/debug/bin/kudu
/tmp/dist-test-task8XuDZV/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765270668420570-20677-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765270668420570-20677-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765270668420570-20677-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765270668420570-20677-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.49.66:0
--local_ip_for_outbound_sockets=127.20.49.66
--webserver_interface=127.20.49.66
--webserver_port=0
--tserver_master_addrs=127.20.49.126:35129
--keytab_file=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765270668420570-20677-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.49.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.20.49.84:37945
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765270668420570-20677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765270668420570-20677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765270668420570-20677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251209 08:57:57.966876 22030 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251209 08:57:57.967115 22030 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251209 08:57:57.967206 22030 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251209 08:57:57.970441 22030 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251209 08:57:57.970510 22030 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251209 08:57:57.970597 22030 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20251209 08:57:57.970674 22030 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.49.66
I20251209 08:57:57.974824 22030 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.49.84:37945
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765270668420570-20677-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765270668420570-20677-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765270668420570-20677-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.49.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.49.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765270668420570-20677-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.49.66
--webserver_port=0
--webserver_require_spnego=true
--enable_txn_system_client_init=true
--tserver_master_addrs=127.20.49.126:35129
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22030
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.49.66
--log_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765270668420570-20677-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 468897755969d686fb08386f42f7835685e7953a
build type DEBUG
built by None at 09 Dec 2025 08:43:15 UTC on 24a791456cd2
build id 9426
I20251209 08:57:57.975853 22030 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251209 08:57:57.976642 22030 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251209 08:57:57.982925 22035 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251209 08:57:57.983070 22038 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251209 08:57:57.982906 22036 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251209 08:57:57.983348 22030 server_base.cc:1047] running on GCE node
I20251209 08:57:57.983664 22030 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251209 08:57:57.984151 22030 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251209 08:57:57.985316 22030 hybrid_clock.cc:648] HybridClock initialized: now 1765270677985308 us; error 29 us; skew 500 ppm
Dec 09 08:57:57 dist-test-slave-8w6c krb5kdc[21766](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765270677, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.49.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251209 08:57:57.987968 22030 init.cc:377] Logged in from keytab as kudu/127.20.49.66@KRBTEST.COM (short username kudu)
I20251209 08:57:57.989050 22030 webserver.cc:492] Webserver started at http://127.20.49.66:42123/ using document root <none> and password file <none>
I20251209 08:57:57.989595 22030 fs_manager.cc:362] Metadata directory not provided
I20251209 08:57:57.989650 22030 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251209 08:57:57.989871 22030 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251209 08:57:57.991575 22030 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765270668420570-20677-0/minicluster-data/ts-1/data/instance:
uuid: "6b11fb305a694cb0bb64203a3b2ba5c8"
format_stamp: "Formatted at 2025-12-09 08:57:57 on dist-test-slave-8w6c"
server_key: "c5d7c8dca320057dd8729a254be81876"
server_key_iv: "811df4892965a2c7b1ff616ff179b8ce"
server_key_version: "encryptionkey@0"
I20251209 08:57:57.992053 22030 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765270668420570-20677-0/minicluster-data/ts-1/wal/instance:
uuid: "6b11fb305a694cb0bb64203a3b2ba5c8"
format_stamp: "Formatted at 2025-12-09 08:57:57 on dist-test-slave-8w6c"
server_key: "c5d7c8dca320057dd8729a254be81876"
server_key_iv: "811df4892965a2c7b1ff616ff179b8ce"
server_key_version: "encryptionkey@0"
I20251209 08:57:57.995579 22030 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20251209 08:57:57.997800 22045 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251209 08:57:57.998899 22030 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20251209 08:57:57.999012 22030 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765270668420570-20677-0/minicluster-data/ts-1/data,/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765270668420570-20677-0/minicluster-data/ts-1/wal
uuid: "6b11fb305a694cb0bb64203a3b2ba5c8"
format_stamp: "Formatted at 2025-12-09 08:57:57 on dist-test-slave-8w6c"
server_key: "c5d7c8dca320057dd8729a254be81876"
server_key_iv: "811df4892965a2c7b1ff616ff179b8ce"
server_key_version: "encryptionkey@0"
I20251209 08:57:57.999111 22030 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765270668420570-20677-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765270668420570-20677-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765270668420570-20677-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251209 08:57:58.011688 22030 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251209 08:57:58.015025 22030 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251209 08:57:58.015230 22030 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251209 08:57:58.016746 22030 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251209 08:57:58.016801 22030 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251209 08:57:58.016861 22030 ts_tablet_manager.cc:616] Registered 0 tablets
I20251209 08:57:58.016892 22030 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Dec 09 08:57:58 dist-test-slave-8w6c krb5kdc[21766](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765270677, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.49.66@KRBTEST.COM for kudu/127.20.49.126@KRBTEST.COM
I20251209 08:57:58.028175 22030 rpc_server.cc:307] RPC server started. Bound to: 127.20.49.66:39533
I20251209 08:57:58.028206 22160 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.49.66:39533 every 8 connection(s)
I20251209 08:57:58.029107 22030 server_base.cc:1179] Dumped server information to /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765270668420570-20677-0/minicluster-data/ts-1/data/info.pb
I20251209 08:57:58.029433 21863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:57:58.017392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.49.66:40403 (local address 127.20.49.126:35129)
1209 08:57:58.017548 (+ 156us) server_negotiation.cc:207] Beginning negotiation
1209 08:57:58.017554 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1209 08:57:58.018387 (+ 833us) server_negotiation.cc:408] Connection header received
1209 08:57:58.019535 (+ 1148us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1209 08:57:58.019538 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1209 08:57:58.019588 (+ 50us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1209 08:57:58.019663 (+ 75us) server_negotiation.cc:227] Negotiated authn=SASL
1209 08:57:58.021460 (+ 1797us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:57:58.022097 (+ 637us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1209 08:57:58.023065 (+ 968us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:57:58.023279 (+ 214us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:57:58.025925 (+ 2646us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1209 08:57:58.025945 (+ 20us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1209 08:57:58.025947 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1209 08:57:58.025973 (+ 26us) server_negotiation.cc:923] calling sasl_server_start()
1209 08:57:58.027554 (+ 1581us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1209 08:57:58.028053 (+ 499us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1209 08:57:58.028059 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1209 08:57:58.028061 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1209 08:57:58.028105 (+ 44us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1209 08:57:58.028385 (+ 280us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1209 08:57:58.028388 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1209 08:57:58.028390 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1209 08:57:58.028545 (+ 155us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1209 08:57:58.028634 (+ 89us) server_negotiation.cc:1036] Waiting for connection context
1209 08:57:58.029122 (+ 488us) server_negotiation.cc:300] Negotiation successful
1209 08:57:58.029255 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20251209 08:57:58.030124 22054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:57:58.017660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.49.126:35129 (local address 127.20.49.66:40403)
1209 08:57:58.018209 (+ 549us) negotiation.cc:107] Waiting for socket to connect
1209 08:57:58.018264 (+ 55us) client_negotiation.cc:174] Beginning negotiation
1209 08:57:58.019253 (+ 989us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1209 08:57:58.019838 (+ 585us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1209 08:57:58.019853 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
1209 08:57:58.020469 (+ 616us) client_negotiation.cc:189] Negotiated authn=SASL
1209 08:57:58.021260 (+ 791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:57:58.021288 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:57:58.022229 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1209 08:57:58.022234 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1209 08:57:58.022935 (+ 701us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:57:58.022947 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:57:58.023218 (+ 271us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:57:58.023988 (+ 770us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1209 08:57:58.024019 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
1209 08:57:58.025772 (+ 1753us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1209 08:57:58.027673 (+ 1901us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1209 08:57:58.027679 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1209 08:57:58.027692 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
1209 08:57:58.027952 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1209 08:57:58.028188 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1209 08:57:58.028192 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1209 08:57:58.028194 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1209 08:57:58.028304 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1209 08:57:58.028647 (+ 343us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1209 08:57:58.028656 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1209 08:57:58.028890 (+ 234us) client_negotiation.cc:769] Sending connection context
1209 08:57:58.029076 (+ 186us) client_negotiation.cc:240] Negotiation successful
1209 08:57:58.029340 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":108,"threads_started":1}
I20251209 08:57:58.031191 22161 heartbeater.cc:344] Connected to a master server at 127.20.49.126:35129
I20251209 08:57:58.031365 22161 heartbeater.cc:461] Registering TS with master...
I20251209 08:57:58.031844 22161 heartbeater.cc:507] Master 127.20.49.126:35129 requested a full tablet report, sending...
I20251209 08:57:58.032816 21814 ts_manager.cc:194] Registered new tserver with Master: 6b11fb305a694cb0bb64203a3b2ba5c8 (127.20.49.66:39533)
I20251209 08:57:58.033602 21814 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.49.66@KRBTEST.COM'} at 127.20.49.66:40403
I20251209 08:57:58.034834 20677 external_mini_cluster.cc:1428] Started /tmp/dist-test-task8XuDZV/build/debug/bin/kudu as pid 22030
I20251209 08:57:58.034941 20677 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765270668420570-20677-0/minicluster-data/ts-1/wal/instance
I20251209 08:57:58.035208 20677 external_mini_cluster.cc:1468] Setting key effde2f6890a2f57f258b00f61c2325c
I20251209 08:57:58.039861 22167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:57:58.033101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.49.126:35129 (local address 127.20.49.66:34835)
1209 08:57:58.033417 (+ 316us) negotiation.cc:107] Waiting for socket to connect
1209 08:57:58.033432 (+ 15us) client_negotiation.cc:174] Beginning negotiation
1209 08:57:58.033538 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1209 08:57:58.033794 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1209 08:57:58.033797 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1209 08:57:58.034000 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
1209 08:57:58.034268 (+ 268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:57:58.034276 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:57:58.035177 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1209 08:57:58.035180 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1209 08:57:58.035658 (+ 478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:57:58.035667 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:57:58.035777 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:57:58.036432 (+ 655us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1209 08:57:58.036447 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
1209 08:57:58.036817 (+ 370us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1209 08:57:58.038506 (+ 1689us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1209 08:57:58.038513 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1209 08:57:58.038517 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
1209 08:57:58.038761 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1209 08:57:58.038966 (+ 205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1209 08:57:58.038969 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1209 08:57:58.038971 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1209 08:57:58.039031 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1209 08:57:58.039454 (+ 423us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1209 08:57:58.039457 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1209 08:57:58.039531 (+ 74us) client_negotiation.cc:769] Sending connection context
1209 08:57:58.039618 (+ 87us) client_negotiation.cc:240] Negotiation successful
1209 08:57:58.039729 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":98,"threads_started":1}
I20251209 08:57:58.040055 21863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:57:58.033161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.49.66:34835 (local address 127.20.49.126:35129)
1209 08:57:58.033268 (+ 107us) server_negotiation.cc:207] Beginning negotiation
1209 08:57:58.033271 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1209 08:57:58.033504 (+ 233us) server_negotiation.cc:408] Connection header received
1209 08:57:58.033624 (+ 120us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1209 08:57:58.033626 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1209 08:57:58.033664 (+ 38us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1209 08:57:58.033771 (+ 107us) server_negotiation.cc:227] Negotiated authn=SASL
1209 08:57:58.034381 (+ 610us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:57:58.035025 (+ 644us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1209 08:57:58.035772 (+ 747us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:57:58.035977 (+ 205us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:57:58.036943 (+ 966us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1209 08:57:58.036957 (+ 14us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1209 08:57:58.036959 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1209 08:57:58.036993 (+ 34us) server_negotiation.cc:923] calling sasl_server_start()
1209 08:57:58.038410 (+ 1417us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1209 08:57:58.038854 (+ 444us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1209 08:57:58.038857 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1209 08:57:58.038858 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1209 08:57:58.038894 (+ 36us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1209 08:57:58.039106 (+ 212us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1209 08:57:58.039108 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1209 08:57:58.039109 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1209 08:57:58.039351 (+ 242us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1209 08:57:58.039459 (+ 108us) server_negotiation.cc:1036] Waiting for connection context
1209 08:57:58.039694 (+ 235us) server_negotiation.cc:300] Negotiation successful
1209 08:57:58.039829 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
WARNING: no policy specified for kudu/127.20.49.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.49.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.49.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765270668420570-20677-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.20.49.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765270668420570-20677-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.20.49.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.20.49.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.20.49.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765270668420570-20677-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.20.49.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765270668420570-20677-0/krb5kdc/kudu.keytab.
I20251209 08:57:58.084873 20677 external_mini_cluster.cc:1366] Running /tmp/dist-test-task8XuDZV/build/debug/bin/kudu
/tmp/dist-test-task8XuDZV/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765270668420570-20677-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765270668420570-20677-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765270668420570-20677-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765270668420570-20677-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.49.67:0
--local_ip_for_outbound_sockets=127.20.49.67
--webserver_interface=127.20.49.67
--webserver_port=0
--tserver_master_addrs=127.20.49.126:35129
--keytab_file=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765270668420570-20677-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.49.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.20.49.84:37945
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765270668420570-20677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765270668420570-20677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765270668420570-20677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251209 08:57:58.191232 22172 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251209 08:57:58.191444 22172 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251209 08:57:58.191504 22172 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251209 08:57:58.194715 22172 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251209 08:57:58.194772 22172 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251209 08:57:58.194847 22172 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20251209 08:57:58.194927 22172 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.49.67
I20251209 08:57:58.199034 22172 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.49.84:37945
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765270668420570-20677-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765270668420570-20677-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765270668420570-20677-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.49.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.49.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765270668420570-20677-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.49.67
--webserver_port=0
--webserver_require_spnego=true
--enable_txn_system_client_init=true
--tserver_master_addrs=127.20.49.126:35129
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22172
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.49.67
--log_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765270668420570-20677-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 468897755969d686fb08386f42f7835685e7953a
build type DEBUG
built by None at 09 Dec 2025 08:43:15 UTC on 24a791456cd2
build id 9426
I20251209 08:57:58.200069 22172 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251209 08:57:58.200935 22172 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251209 08:57:58.207477 22177 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251209 08:57:58.207605 22172 server_base.cc:1047] running on GCE node
W20251209 08:57:58.207496 22180 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251209 08:57:58.207472 22178 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251209 08:57:58.208073 22172 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251209 08:57:58.208623 22172 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251209 08:57:58.209751 22172 hybrid_clock.cc:648] HybridClock initialized: now 1765270678209737 us; error 26 us; skew 500 ppm
Dec 09 08:57:58 dist-test-slave-8w6c krb5kdc[21766](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765270678, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.49.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251209 08:57:58.212373 22172 init.cc:377] Logged in from keytab as kudu/127.20.49.67@KRBTEST.COM (short username kudu)
I20251209 08:57:58.213357 22172 webserver.cc:492] Webserver started at http://127.20.49.67:43319/ using document root <none> and password file <none>
I20251209 08:57:58.213933 22172 fs_manager.cc:362] Metadata directory not provided
I20251209 08:57:58.213990 22172 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251209 08:57:58.214174 22172 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251209 08:57:58.215847 22172 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765270668420570-20677-0/minicluster-data/ts-2/data/instance:
uuid: "2da3b1d7e30440909571db652b3bf08e"
format_stamp: "Formatted at 2025-12-09 08:57:58 on dist-test-slave-8w6c"
server_key: "9ef71097a0cdc416c3870e681f4b81bd"
server_key_iv: "f80a672cbcd1ec91730a925280f2df76"
server_key_version: "encryptionkey@0"
I20251209 08:57:58.216339 22172 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765270668420570-20677-0/minicluster-data/ts-2/wal/instance:
uuid: "2da3b1d7e30440909571db652b3bf08e"
format_stamp: "Formatted at 2025-12-09 08:57:58 on dist-test-slave-8w6c"
server_key: "9ef71097a0cdc416c3870e681f4b81bd"
server_key_iv: "f80a672cbcd1ec91730a925280f2df76"
server_key_version: "encryptionkey@0"
I20251209 08:57:58.219722 22172 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20251209 08:57:58.221866 22187 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251209 08:57:58.222887 22172 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20251209 08:57:58.223001 22172 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765270668420570-20677-0/minicluster-data/ts-2/data,/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765270668420570-20677-0/minicluster-data/ts-2/wal
uuid: "2da3b1d7e30440909571db652b3bf08e"
format_stamp: "Formatted at 2025-12-09 08:57:58 on dist-test-slave-8w6c"
server_key: "9ef71097a0cdc416c3870e681f4b81bd"
server_key_iv: "f80a672cbcd1ec91730a925280f2df76"
server_key_version: "encryptionkey@0"
I20251209 08:57:58.223107 22172 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765270668420570-20677-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765270668420570-20677-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765270668420570-20677-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251209 08:57:58.242012 22172 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251209 08:57:58.245013 22172 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251209 08:57:58.245201 22172 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251209 08:57:58.247078 22172 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251209 08:57:58.247138 22172 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251209 08:57:58.247198 22172 ts_tablet_manager.cc:616] Registered 0 tablets
I20251209 08:57:58.247232 22172 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Dec 09 08:57:58 dist-test-slave-8w6c krb5kdc[21766](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765270678, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.49.67@KRBTEST.COM for kudu/127.20.49.126@KRBTEST.COM
I20251209 08:57:58.258414 22172 rpc_server.cc:307] RPC server started. Bound to: 127.20.49.67:35127
I20251209 08:57:58.258473 22302 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.49.67:35127 every 8 connection(s)
I20251209 08:57:58.259568 22172 server_base.cc:1179] Dumped server information to /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765270668420570-20677-0/minicluster-data/ts-2/data/info.pb
I20251209 08:57:58.259706 21863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:57:58.247670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.49.67:44295 (local address 127.20.49.126:35129)
1209 08:57:58.247829 (+ 159us) server_negotiation.cc:207] Beginning negotiation
1209 08:57:58.247834 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1209 08:57:58.248589 (+ 755us) server_negotiation.cc:408] Connection header received
1209 08:57:58.249468 (+ 879us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1209 08:57:58.249471 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1209 08:57:58.249536 (+ 65us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1209 08:57:58.249611 (+ 75us) server_negotiation.cc:227] Negotiated authn=SASL
1209 08:57:58.251054 (+ 1443us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:57:58.251751 (+ 697us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1209 08:57:58.252730 (+ 979us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:57:58.252974 (+ 244us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:57:58.255844 (+ 2870us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1209 08:57:58.255865 (+ 21us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1209 08:57:58.255867 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1209 08:57:58.255892 (+ 25us) server_negotiation.cc:923] calling sasl_server_start()
1209 08:57:58.257628 (+ 1736us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1209 08:57:58.258182 (+ 554us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1209 08:57:58.258188 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1209 08:57:58.258190 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1209 08:57:58.258242 (+ 52us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1209 08:57:58.258613 (+ 371us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1209 08:57:58.258617 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1209 08:57:58.258619 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1209 08:57:58.258809 (+ 190us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1209 08:57:58.258913 (+ 104us) server_negotiation.cc:1036] Waiting for connection context
1209 08:57:58.259397 (+ 484us) server_negotiation.cc:300] Negotiation successful
1209 08:57:58.259533 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20251209 08:57:58.260332 22196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:57:58.247946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.49.126:35129 (local address 127.20.49.67:44295)
1209 08:57:58.248427 (+ 481us) negotiation.cc:107] Waiting for socket to connect
1209 08:57:58.248470 (+ 43us) client_negotiation.cc:174] Beginning negotiation
1209 08:57:58.249256 (+ 786us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1209 08:57:58.249803 (+ 547us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1209 08:57:58.249811 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
1209 08:57:58.250337 (+ 526us) client_negotiation.cc:189] Negotiated authn=SASL
1209 08:57:58.250902 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:57:58.250914 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:57:58.251886 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1209 08:57:58.251894 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1209 08:57:58.252586 (+ 692us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:57:58.252601 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:57:58.252858 (+ 257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:57:58.253767 (+ 909us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1209 08:57:58.253801 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
1209 08:57:58.255686 (+ 1885us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1209 08:57:58.257805 (+ 2119us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1209 08:57:58.257810 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1209 08:57:58.257822 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
1209 08:57:58.258057 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1209 08:57:58.258352 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1209 08:57:58.258356 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1209 08:57:58.258359 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
1209 08:57:58.258478 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1209 08:57:58.258913 (+ 435us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1209 08:57:58.258918 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1209 08:57:58.259143 (+ 225us) client_negotiation.cc:769] Sending connection context
1209 08:57:58.259312 (+ 169us) client_negotiation.cc:240] Negotiation successful
1209 08:57:58.259522 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":132,"threads_started":1}
I20251209 08:57:58.260845 20677 external_mini_cluster.cc:1428] Started /tmp/dist-test-task8XuDZV/build/debug/bin/kudu as pid 22172
I20251209 08:57:58.260969 20677 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765270668420570-20677-0/minicluster-data/ts-2/wal/instance
I20251209 08:57:58.261251 20677 external_mini_cluster.cc:1468] Setting key b4dd3abd8ae7ee3ce9ad24423561ab97
I20251209 08:57:58.261729 22303 heartbeater.cc:344] Connected to a master server at 127.20.49.126:35129
I20251209 08:57:58.261919 22303 heartbeater.cc:461] Registering TS with master...
I20251209 08:57:58.262553 22303 heartbeater.cc:507] Master 127.20.49.126:35129 requested a full tablet report, sending...
I20251209 08:57:58.263502 21813 ts_manager.cc:194] Registered new tserver with Master: 2da3b1d7e30440909571db652b3bf08e (127.20.49.67:35127)
I20251209 08:57:58.264178 21813 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.49.67@KRBTEST.COM'} at 127.20.49.67:44295
I20251209 08:57:58.270053 22309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:57:58.263815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.49.126:35129 (local address 127.20.49.67:43887)
1209 08:57:58.264495 (+ 680us) negotiation.cc:107] Waiting for socket to connect
1209 08:57:58.264509 (+ 14us) client_negotiation.cc:174] Beginning negotiation
1209 08:57:58.264605 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1209 08:57:58.264849 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1209 08:57:58.264852 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1209 08:57:58.265051 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
1209 08:57:58.265339 (+ 288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:57:58.265348 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:57:58.266061 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1209 08:57:58.266065 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1209 08:57:58.266526 (+ 461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:57:58.266535 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:57:58.266629 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:57:58.267156 (+ 527us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1209 08:57:58.267168 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
1209 08:57:58.267435 (+ 267us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1209 08:57:58.269032 (+ 1597us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1209 08:57:58.269035 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1209 08:57:58.269037 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1209 08:57:58.269191 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1209 08:57:58.269374 (+ 183us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1209 08:57:58.269376 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1209 08:57:58.269377 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1209 08:57:58.269412 (+ 35us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1209 08:57:58.269642 (+ 230us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1209 08:57:58.269644 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1209 08:57:58.269750 (+ 106us) client_negotiation.cc:769] Sending connection context
1209 08:57:58.269831 (+ 81us) client_negotiation.cc:240] Negotiation successful
1209 08:57:58.269924 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":612,"thread_start_us":73,"threads_started":1}
I20251209 08:57:58.270076 21863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:57:58.263870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.49.67:43887 (local address 127.20.49.126:35129)
1209 08:57:58.263992 (+ 122us) server_negotiation.cc:207] Beginning negotiation
1209 08:57:58.263996 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1209 08:57:58.264583 (+ 587us) server_negotiation.cc:408] Connection header received
1209 08:57:58.264708 (+ 125us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1209 08:57:58.264711 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1209 08:57:58.264750 (+ 39us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1209 08:57:58.264816 (+ 66us) server_negotiation.cc:227] Negotiated authn=SASL
1209 08:57:58.265452 (+ 636us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:57:58.265959 (+ 507us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1209 08:57:58.266634 (+ 675us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:57:58.266760 (+ 126us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:57:58.267545 (+ 785us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1209 08:57:58.267559 (+ 14us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1209 08:57:58.267561 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1209 08:57:58.267582 (+ 21us) server_negotiation.cc:923] calling sasl_server_start()
1209 08:57:58.268936 (+ 1354us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1209 08:57:58.269260 (+ 324us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1209 08:57:58.269263 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1209 08:57:58.269264 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1209 08:57:58.269312 (+ 48us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1209 08:57:58.269468 (+ 156us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1209 08:57:58.269470 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1209 08:57:58.269471 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1209 08:57:58.269582 (+ 111us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1209 08:57:58.269650 (+ 68us) server_negotiation.cc:1036] Waiting for connection context
1209 08:57:58.269886 (+ 236us) server_negotiation.cc:300] Negotiation successful
1209 08:57:58.269972 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20251209 08:57:58.274744 20677 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
Dec 09 08:57:58 dist-test-slave-8w6c krb5kdc[21766](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765270678, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Dec 09 08:57:58 dist-test-slave-8w6c krb5kdc[21766](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765270678, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.20.49.126@KRBTEST.COM
I20251209 08:57:58.301558 21863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:57:58.293812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42554 (local address 127.20.49.126:35129)
1209 08:57:58.293958 (+ 146us) server_negotiation.cc:207] Beginning negotiation
1209 08:57:58.293963 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1209 08:57:58.293975 (+ 12us) server_negotiation.cc:408] Connection header received
1209 08:57:58.294024 (+ 49us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1209 08:57:58.294027 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1209 08:57:58.294077 (+ 50us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1209 08:57:58.294162 (+ 85us) server_negotiation.cc:227] Negotiated authn=SASL
1209 08:57:58.294979 (+ 817us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:57:58.295576 (+ 597us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1209 08:57:58.296287 (+ 711us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:57:58.296513 (+ 226us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:57:58.298672 (+ 2159us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1209 08:57:58.298691 (+ 19us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1209 08:57:58.298693 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1209 08:57:58.298718 (+ 25us) server_negotiation.cc:923] calling sasl_server_start()
1209 08:57:58.300143 (+ 1425us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1209 08:57:58.300531 (+ 388us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1209 08:57:58.300534 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1209 08:57:58.300536 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1209 08:57:58.300577 (+ 41us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1209 08:57:58.300785 (+ 208us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1209 08:57:58.300789 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1209 08:57:58.300791 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1209 08:57:58.300937 (+ 146us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1209 08:57:58.301075 (+ 138us) server_negotiation.cc:1036] Waiting for connection context
1209 08:57:58.301275 (+ 200us) server_negotiation.cc:300] Negotiation successful
1209 08:57:58.301396 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20251209 08:57:58.304311 21813 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42554:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251209 08:57:58.306892 21813 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251209 08:57:58.319563 22322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:57:58.315659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.49.67:35127 (local address 127.0.0.1:40448)
1209 08:57:58.316082 (+ 423us) negotiation.cc:107] Waiting for socket to connect
1209 08:57:58.316096 (+ 14us) client_negotiation.cc:174] Beginning negotiation
1209 08:57:58.316194 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1209 08:57:58.316963 (+ 769us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1209 08:57:58.316969 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
1209 08:57:58.317015 (+ 46us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1209 08:57:58.317285 (+ 270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:57:58.317307 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:57:58.318310 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1209 08:57:58.318313 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1209 08:57:58.319055 (+ 742us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:57:58.319063 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:57:58.319172 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:57:58.319228 (+ 56us) client_negotiation.cc:769] Sending connection context
1209 08:57:58.319329 (+ 101us) client_negotiation.cc:240] Negotiation successful
1209 08:57:58.319416 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":361,"thread_start_us":91,"threads_started":1}
I20251209 08:57:58.320676 22324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:57:58.316043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40448 (local address 127.20.49.67:35127)
1209 08:57:58.316371 (+ 328us) server_negotiation.cc:207] Beginning negotiation
1209 08:57:58.316377 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1209 08:57:58.316397 (+ 20us) server_negotiation.cc:408] Connection header received
1209 08:57:58.316462 (+ 65us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1209 08:57:58.316466 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1209 08:57:58.316642 (+ 176us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1209 08:57:58.316788 (+ 146us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1209 08:57:58.317419 (+ 631us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:57:58.318187 (+ 768us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1209 08:57:58.319919 (+ 1732us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:57:58.320370 (+ 451us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:57:58.320451 (+ 81us) server_negotiation.cc:1036] Waiting for connection context
1209 08:57:58.320506 (+ 55us) server_negotiation.cc:300] Negotiation successful
1209 08:57:58.320570 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":236,"thread_start_us":130,"threads_started":1}
I20251209 08:57:58.320711 22325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:57:58.316394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40412 (local address 127.20.49.65:43253)
1209 08:57:58.316650 (+ 256us) server_negotiation.cc:207] Beginning negotiation
1209 08:57:58.316658 (+ 8us) server_negotiation.cc:400] Waiting for connection header
1209 08:57:58.316678 (+ 20us) server_negotiation.cc:408] Connection header received
1209 08:57:58.316756 (+ 78us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1209 08:57:58.316761 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1209 08:57:58.316899 (+ 138us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1209 08:57:58.317129 (+ 230us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1209 08:57:58.317464 (+ 335us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:57:58.318613 (+ 1149us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1209 08:57:58.319748 (+ 1135us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:57:58.320280 (+ 532us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:57:58.320410 (+ 130us) server_negotiation.cc:1036] Waiting for connection context
1209 08:57:58.320478 (+ 68us) server_negotiation.cc:300] Negotiation successful
1209 08:57:58.320573 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":162,"thread_start_us":76,"threads_started":1}
I20251209 08:57:58.319953 22320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:57:58.315144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.49.65:43253 (local address 127.0.0.1:40412)
1209 08:57:58.315494 (+ 350us) negotiation.cc:107] Waiting for socket to connect
1209 08:57:58.315536 (+ 42us) client_negotiation.cc:174] Beginning negotiation
1209 08:57:58.315916 (+ 380us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1209 08:57:58.317105 (+ 1189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1209 08:57:58.317109 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
1209 08:57:58.317124 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1209 08:57:58.317329 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:57:58.317339 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:57:58.318732 (+ 1393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1209 08:57:58.318736 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1209 08:57:58.319594 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:57:58.319606 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:57:58.319716 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:57:58.319730 (+ 14us) client_negotiation.cc:769] Sending connection context
1209 08:57:58.319778 (+ 48us) client_negotiation.cc:240] Negotiation successful
1209 08:57:58.319829 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":192,"spinlock_wait_cycles":23296,"thread_start_us":91,"threads_started":1}
I20251209 08:57:58.320101 22321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:57:58.314966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.49.66:39533 (local address 127.0.0.1:41788)
1209 08:57:58.315797 (+ 831us) negotiation.cc:107] Waiting for socket to connect
1209 08:57:58.315809 (+ 12us) client_negotiation.cc:174] Beginning negotiation
1209 08:57:58.315916 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1209 08:57:58.316933 (+ 1017us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1209 08:57:58.316940 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
1209 08:57:58.316961 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1209 08:57:58.317282 (+ 321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:57:58.317290 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:57:58.318852 (+ 1562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1209 08:57:58.318855 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1209 08:57:58.319708 (+ 853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:57:58.319718 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:57:58.319822 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:57:58.319840 (+ 18us) client_negotiation.cc:769] Sending connection context
1209 08:57:58.319887 (+ 47us) client_negotiation.cc:240] Negotiation successful
1209 08:57:58.319944 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":356,"spinlock_wait_cycles":896,"thread_start_us":85,"threads_started":1}
I20251209 08:57:58.321154 22323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:57:58.315641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41788 (local address 127.20.49.66:39533)
1209 08:57:58.316457 (+ 816us) server_negotiation.cc:207] Beginning negotiation
1209 08:57:58.316466 (+ 9us) server_negotiation.cc:400] Waiting for connection header
1209 08:57:58.316490 (+ 24us) server_negotiation.cc:408] Connection header received
1209 08:57:58.316576 (+ 86us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1209 08:57:58.316581 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1209 08:57:58.316751 (+ 170us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1209 08:57:58.317446 (+ 695us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1209 08:57:58.317610 (+ 164us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:57:58.318723 (+ 1113us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1209 08:57:58.320032 (+ 1309us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:57:58.320530 (+ 498us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:57:58.320611 (+ 81us) server_negotiation.cc:1036] Waiting for connection context
1209 08:57:58.320669 (+ 58us) server_negotiation.cc:300] Negotiation successful
1209 08:57:58.320751 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":713,"thread_start_us":100,"threads_started":1}
I20251209 08:57:58.323678 21953 tablet_service.cc:1505] Processing CreateTablet for tablet 698ac2c2e52949d591e7ad678b60c888 (DEFAULT_TABLE table=test-table [id=5bb7dbac04844e2994052f16f33440e5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251209 08:57:58.323680 22237 tablet_service.cc:1505] Processing CreateTablet for tablet 698ac2c2e52949d591e7ad678b60c888 (DEFAULT_TABLE table=test-table [id=5bb7dbac04844e2994052f16f33440e5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251209 08:57:58.324558 21953 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 698ac2c2e52949d591e7ad678b60c888. 1 dirs total, 0 dirs full, 0 dirs failed
I20251209 08:57:58.324589 22237 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 698ac2c2e52949d591e7ad678b60c888. 1 dirs total, 0 dirs full, 0 dirs failed
I20251209 08:57:58.329998 22326 tablet_bootstrap.cc:492] T 698ac2c2e52949d591e7ad678b60c888 P 4764a44e2a814f6db9d0185ea0bc2425: Bootstrap starting.
I20251209 08:57:58.331346 22327 tablet_bootstrap.cc:492] T 698ac2c2e52949d591e7ad678b60c888 P 2da3b1d7e30440909571db652b3bf08e: Bootstrap starting.
I20251209 08:57:58.332697 22095 tablet_service.cc:1505] Processing CreateTablet for tablet 698ac2c2e52949d591e7ad678b60c888 (DEFAULT_TABLE table=test-table [id=5bb7dbac04844e2994052f16f33440e5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251209 08:57:58.332994 22326 tablet_bootstrap.cc:654] T 698ac2c2e52949d591e7ad678b60c888 P 4764a44e2a814f6db9d0185ea0bc2425: Neither blocks nor log segments found. Creating new log.
I20251209 08:57:58.333479 22327 tablet_bootstrap.cc:654] T 698ac2c2e52949d591e7ad678b60c888 P 2da3b1d7e30440909571db652b3bf08e: Neither blocks nor log segments found. Creating new log.
I20251209 08:57:58.333627 22095 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 698ac2c2e52949d591e7ad678b60c888. 1 dirs total, 0 dirs full, 0 dirs failed
I20251209 08:57:58.333943 22326 log.cc:826] T 698ac2c2e52949d591e7ad678b60c888 P 4764a44e2a814f6db9d0185ea0bc2425: Log is configured to *not* fsync() on all Append() calls
I20251209 08:57:58.334720 22327 log.cc:826] T 698ac2c2e52949d591e7ad678b60c888 P 2da3b1d7e30440909571db652b3bf08e: Log is configured to *not* fsync() on all Append() calls
I20251209 08:57:58.335846 22326 tablet_bootstrap.cc:492] T 698ac2c2e52949d591e7ad678b60c888 P 4764a44e2a814f6db9d0185ea0bc2425: No bootstrap required, opened a new log
I20251209 08:57:58.336040 22326 ts_tablet_manager.cc:1403] T 698ac2c2e52949d591e7ad678b60c888 P 4764a44e2a814f6db9d0185ea0bc2425: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20251209 08:57:58.338907 22326 raft_consensus.cc:359] T 698ac2c2e52949d591e7ad678b60c888 P 4764a44e2a814f6db9d0185ea0bc2425 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b11fb305a694cb0bb64203a3b2ba5c8" member_type: VOTER last_known_addr { host: "127.20.49.66" port: 39533 } } peers { permanent_uuid: "4764a44e2a814f6db9d0185ea0bc2425" member_type: VOTER last_known_addr { host: "127.20.49.65" port: 43253 } } peers { permanent_uuid: "2da3b1d7e30440909571db652b3bf08e" member_type: VOTER last_known_addr { host: "127.20.49.67" port: 35127 } }
I20251209 08:57:58.339372 22326 raft_consensus.cc:385] T 698ac2c2e52949d591e7ad678b60c888 P 4764a44e2a814f6db9d0185ea0bc2425 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251209 08:57:58.339394 22330 tablet_bootstrap.cc:492] T 698ac2c2e52949d591e7ad678b60c888 P 6b11fb305a694cb0bb64203a3b2ba5c8: Bootstrap starting.
I20251209 08:57:58.339483 22326 raft_consensus.cc:740] T 698ac2c2e52949d591e7ad678b60c888 P 4764a44e2a814f6db9d0185ea0bc2425 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4764a44e2a814f6db9d0185ea0bc2425, State: Initialized, Role: FOLLOWER
I20251209 08:57:58.339641 22327 tablet_bootstrap.cc:492] T 698ac2c2e52949d591e7ad678b60c888 P 2da3b1d7e30440909571db652b3bf08e: No bootstrap required, opened a new log
I20251209 08:57:58.339870 22327 ts_tablet_manager.cc:1403] T 698ac2c2e52949d591e7ad678b60c888 P 2da3b1d7e30440909571db652b3bf08e: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.005s
I20251209 08:57:58.340072 22326 consensus_queue.cc:260] T 698ac2c2e52949d591e7ad678b60c888 P 4764a44e2a814f6db9d0185ea0bc2425 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b11fb305a694cb0bb64203a3b2ba5c8" member_type: VOTER last_known_addr { host: "127.20.49.66" port: 39533 } } peers { permanent_uuid: "4764a44e2a814f6db9d0185ea0bc2425" member_type: VOTER last_known_addr { host: "127.20.49.65" port: 43253 } } peers { permanent_uuid: "2da3b1d7e30440909571db652b3bf08e" member_type: VOTER last_known_addr { host: "127.20.49.67" port: 35127 } }
I20251209 08:57:58.340781 22019 heartbeater.cc:499] Master 127.20.49.126:35129 was elected leader, sending a full tablet report...
I20251209 08:57:58.340947 22326 ts_tablet_manager.cc:1434] T 698ac2c2e52949d591e7ad678b60c888 P 4764a44e2a814f6db9d0185ea0bc2425: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20251209 08:57:58.341812 22330 tablet_bootstrap.cc:654] T 698ac2c2e52949d591e7ad678b60c888 P 6b11fb305a694cb0bb64203a3b2ba5c8: Neither blocks nor log segments found. Creating new log.
I20251209 08:57:58.342892 22330 log.cc:826] T 698ac2c2e52949d591e7ad678b60c888 P 6b11fb305a694cb0bb64203a3b2ba5c8: Log is configured to *not* fsync() on all Append() calls
I20251209 08:57:58.343714 22327 raft_consensus.cc:359] T 698ac2c2e52949d591e7ad678b60c888 P 2da3b1d7e30440909571db652b3bf08e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b11fb305a694cb0bb64203a3b2ba5c8" member_type: VOTER last_known_addr { host: "127.20.49.66" port: 39533 } } peers { permanent_uuid: "4764a44e2a814f6db9d0185ea0bc2425" member_type: VOTER last_known_addr { host: "127.20.49.65" port: 43253 } } peers { permanent_uuid: "2da3b1d7e30440909571db652b3bf08e" member_type: VOTER last_known_addr { host: "127.20.49.67" port: 35127 } }
I20251209 08:57:58.343959 22327 raft_consensus.cc:385] T 698ac2c2e52949d591e7ad678b60c888 P 2da3b1d7e30440909571db652b3bf08e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251209 08:57:58.344026 22327 raft_consensus.cc:740] T 698ac2c2e52949d591e7ad678b60c888 P 2da3b1d7e30440909571db652b3bf08e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2da3b1d7e30440909571db652b3bf08e, State: Initialized, Role: FOLLOWER
I20251209 08:57:58.344492 22327 consensus_queue.cc:260] T 698ac2c2e52949d591e7ad678b60c888 P 2da3b1d7e30440909571db652b3bf08e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b11fb305a694cb0bb64203a3b2ba5c8" member_type: VOTER last_known_addr { host: "127.20.49.66" port: 39533 } } peers { permanent_uuid: "4764a44e2a814f6db9d0185ea0bc2425" member_type: VOTER last_known_addr { host: "127.20.49.65" port: 43253 } } peers { permanent_uuid: "2da3b1d7e30440909571db652b3bf08e" member_type: VOTER last_known_addr { host: "127.20.49.67" port: 35127 } }
I20251209 08:57:58.344787 22330 tablet_bootstrap.cc:492] T 698ac2c2e52949d591e7ad678b60c888 P 6b11fb305a694cb0bb64203a3b2ba5c8: No bootstrap required, opened a new log
I20251209 08:57:58.345031 22330 ts_tablet_manager.cc:1403] T 698ac2c2e52949d591e7ad678b60c888 P 6b11fb305a694cb0bb64203a3b2ba5c8: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.005s
I20251209 08:57:58.346081 22327 ts_tablet_manager.cc:1434] T 698ac2c2e52949d591e7ad678b60c888 P 2da3b1d7e30440909571db652b3bf08e: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20251209 08:57:58.346289 22303 heartbeater.cc:499] Master 127.20.49.126:35129 was elected leader, sending a full tablet report...
I20251209 08:57:58.347918 22330 raft_consensus.cc:359] T 698ac2c2e52949d591e7ad678b60c888 P 6b11fb305a694cb0bb64203a3b2ba5c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b11fb305a694cb0bb64203a3b2ba5c8" member_type: VOTER last_known_addr { host: "127.20.49.66" port: 39533 } } peers { permanent_uuid: "4764a44e2a814f6db9d0185ea0bc2425" member_type: VOTER last_known_addr { host: "127.20.49.65" port: 43253 } } peers { permanent_uuid: "2da3b1d7e30440909571db652b3bf08e" member_type: VOTER last_known_addr { host: "127.20.49.67" port: 35127 } }
I20251209 08:57:58.348099 22330 raft_consensus.cc:385] T 698ac2c2e52949d591e7ad678b60c888 P 6b11fb305a694cb0bb64203a3b2ba5c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251209 08:57:58.348168 22330 raft_consensus.cc:740] T 698ac2c2e52949d591e7ad678b60c888 P 6b11fb305a694cb0bb64203a3b2ba5c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b11fb305a694cb0bb64203a3b2ba5c8, State: Initialized, Role: FOLLOWER
I20251209 08:57:58.348595 22330 consensus_queue.cc:260] T 698ac2c2e52949d591e7ad678b60c888 P 6b11fb305a694cb0bb64203a3b2ba5c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b11fb305a694cb0bb64203a3b2ba5c8" member_type: VOTER last_known_addr { host: "127.20.49.66" port: 39533 } } peers { permanent_uuid: "4764a44e2a814f6db9d0185ea0bc2425" member_type: VOTER last_known_addr { host: "127.20.49.65" port: 43253 } } peers { permanent_uuid: "2da3b1d7e30440909571db652b3bf08e" member_type: VOTER last_known_addr { host: "127.20.49.67" port: 35127 } }
I20251209 08:57:58.349298 22161 heartbeater.cc:499] Master 127.20.49.126:35129 was elected leader, sending a full tablet report...
I20251209 08:57:58.349385 22330 ts_tablet_manager.cc:1434] T 698ac2c2e52949d591e7ad678b60c888 P 6b11fb305a694cb0bb64203a3b2ba5c8: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20251209 08:57:58.476859 22333 raft_consensus.cc:493] T 698ac2c2e52949d591e7ad678b60c888 P 2da3b1d7e30440909571db652b3bf08e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251209 08:57:58.477136 22333 raft_consensus.cc:515] T 698ac2c2e52949d591e7ad678b60c888 P 2da3b1d7e30440909571db652b3bf08e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b11fb305a694cb0bb64203a3b2ba5c8" member_type: VOTER last_known_addr { host: "127.20.49.66" port: 39533 } } peers { permanent_uuid: "4764a44e2a814f6db9d0185ea0bc2425" member_type: VOTER last_known_addr { host: "127.20.49.65" port: 43253 } } peers { permanent_uuid: "2da3b1d7e30440909571db652b3bf08e" member_type: VOTER last_known_addr { host: "127.20.49.67" port: 35127 } }
I20251209 08:57:58.478341 22333 leader_election.cc:290] T 698ac2c2e52949d591e7ad678b60c888 P 2da3b1d7e30440909571db652b3bf08e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6b11fb305a694cb0bb64203a3b2ba5c8 (127.20.49.66:39533), 4764a44e2a814f6db9d0185ea0bc2425 (127.20.49.65:43253)
I20251209 08:57:58.481088 22196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:57:58.478558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.49.66:39533 (local address 127.20.49.67:53095)
1209 08:57:58.478686 (+ 128us) negotiation.cc:107] Waiting for socket to connect
1209 08:57:58.478702 (+ 16us) client_negotiation.cc:174] Beginning negotiation
1209 08:57:58.478804 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1209 08:57:58.479131 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1209 08:57:58.479134 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1209 08:57:58.479153 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1209 08:57:58.479364 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:57:58.479370 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:57:58.480106 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1209 08:57:58.480111 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1209 08:57:58.480679 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:57:58.480687 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:57:58.480800 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:57:58.480823 (+ 23us) client_negotiation.cc:769] Sending connection context
1209 08:57:58.480876 (+ 53us) client_negotiation.cc:240] Negotiation successful
1209 08:57:58.480937 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20251209 08:57:58.481391 22335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:57:58.478632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.49.65:43253 (local address 127.20.49.67:41689)
1209 08:57:58.478995 (+ 363us) negotiation.cc:107] Waiting for socket to connect
1209 08:57:58.479006 (+ 11us) client_negotiation.cc:174] Beginning negotiation
1209 08:57:58.479116 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1209 08:57:58.479365 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1209 08:57:58.479369 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
1209 08:57:58.479385 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1209 08:57:58.479623 (+ 238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:57:58.479628 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:57:58.480342 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1209 08:57:58.480347 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1209 08:57:58.480921 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:57:58.480928 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:57:58.481037 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:57:58.481055 (+ 18us) client_negotiation.cc:769] Sending connection context
1209 08:57:58.481106 (+ 51us) client_negotiation.cc:240] Negotiation successful
1209 08:57:58.481157 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":305,"thread_start_us":145,"threads_started":1}
I20251209 08:57:58.481597 22323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:57:58.478778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.49.67:53095 (local address 127.20.49.66:39533)
1209 08:57:58.478934 (+ 156us) server_negotiation.cc:207] Beginning negotiation
1209 08:57:58.478938 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1209 08:57:58.478950 (+ 12us) server_negotiation.cc:408] Connection header received
1209 08:57:58.478995 (+ 45us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1209 08:57:58.478997 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1209 08:57:58.479038 (+ 41us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1209 08:57:58.479109 (+ 71us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1209 08:57:58.479490 (+ 381us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:57:58.479994 (+ 504us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1209 08:57:58.480807 (+ 813us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:57:58.481325 (+ 518us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:57:58.481364 (+ 39us) server_negotiation.cc:1036] Waiting for connection context
1209 08:57:58.481414 (+ 50us) server_negotiation.cc:300] Negotiation successful
1209 08:57:58.481472 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20251209 08:57:58.481842 22325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:57:58.478698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.49.67:41689 (local address 127.20.49.65:43253)
1209 08:57:58.478841 (+ 143us) server_negotiation.cc:207] Beginning negotiation
1209 08:57:58.478846 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1209 08:57:58.479125 (+ 279us) server_negotiation.cc:408] Connection header received
1209 08:57:58.479210 (+ 85us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1209 08:57:58.479213 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1209 08:57:58.479260 (+ 47us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1209 08:57:58.479334 (+ 74us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1209 08:57:58.479749 (+ 415us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:57:58.480236 (+ 487us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1209 08:57:58.481048 (+ 812us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:57:58.481535 (+ 487us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:57:58.481576 (+ 41us) server_negotiation.cc:1036] Waiting for connection context
1209 08:57:58.481632 (+ 56us) server_negotiation.cc:300] Negotiation successful
1209 08:57:58.481724 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20251209 08:57:58.482204 22114 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "698ac2c2e52949d591e7ad678b60c888" candidate_uuid: "2da3b1d7e30440909571db652b3bf08e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b11fb305a694cb0bb64203a3b2ba5c8" is_pre_election: true
I20251209 08:57:58.482456 22114 raft_consensus.cc:2468] T 698ac2c2e52949d591e7ad678b60c888 P 6b11fb305a694cb0bb64203a3b2ba5c8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2da3b1d7e30440909571db652b3bf08e in term 0.
I20251209 08:57:58.482661 21973 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "698ac2c2e52949d591e7ad678b60c888" candidate_uuid: "2da3b1d7e30440909571db652b3bf08e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4764a44e2a814f6db9d0185ea0bc2425" is_pre_election: true
I20251209 08:57:58.482883 21973 raft_consensus.cc:2468] T 698ac2c2e52949d591e7ad678b60c888 P 4764a44e2a814f6db9d0185ea0bc2425 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2da3b1d7e30440909571db652b3bf08e in term 0.
I20251209 08:57:58.482882 22188 leader_election.cc:304] T 698ac2c2e52949d591e7ad678b60c888 P 2da3b1d7e30440909571db652b3bf08e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2da3b1d7e30440909571db652b3bf08e, 6b11fb305a694cb0bb64203a3b2ba5c8; no voters:
I20251209 08:57:58.483085 22333 raft_consensus.cc:2804] T 698ac2c2e52949d591e7ad678b60c888 P 2da3b1d7e30440909571db652b3bf08e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251209 08:57:58.483157 22333 raft_consensus.cc:493] T 698ac2c2e52949d591e7ad678b60c888 P 2da3b1d7e30440909571db652b3bf08e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251209 08:57:58.483217 22333 raft_consensus.cc:3060] T 698ac2c2e52949d591e7ad678b60c888 P 2da3b1d7e30440909571db652b3bf08e [term 0 FOLLOWER]: Advancing to term 1
I20251209 08:57:58.484247 22333 raft_consensus.cc:515] T 698ac2c2e52949d591e7ad678b60c888 P 2da3b1d7e30440909571db652b3bf08e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b11fb305a694cb0bb64203a3b2ba5c8" member_type: VOTER last_known_addr { host: "127.20.49.66" port: 39533 } } peers { permanent_uuid: "4764a44e2a814f6db9d0185ea0bc2425" member_type: VOTER last_known_addr { host: "127.20.49.65" port: 43253 } } peers { permanent_uuid: "2da3b1d7e30440909571db652b3bf08e" member_type: VOTER last_known_addr { host: "127.20.49.67" port: 35127 } }
I20251209 08:57:58.484601 22333 leader_election.cc:290] T 698ac2c2e52949d591e7ad678b60c888 P 2da3b1d7e30440909571db652b3bf08e [CANDIDATE]: Term 1 election: Requested vote from peers 6b11fb305a694cb0bb64203a3b2ba5c8 (127.20.49.66:39533), 4764a44e2a814f6db9d0185ea0bc2425 (127.20.49.65:43253)
I20251209 08:57:58.484912 22114 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "698ac2c2e52949d591e7ad678b60c888" candidate_uuid: "2da3b1d7e30440909571db652b3bf08e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b11fb305a694cb0bb64203a3b2ba5c8"
I20251209 08:57:58.485039 22114 raft_consensus.cc:3060] T 698ac2c2e52949d591e7ad678b60c888 P 6b11fb305a694cb0bb64203a3b2ba5c8 [term 0 FOLLOWER]: Advancing to term 1
I20251209 08:57:58.484973 21973 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "698ac2c2e52949d591e7ad678b60c888" candidate_uuid: "2da3b1d7e30440909571db652b3bf08e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4764a44e2a814f6db9d0185ea0bc2425"
I20251209 08:57:58.485162 21973 raft_consensus.cc:3060] T 698ac2c2e52949d591e7ad678b60c888 P 4764a44e2a814f6db9d0185ea0bc2425 [term 0 FOLLOWER]: Advancing to term 1
I20251209 08:57:58.486335 22114 raft_consensus.cc:2468] T 698ac2c2e52949d591e7ad678b60c888 P 6b11fb305a694cb0bb64203a3b2ba5c8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2da3b1d7e30440909571db652b3bf08e in term 1.
I20251209 08:57:58.486335 21973 raft_consensus.cc:2468] T 698ac2c2e52949d591e7ad678b60c888 P 4764a44e2a814f6db9d0185ea0bc2425 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2da3b1d7e30440909571db652b3bf08e in term 1.
I20251209 08:57:58.486641 22188 leader_election.cc:304] T 698ac2c2e52949d591e7ad678b60c888 P 2da3b1d7e30440909571db652b3bf08e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2da3b1d7e30440909571db652b3bf08e, 6b11fb305a694cb0bb64203a3b2ba5c8; no voters:
I20251209 08:57:58.486819 22333 raft_consensus.cc:2804] T 698ac2c2e52949d591e7ad678b60c888 P 2da3b1d7e30440909571db652b3bf08e [term 1 FOLLOWER]: Leader election won for term 1
I20251209 08:57:58.487039 22333 raft_consensus.cc:697] T 698ac2c2e52949d591e7ad678b60c888 P 2da3b1d7e30440909571db652b3bf08e [term 1 LEADER]: Becoming Leader. State: Replica: 2da3b1d7e30440909571db652b3bf08e, State: Running, Role: LEADER
I20251209 08:57:58.487380 22333 consensus_queue.cc:237] T 698ac2c2e52949d591e7ad678b60c888 P 2da3b1d7e30440909571db652b3bf08e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b11fb305a694cb0bb64203a3b2ba5c8" member_type: VOTER last_known_addr { host: "127.20.49.66" port: 39533 } } peers { permanent_uuid: "4764a44e2a814f6db9d0185ea0bc2425" member_type: VOTER last_known_addr { host: "127.20.49.65" port: 43253 } } peers { permanent_uuid: "2da3b1d7e30440909571db652b3bf08e" member_type: VOTER last_known_addr { host: "127.20.49.67" port: 35127 } }
I20251209 08:57:58.490559 21814 catalog_manager.cc:5654] T 698ac2c2e52949d591e7ad678b60c888 P 2da3b1d7e30440909571db652b3bf08e reported cstate change: term changed from 0 to 1, leader changed from <none> to 2da3b1d7e30440909571db652b3bf08e (127.20.49.67). New cstate: current_term: 1 leader_uuid: "2da3b1d7e30440909571db652b3bf08e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b11fb305a694cb0bb64203a3b2ba5c8" member_type: VOTER last_known_addr { host: "127.20.49.66" port: 39533 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4764a44e2a814f6db9d0185ea0bc2425" member_type: VOTER last_known_addr { host: "127.20.49.65" port: 43253 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2da3b1d7e30440909571db652b3bf08e" member_type: VOTER last_known_addr { host: "127.20.49.67" port: 35127 } health_report { overall_health: HEALTHY } } }
W20251209 08:57:58.510497 22304 tablet.cc:2378] T 698ac2c2e52949d591e7ad678b60c888 P 2da3b1d7e30440909571db652b3bf08e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20251209 08:57:58.513404 22324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:57:58.509818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40464 (local address 127.20.49.67:35127)
1209 08:57:58.509941 (+ 123us) server_negotiation.cc:207] Beginning negotiation
1209 08:57:58.509945 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1209 08:57:58.509958 (+ 13us) server_negotiation.cc:408] Connection header received
1209 08:57:58.510112 (+ 154us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1209 08:57:58.510115 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1209 08:57:58.510170 (+ 55us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1209 08:57:58.510252 (+ 82us) server_negotiation.cc:227] Negotiated authn=TOKEN
1209 08:57:58.510692 (+ 440us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:57:58.511439 (+ 747us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1209 08:57:58.512126 (+ 687us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:57:58.512430 (+ 304us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:57:58.512510 (+ 80us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1209 08:57:58.513029 (+ 519us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1209 08:57:58.513139 (+ 110us) server_negotiation.cc:1036] Waiting for connection context
1209 08:57:58.513221 (+ 82us) server_negotiation.cc:300] Negotiation successful
1209 08:57:58.513273 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20251209 08:57:58.518797 21973 raft_consensus.cc:1275] T 698ac2c2e52949d591e7ad678b60c888 P 4764a44e2a814f6db9d0185ea0bc2425 [term 1 FOLLOWER]: Refusing update from remote peer 2da3b1d7e30440909571db652b3bf08e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251209 08:57:58.518797 22114 raft_consensus.cc:1275] T 698ac2c2e52949d591e7ad678b60c888 P 6b11fb305a694cb0bb64203a3b2ba5c8 [term 1 FOLLOWER]: Refusing update from remote peer 2da3b1d7e30440909571db652b3bf08e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251209 08:57:58.519402 22333 consensus_queue.cc:1048] T 698ac2c2e52949d591e7ad678b60c888 P 2da3b1d7e30440909571db652b3bf08e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b11fb305a694cb0bb64203a3b2ba5c8" member_type: VOTER last_known_addr { host: "127.20.49.66" port: 39533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251209 08:57:58.519577 22336 consensus_queue.cc:1048] T 698ac2c2e52949d591e7ad678b60c888 P 2da3b1d7e30440909571db652b3bf08e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4764a44e2a814f6db9d0185ea0bc2425" member_type: VOTER last_known_addr { host: "127.20.49.65" port: 43253 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251209 08:57:58.526751 22339 mvcc.cc:204] Tried to move back new op lower bound from 7230548699206537216 to 7230548699086938112. Current Snapshot: MvccSnapshot[applied={T|T < 7230548699206537216}]
I20251209 08:57:58.528478 22340 mvcc.cc:204] Tried to move back new op lower bound from 7230548699206537216 to 7230548699086938112. Current Snapshot: MvccSnapshot[applied={T|T < 7230548699206537216}]
I20251209 08:57:58.529268 22341 mvcc.cc:204] Tried to move back new op lower bound from 7230548699206537216 to 7230548699086938112. Current Snapshot: MvccSnapshot[applied={T|T < 7230548699206537216}]
I20251209 08:57:58.537416 21813 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42554:
table { table_name: "test-table" } modify_external_catalogs: true
I20251209 08:57:58.537567 21813 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42554:
table { table_name: "test-table" } modify_external_catalogs: true
I20251209 08:57:58.540521 21813 catalog_manager.cc:5941] T 00000000000000000000000000000000 P fc08086ef48c4a79a51eac09446fa095: Sending DeleteTablet for 3 replicas of tablet 698ac2c2e52949d591e7ad678b60c888
I20251209 08:57:58.541352 21953 tablet_service.cc:1552] Processing DeleteTablet for tablet 698ac2c2e52949d591e7ad678b60c888 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-09 08:57:58 UTC) from {username='kudu', principal='kudu/127.20.49.126@KRBTEST.COM'} at 127.0.0.1:40412
I20251209 08:57:58.541474 22095 tablet_service.cc:1552] Processing DeleteTablet for tablet 698ac2c2e52949d591e7ad678b60c888 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-09 08:57:58 UTC) from {username='kudu', principal='kudu/127.20.49.126@KRBTEST.COM'} at 127.0.0.1:41788
I20251209 08:57:58.542028 22348 tablet_replica.cc:333] T 698ac2c2e52949d591e7ad678b60c888 P 6b11fb305a694cb0bb64203a3b2ba5c8: stopping tablet replica
I20251209 08:57:58.542328 22348 raft_consensus.cc:2243] T 698ac2c2e52949d591e7ad678b60c888 P 6b11fb305a694cb0bb64203a3b2ba5c8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251209 08:57:58.542378 22237 tablet_service.cc:1552] Processing DeleteTablet for tablet 698ac2c2e52949d591e7ad678b60c888 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-09 08:57:58 UTC) from {username='kudu', principal='kudu/127.20.49.126@KRBTEST.COM'} at 127.0.0.1:40448
I20251209 08:57:58.542675 22348 raft_consensus.cc:2272] T 698ac2c2e52949d591e7ad678b60c888 P 6b11fb305a694cb0bb64203a3b2ba5c8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251209 08:57:58.543023 22349 tablet_replica.cc:333] T 698ac2c2e52949d591e7ad678b60c888 P 4764a44e2a814f6db9d0185ea0bc2425: stopping tablet replica
I20251209 08:57:58.543046 21814 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42554:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251209 08:57:58.543289 22349 raft_consensus.cc:2243] T 698ac2c2e52949d591e7ad678b60c888 P 4764a44e2a814f6db9d0185ea0bc2425 [term 1 FOLLOWER]: Raft consensus shutting down.
W20251209 08:57:58.543509 21814 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251209 08:57:58.543550 22349 raft_consensus.cc:2272] T 698ac2c2e52949d591e7ad678b60c888 P 4764a44e2a814f6db9d0185ea0bc2425 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251209 08:57:58.544852 22348 ts_tablet_manager.cc:1916] T 698ac2c2e52949d591e7ad678b60c888 P 6b11fb305a694cb0bb64203a3b2ba5c8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251209 08:57:58.545552 22349 ts_tablet_manager.cc:1916] T 698ac2c2e52949d591e7ad678b60c888 P 4764a44e2a814f6db9d0185ea0bc2425: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251209 08:57:58.547667 22348 ts_tablet_manager.cc:1929] T 698ac2c2e52949d591e7ad678b60c888 P 6b11fb305a694cb0bb64203a3b2ba5c8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251209 08:57:58.547823 22348 log.cc:1199] T 698ac2c2e52949d591e7ad678b60c888 P 6b11fb305a694cb0bb64203a3b2ba5c8: Deleting WAL directory at /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765270668420570-20677-0/minicluster-data/ts-1/wal/wals/698ac2c2e52949d591e7ad678b60c888
I20251209 08:57:58.548126 22349 ts_tablet_manager.cc:1929] T 698ac2c2e52949d591e7ad678b60c888 P 4764a44e2a814f6db9d0185ea0bc2425: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251209 08:57:58.548220 22349 log.cc:1199] T 698ac2c2e52949d591e7ad678b60c888 P 4764a44e2a814f6db9d0185ea0bc2425: Deleting WAL directory at /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765270668420570-20677-0/minicluster-data/ts-0/wal/wals/698ac2c2e52949d591e7ad678b60c888
I20251209 08:57:58.548318 22348 ts_tablet_manager.cc:1950] T 698ac2c2e52949d591e7ad678b60c888 P 6b11fb305a694cb0bb64203a3b2ba5c8: Deleting consensus metadata
I20251209 08:57:58.548509 22349 ts_tablet_manager.cc:1950] T 698ac2c2e52949d591e7ad678b60c888 P 4764a44e2a814f6db9d0185ea0bc2425: Deleting consensus metadata
I20251209 08:57:58.549629 21798 catalog_manager.cc:4985] TS 6b11fb305a694cb0bb64203a3b2ba5c8 (127.20.49.66:39533): tablet 698ac2c2e52949d591e7ad678b60c888 (table test-table [id=5bb7dbac04844e2994052f16f33440e5]) successfully deleted
I20251209 08:57:58.549722 22350 tablet_replica.cc:333] T 698ac2c2e52949d591e7ad678b60c888 P 2da3b1d7e30440909571db652b3bf08e: stopping tablet replica
I20251209 08:57:58.549870 22095 tablet_service.cc:1505] Processing CreateTablet for tablet 83a821ea6f9946628a9ca70d835606c0 (DEFAULT_TABLE table=test-table [id=e6a5e9b717c540c1998ff3a64857c633]), partition=RANGE (key) PARTITION UNBOUNDED
I20251209 08:57:58.549990 22350 raft_consensus.cc:2243] T 698ac2c2e52949d591e7ad678b60c888 P 2da3b1d7e30440909571db652b3bf08e [term 1 LEADER]: Raft consensus shutting down.
I20251209 08:57:58.550107 22095 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 83a821ea6f9946628a9ca70d835606c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251209 08:57:58.550204 22237 tablet_service.cc:1505] Processing CreateTablet for tablet 83a821ea6f9946628a9ca70d835606c0 (DEFAULT_TABLE table=test-table [id=e6a5e9b717c540c1998ff3a64857c633]), partition=RANGE (key) PARTITION UNBOUNDED
I20251209 08:57:58.550460 22237 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 83a821ea6f9946628a9ca70d835606c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251209 08:57:58.550457 22350 raft_consensus.cc:2272] T 698ac2c2e52949d591e7ad678b60c888 P 2da3b1d7e30440909571db652b3bf08e [term 1 FOLLOWER]: Raft consensus is shut down!
I20251209 08:57:58.551760 21801 catalog_manager.cc:4985] TS 4764a44e2a814f6db9d0185ea0bc2425 (127.20.49.65:43253): tablet 698ac2c2e52949d591e7ad678b60c888 (table test-table [id=5bb7dbac04844e2994052f16f33440e5]) successfully deleted
I20251209 08:57:58.552697 22330 tablet_bootstrap.cc:492] T 83a821ea6f9946628a9ca70d835606c0 P 6b11fb305a694cb0bb64203a3b2ba5c8: Bootstrap starting.
I20251209 08:57:58.552697 22327 tablet_bootstrap.cc:492] T 83a821ea6f9946628a9ca70d835606c0 P 2da3b1d7e30440909571db652b3bf08e: Bootstrap starting.
I20251209 08:57:58.553758 22330 tablet_bootstrap.cc:654] T 83a821ea6f9946628a9ca70d835606c0 P 6b11fb305a694cb0bb64203a3b2ba5c8: Neither blocks nor log segments found. Creating new log.
I20251209 08:57:58.553758 22327 tablet_bootstrap.cc:654] T 83a821ea6f9946628a9ca70d835606c0 P 2da3b1d7e30440909571db652b3bf08e: Neither blocks nor log segments found. Creating new log.
I20251209 08:57:58.554088 21953 tablet_service.cc:1505] Processing CreateTablet for tablet 83a821ea6f9946628a9ca70d835606c0 (DEFAULT_TABLE table=test-table [id=e6a5e9b717c540c1998ff3a64857c633]), partition=RANGE (key) PARTITION UNBOUNDED
I20251209 08:57:58.554363 21953 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 83a821ea6f9946628a9ca70d835606c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251209 08:57:58.555197 22350 ts_tablet_manager.cc:1916] T 698ac2c2e52949d591e7ad678b60c888 P 2da3b1d7e30440909571db652b3bf08e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251209 08:57:58.556468 22326 tablet_bootstrap.cc:492] T 83a821ea6f9946628a9ca70d835606c0 P 4764a44e2a814f6db9d0185ea0bc2425: Bootstrap starting.
I20251209 08:57:58.557220 22350 ts_tablet_manager.cc:1929] T 698ac2c2e52949d591e7ad678b60c888 P 2da3b1d7e30440909571db652b3bf08e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251209 08:57:58.557312 22350 log.cc:1199] T 698ac2c2e52949d591e7ad678b60c888 P 2da3b1d7e30440909571db652b3bf08e: Deleting WAL directory at /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765270668420570-20677-0/minicluster-data/ts-2/wal/wals/698ac2c2e52949d591e7ad678b60c888
I20251209 08:57:58.557605 22326 tablet_bootstrap.cc:654] T 83a821ea6f9946628a9ca70d835606c0 P 4764a44e2a814f6db9d0185ea0bc2425: Neither blocks nor log segments found. Creating new log.
I20251209 08:57:58.557648 22350 ts_tablet_manager.cc:1950] T 698ac2c2e52949d591e7ad678b60c888 P 2da3b1d7e30440909571db652b3bf08e: Deleting consensus metadata
I20251209 08:57:58.558827 21798 catalog_manager.cc:4985] TS 2da3b1d7e30440909571db652b3bf08e (127.20.49.67:35127): tablet 698ac2c2e52949d591e7ad678b60c888 (table test-table [id=5bb7dbac04844e2994052f16f33440e5]) successfully deleted
I20251209 08:57:58.559458 22327 tablet_bootstrap.cc:492] T 83a821ea6f9946628a9ca70d835606c0 P 2da3b1d7e30440909571db652b3bf08e: No bootstrap required, opened a new log
I20251209 08:57:58.559545 22327 ts_tablet_manager.cc:1403] T 83a821ea6f9946628a9ca70d835606c0 P 2da3b1d7e30440909571db652b3bf08e: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20251209 08:57:58.559715 22326 tablet_bootstrap.cc:492] T 83a821ea6f9946628a9ca70d835606c0 P 4764a44e2a814f6db9d0185ea0bc2425: No bootstrap required, opened a new log
I20251209 08:57:58.559729 22330 tablet_bootstrap.cc:492] T 83a821ea6f9946628a9ca70d835606c0 P 6b11fb305a694cb0bb64203a3b2ba5c8: No bootstrap required, opened a new log
I20251209 08:57:58.559782 22326 ts_tablet_manager.cc:1403] T 83a821ea6f9946628a9ca70d835606c0 P 4764a44e2a814f6db9d0185ea0bc2425: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20251209 08:57:58.559808 22330 ts_tablet_manager.cc:1403] T 83a821ea6f9946628a9ca70d835606c0 P 6b11fb305a694cb0bb64203a3b2ba5c8: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20251209 08:57:58.560030 22327 raft_consensus.cc:359] T 83a821ea6f9946628a9ca70d835606c0 P 2da3b1d7e30440909571db652b3bf08e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4764a44e2a814f6db9d0185ea0bc2425" member_type: VOTER last_known_addr { host: "127.20.49.65" port: 43253 } } peers { permanent_uuid: "6b11fb305a694cb0bb64203a3b2ba5c8" member_type: VOTER last_known_addr { host: "127.20.49.66" port: 39533 } } peers { permanent_uuid: "2da3b1d7e30440909571db652b3bf08e" member_type: VOTER last_known_addr { host: "127.20.49.67" port: 35127 } }
I20251209 08:57:58.560134 22327 raft_consensus.cc:385] T 83a821ea6f9946628a9ca70d835606c0 P 2da3b1d7e30440909571db652b3bf08e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251209 08:57:58.560168 22327 raft_consensus.cc:740] T 83a821ea6f9946628a9ca70d835606c0 P 2da3b1d7e30440909571db652b3bf08e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2da3b1d7e30440909571db652b3bf08e, State: Initialized, Role: FOLLOWER
I20251209 08:57:58.560251 22326 raft_consensus.cc:359] T 83a821ea6f9946628a9ca70d835606c0 P 4764a44e2a814f6db9d0185ea0bc2425 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4764a44e2a814f6db9d0185ea0bc2425" member_type: VOTER last_known_addr { host: "127.20.49.65" port: 43253 } } peers { permanent_uuid: "6b11fb305a694cb0bb64203a3b2ba5c8" member_type: VOTER last_known_addr { host: "127.20.49.66" port: 39533 } } peers { permanent_uuid: "2da3b1d7e30440909571db652b3bf08e" member_type: VOTER last_known_addr { host: "127.20.49.67" port: 35127 } }
I20251209 08:57:58.560277 22330 raft_consensus.cc:359] T 83a821ea6f9946628a9ca70d835606c0 P 6b11fb305a694cb0bb64203a3b2ba5c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4764a44e2a814f6db9d0185ea0bc2425" member_type: VOTER last_known_addr { host: "127.20.49.65" port: 43253 } } peers { permanent_uuid: "6b11fb305a694cb0bb64203a3b2ba5c8" member_type: VOTER last_known_addr { host: "127.20.49.66" port: 39533 } } peers { permanent_uuid: "2da3b1d7e30440909571db652b3bf08e" member_type: VOTER last_known_addr { host: "127.20.49.67" port: 35127 } }
I20251209 08:57:58.560344 22326 raft_consensus.cc:385] T 83a821ea6f9946628a9ca70d835606c0 P 4764a44e2a814f6db9d0185ea0bc2425 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251209 08:57:58.560302 22327 consensus_queue.cc:260] T 83a821ea6f9946628a9ca70d835606c0 P 2da3b1d7e30440909571db652b3bf08e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4764a44e2a814f6db9d0185ea0bc2425" member_type: VOTER last_known_addr { host: "127.20.49.65" port: 43253 } } peers { permanent_uuid: "6b11fb305a694cb0bb64203a3b2ba5c8" member_type: VOTER last_known_addr { host: "127.20.49.66" port: 39533 } } peers { permanent_uuid: "2da3b1d7e30440909571db652b3bf08e" member_type: VOTER last_known_addr { host: "127.20.49.67" port: 35127 } }
I20251209 08:57:58.560374 22330 raft_consensus.cc:385] T 83a821ea6f9946628a9ca70d835606c0 P 6b11fb305a694cb0bb64203a3b2ba5c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251209 08:57:58.560377 22326 raft_consensus.cc:740] T 83a821ea6f9946628a9ca70d835606c0 P 4764a44e2a814f6db9d0185ea0bc2425 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4764a44e2a814f6db9d0185ea0bc2425, State: Initialized, Role: FOLLOWER
I20251209 08:57:58.560406 22330 raft_consensus.cc:740] T 83a821ea6f9946628a9ca70d835606c0 P 6b11fb305a694cb0bb64203a3b2ba5c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b11fb305a694cb0bb64203a3b2ba5c8, State: Initialized, Role: FOLLOWER
I20251209 08:57:58.560492 22326 consensus_queue.cc:260] T 83a821ea6f9946628a9ca70d835606c0 P 4764a44e2a814f6db9d0185ea0bc2425 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4764a44e2a814f6db9d0185ea0bc2425" member_type: VOTER last_known_addr { host: "127.20.49.65" port: 43253 } } peers { permanent_uuid: "6b11fb305a694cb0bb64203a3b2ba5c8" member_type: VOTER last_known_addr { host: "127.20.49.66" port: 39533 } } peers { permanent_uuid: "2da3b1d7e30440909571db652b3bf08e" member_type: VOTER last_known_addr { host: "127.20.49.67" port: 35127 } }
I20251209 08:57:58.560530 22330 consensus_queue.cc:260] T 83a821ea6f9946628a9ca70d835606c0 P 6b11fb305a694cb0bb64203a3b2ba5c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4764a44e2a814f6db9d0185ea0bc2425" member_type: VOTER last_known_addr { host: "127.20.49.65" port: 43253 } } peers { permanent_uuid: "6b11fb305a694cb0bb64203a3b2ba5c8" member_type: VOTER last_known_addr { host: "127.20.49.66" port: 39533 } } peers { permanent_uuid: "2da3b1d7e30440909571db652b3bf08e" member_type: VOTER last_known_addr { host: "127.20.49.67" port: 35127 } }
I20251209 08:57:58.560567 22327 ts_tablet_manager.cc:1434] T 83a821ea6f9946628a9ca70d835606c0 P 2da3b1d7e30440909571db652b3bf08e: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20251209 08:57:58.560784 22330 ts_tablet_manager.cc:1434] T 83a821ea6f9946628a9ca70d835606c0 P 6b11fb305a694cb0bb64203a3b2ba5c8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20251209 08:57:58.561151 22326 ts_tablet_manager.cc:1434] T 83a821ea6f9946628a9ca70d835606c0 P 4764a44e2a814f6db9d0185ea0bc2425: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20251209 08:57:58.656523 22334 raft_consensus.cc:493] T 83a821ea6f9946628a9ca70d835606c0 P 6b11fb305a694cb0bb64203a3b2ba5c8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251209 08:57:58.656679 22334 raft_consensus.cc:515] T 83a821ea6f9946628a9ca70d835606c0 P 6b11fb305a694cb0bb64203a3b2ba5c8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4764a44e2a814f6db9d0185ea0bc2425" member_type: VOTER last_known_addr { host: "127.20.49.65" port: 43253 } } peers { permanent_uuid: "6b11fb305a694cb0bb64203a3b2ba5c8" member_type: VOTER last_known_addr { host: "127.20.49.66" port: 39533 } } peers { permanent_uuid: "2da3b1d7e30440909571db652b3bf08e" member_type: VOTER last_known_addr { host: "127.20.49.67" port: 35127 } }
I20251209 08:57:58.658010 22334 leader_election.cc:290] T 83a821ea6f9946628a9ca70d835606c0 P 6b11fb305a694cb0bb64203a3b2ba5c8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4764a44e2a814f6db9d0185ea0bc2425 (127.20.49.65:43253), 2da3b1d7e30440909571db652b3bf08e (127.20.49.67:35127)
I20251209 08:57:58.661620 22355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:57:58.658272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.49.67:35127 (local address 127.20.49.66:60915)
1209 08:57:58.658572 (+ 300us) negotiation.cc:107] Waiting for socket to connect
1209 08:57:58.658585 (+ 13us) client_negotiation.cc:174] Beginning negotiation
1209 08:57:58.658658 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1209 08:57:58.659022 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1209 08:57:58.659027 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
1209 08:57:58.659054 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1209 08:57:58.659377 (+ 323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:57:58.659385 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:57:58.660291 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1209 08:57:58.660295 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1209 08:57:58.660946 (+ 651us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:57:58.660954 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:57:58.661075 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:57:58.661098 (+ 23us) client_negotiation.cc:769] Sending connection context
1209 08:57:58.661150 (+ 52us) client_negotiation.cc:240] Negotiation successful
1209 08:57:58.661216 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":66,"threads_started":1}
I20251209 08:57:58.661895 22324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:57:58.658614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.49.66:60915 (local address 127.20.49.67:35127)
1209 08:57:58.658772 (+ 158us) server_negotiation.cc:207] Beginning negotiation
1209 08:57:58.658776 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1209 08:57:58.658792 (+ 16us) server_negotiation.cc:408] Connection header received
1209 08:57:58.658846 (+ 54us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1209 08:57:58.658849 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1209 08:57:58.658897 (+ 48us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1209 08:57:58.658979 (+ 82us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1209 08:57:58.659531 (+ 552us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:57:58.660177 (+ 646us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1209 08:57:58.661088 (+ 911us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:57:58.661589 (+ 501us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:57:58.661631 (+ 42us) server_negotiation.cc:1036] Waiting for connection context
1209 08:57:58.661714 (+ 83us) server_negotiation.cc:300] Negotiation successful
1209 08:57:58.661780 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20251209 08:57:58.662676 22257 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "83a821ea6f9946628a9ca70d835606c0" candidate_uuid: "6b11fb305a694cb0bb64203a3b2ba5c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2da3b1d7e30440909571db652b3bf08e" is_pre_election: true
I20251209 08:57:58.662926 22257 raft_consensus.cc:2468] T 83a821ea6f9946628a9ca70d835606c0 P 2da3b1d7e30440909571db652b3bf08e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b11fb305a694cb0bb64203a3b2ba5c8 in term 0.
I20251209 08:57:58.662968 22354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:57:58.658159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.49.65:43253 (local address 127.20.49.66:39471)
1209 08:57:58.658434 (+ 275us) negotiation.cc:107] Waiting for socket to connect
1209 08:57:58.658451 (+ 17us) client_negotiation.cc:174] Beginning negotiation
1209 08:57:58.658556 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1209 08:57:58.659125 (+ 569us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1209 08:57:58.659128 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1209 08:57:58.659147 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1209 08:57:58.659773 (+ 626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:57:58.659783 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:57:58.660803 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1209 08:57:58.660807 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1209 08:57:58.661963 (+ 1156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:57:58.661978 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:57:58.662664 (+ 686us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:57:58.662682 (+ 18us) client_negotiation.cc:769] Sending connection context
1209 08:57:58.662766 (+ 84us) client_negotiation.cc:240] Negotiation successful
1209 08:57:58.662829 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":96,"threads_started":1}
I20251209 08:57:58.663067 22325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:57:58.658766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.49.66:39471 (local address 127.20.49.65:43253)
1209 08:57:58.658903 (+ 137us) server_negotiation.cc:207] Beginning negotiation
1209 08:57:58.658908 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1209 08:57:58.658923 (+ 15us) server_negotiation.cc:408] Connection header received
1209 08:57:58.658977 (+ 54us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1209 08:57:58.658980 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1209 08:57:58.659028 (+ 48us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1209 08:57:58.659102 (+ 74us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1209 08:57:58.659935 (+ 833us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:57:58.660676 (+ 741us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1209 08:57:58.662122 (+ 1446us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:57:58.662558 (+ 436us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:57:58.662597 (+ 39us) server_negotiation.cc:1036] Waiting for connection context
1209 08:57:58.662837 (+ 240us) server_negotiation.cc:300] Negotiation successful
1209 08:57:58.662919 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20251209 08:57:58.663503 22046 leader_election.cc:304] T 83a821ea6f9946628a9ca70d835606c0 P 6b11fb305a694cb0bb64203a3b2ba5c8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2da3b1d7e30440909571db652b3bf08e, 6b11fb305a694cb0bb64203a3b2ba5c8; no voters:
I20251209 08:57:58.663524 21973 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "83a821ea6f9946628a9ca70d835606c0" candidate_uuid: "6b11fb305a694cb0bb64203a3b2ba5c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4764a44e2a814f6db9d0185ea0bc2425" is_pre_election: true
I20251209 08:57:58.663653 21973 raft_consensus.cc:2468] T 83a821ea6f9946628a9ca70d835606c0 P 4764a44e2a814f6db9d0185ea0bc2425 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b11fb305a694cb0bb64203a3b2ba5c8 in term 0.
I20251209 08:57:58.663763 22334 raft_consensus.cc:2804] T 83a821ea6f9946628a9ca70d835606c0 P 6b11fb305a694cb0bb64203a3b2ba5c8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251209 08:57:58.663827 22334 raft_consensus.cc:493] T 83a821ea6f9946628a9ca70d835606c0 P 6b11fb305a694cb0bb64203a3b2ba5c8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251209 08:57:58.663882 22334 raft_consensus.cc:3060] T 83a821ea6f9946628a9ca70d835606c0 P 6b11fb305a694cb0bb64203a3b2ba5c8 [term 0 FOLLOWER]: Advancing to term 1
I20251209 08:57:58.664812 22334 raft_consensus.cc:515] T 83a821ea6f9946628a9ca70d835606c0 P 6b11fb305a694cb0bb64203a3b2ba5c8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4764a44e2a814f6db9d0185ea0bc2425" member_type: VOTER last_known_addr { host: "127.20.49.65" port: 43253 } } peers { permanent_uuid: "6b11fb305a694cb0bb64203a3b2ba5c8" member_type: VOTER last_known_addr { host: "127.20.49.66" port: 39533 } } peers { permanent_uuid: "2da3b1d7e30440909571db652b3bf08e" member_type: VOTER last_known_addr { host: "127.20.49.67" port: 35127 } }
I20251209 08:57:58.665163 22334 leader_election.cc:290] T 83a821ea6f9946628a9ca70d835606c0 P 6b11fb305a694cb0bb64203a3b2ba5c8 [CANDIDATE]: Term 1 election: Requested vote from peers 4764a44e2a814f6db9d0185ea0bc2425 (127.20.49.65:43253), 2da3b1d7e30440909571db652b3bf08e (127.20.49.67:35127)
I20251209 08:57:58.665570 22257 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "83a821ea6f9946628a9ca70d835606c0" candidate_uuid: "6b11fb305a694cb0bb64203a3b2ba5c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2da3b1d7e30440909571db652b3bf08e"
I20251209 08:57:58.665658 21973 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "83a821ea6f9946628a9ca70d835606c0" candidate_uuid: "6b11fb305a694cb0bb64203a3b2ba5c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4764a44e2a814f6db9d0185ea0bc2425"
I20251209 08:57:58.665817 21973 raft_consensus.cc:3060] T 83a821ea6f9946628a9ca70d835606c0 P 4764a44e2a814f6db9d0185ea0bc2425 [term 0 FOLLOWER]: Advancing to term 1
I20251209 08:57:58.666992 21973 raft_consensus.cc:2468] T 83a821ea6f9946628a9ca70d835606c0 P 4764a44e2a814f6db9d0185ea0bc2425 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b11fb305a694cb0bb64203a3b2ba5c8 in term 1.
I20251209 08:57:58.667339 22257 raft_consensus.cc:3060] T 83a821ea6f9946628a9ca70d835606c0 P 2da3b1d7e30440909571db652b3bf08e [term 0 FOLLOWER]: Advancing to term 1
I20251209 08:57:58.667328 22049 leader_election.cc:304] T 83a821ea6f9946628a9ca70d835606c0 P 6b11fb305a694cb0bb64203a3b2ba5c8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4764a44e2a814f6db9d0185ea0bc2425, 6b11fb305a694cb0bb64203a3b2ba5c8; no voters:
I20251209 08:57:58.667541 22334 raft_consensus.cc:2804] T 83a821ea6f9946628a9ca70d835606c0 P 6b11fb305a694cb0bb64203a3b2ba5c8 [term 1 FOLLOWER]: Leader election won for term 1
I20251209 08:57:58.667748 22334 raft_consensus.cc:697] T 83a821ea6f9946628a9ca70d835606c0 P 6b11fb305a694cb0bb64203a3b2ba5c8 [term 1 LEADER]: Becoming Leader. State: Replica: 6b11fb305a694cb0bb64203a3b2ba5c8, State: Running, Role: LEADER
I20251209 08:57:58.668001 22334 consensus_queue.cc:237] T 83a821ea6f9946628a9ca70d835606c0 P 6b11fb305a694cb0bb64203a3b2ba5c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4764a44e2a814f6db9d0185ea0bc2425" member_type: VOTER last_known_addr { host: "127.20.49.65" port: 43253 } } peers { permanent_uuid: "6b11fb305a694cb0bb64203a3b2ba5c8" member_type: VOTER last_known_addr { host: "127.20.49.66" port: 39533 } } peers { permanent_uuid: "2da3b1d7e30440909571db652b3bf08e" member_type: VOTER last_known_addr { host: "127.20.49.67" port: 35127 } }
I20251209 08:57:58.668566 22257 raft_consensus.cc:2468] T 83a821ea6f9946628a9ca70d835606c0 P 2da3b1d7e30440909571db652b3bf08e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b11fb305a694cb0bb64203a3b2ba5c8 in term 1.
I20251209 08:57:58.670730 21813 catalog_manager.cc:5654] T 83a821ea6f9946628a9ca70d835606c0 P 6b11fb305a694cb0bb64203a3b2ba5c8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6b11fb305a694cb0bb64203a3b2ba5c8 (127.20.49.66). New cstate: current_term: 1 leader_uuid: "6b11fb305a694cb0bb64203a3b2ba5c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4764a44e2a814f6db9d0185ea0bc2425" member_type: VOTER last_known_addr { host: "127.20.49.65" port: 43253 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6b11fb305a694cb0bb64203a3b2ba5c8" member_type: VOTER last_known_addr { host: "127.20.49.66" port: 39533 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2da3b1d7e30440909571db652b3bf08e" member_type: VOTER last_known_addr { host: "127.20.49.67" port: 35127 } health_report { overall_health: UNKNOWN } } }
Dec 09 08:57:58 dist-test-slave-8w6c krb5kdc[21766](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765270677, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.49.126@KRBTEST.COM for kudu/127.20.49.126@KRBTEST.COM
I20251209 08:57:58.719969 22363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:57:58.711649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.49.126:35129 (local address 127.0.0.1:42556)
1209 08:57:58.711991 (+ 342us) negotiation.cc:107] Waiting for socket to connect
1209 08:57:58.712010 (+ 19us) client_negotiation.cc:174] Beginning negotiation
1209 08:57:58.712157 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1209 08:57:58.712435 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1209 08:57:58.712438 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1209 08:57:58.712874 (+ 436us) client_negotiation.cc:189] Negotiated authn=SASL
1209 08:57:58.713074 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:57:58.713082 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:57:58.713802 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1209 08:57:58.713809 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1209 08:57:58.714188 (+ 379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:57:58.714196 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:57:58.714315 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:57:58.715004 (+ 689us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1209 08:57:58.715019 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
1209 08:57:58.716815 (+ 1796us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1209 08:57:58.718560 (+ 1745us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1209 08:57:58.718570 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1209 08:57:58.718575 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
1209 08:57:58.718789 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1209 08:57:58.719004 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1209 08:57:58.719015 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1209 08:57:58.719017 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1209 08:57:58.719058 (+ 41us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1209 08:57:58.719389 (+ 331us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1209 08:57:58.719399 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1209 08:57:58.719495 (+ 96us) client_negotiation.cc:769] Sending connection context
1209 08:57:58.719601 (+ 106us) client_negotiation.cc:240] Negotiation successful
1209 08:57:58.719769 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":115,"threads_started":1}
I20251209 08:57:58.719969 21863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:57:58.711689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42556 (local address 127.20.49.126:35129)
1209 08:57:58.711825 (+ 136us) server_negotiation.cc:207] Beginning negotiation
1209 08:57:58.711830 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1209 08:57:58.712153 (+ 323us) server_negotiation.cc:408] Connection header received
1209 08:57:58.712246 (+ 93us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1209 08:57:58.712249 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1209 08:57:58.712297 (+ 48us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1209 08:57:58.712395 (+ 98us) server_negotiation.cc:227] Negotiated authn=SASL
1209 08:57:58.713200 (+ 805us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:57:58.713670 (+ 470us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1209 08:57:58.714298 (+ 628us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:57:58.714461 (+ 163us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:57:58.716939 (+ 2478us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1209 08:57:58.716956 (+ 17us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1209 08:57:58.716959 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1209 08:57:58.716981 (+ 22us) server_negotiation.cc:923] calling sasl_server_start()
1209 08:57:58.718450 (+ 1469us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1209 08:57:58.718885 (+ 435us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1209 08:57:58.718888 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1209 08:57:58.718891 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
1209 08:57:58.718927 (+ 36us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1209 08:57:58.719134 (+ 207us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1209 08:57:58.719137 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1209 08:57:58.719139 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1209 08:57:58.719289 (+ 150us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1209 08:57:58.719376 (+ 87us) server_negotiation.cc:1036] Waiting for connection context
1209 08:57:58.719655 (+ 279us) server_negotiation.cc:300] Negotiation successful
1209 08:57:58.719832 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":72,"server-negotiator.queue_time_us":49}
I20251209 08:57:58.723992 21813 catalog_manager.cc:2257] Servicing CreateTable request from {username='kudu', principal='kudu/127.20.49.126@KRBTEST.COM'} at 127.0.0.1:42556:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20251209 08:57:58.724727 21813 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251209 08:57:58.729959 21953 tablet_service.cc:1505] Processing CreateTablet for tablet 18aba1ab36b74aa09b512665fa2327e2 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=f5515f8be5ae4b4c90e47e3707748292]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20251209 08:57:58.730059 22095 tablet_service.cc:1505] Processing CreateTablet for tablet 18aba1ab36b74aa09b512665fa2327e2 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=f5515f8be5ae4b4c90e47e3707748292]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20251209 08:57:58.730374 21953 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 18aba1ab36b74aa09b512665fa2327e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251209 08:57:58.730414 22095 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 18aba1ab36b74aa09b512665fa2327e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251209 08:57:58.731040 22237 tablet_service.cc:1505] Processing CreateTablet for tablet 18aba1ab36b74aa09b512665fa2327e2 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=f5515f8be5ae4b4c90e47e3707748292]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20251209 08:57:58.731371 22237 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 18aba1ab36b74aa09b512665fa2327e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251209 08:57:58.733072 22330 tablet_bootstrap.cc:492] T 18aba1ab36b74aa09b512665fa2327e2 P 6b11fb305a694cb0bb64203a3b2ba5c8: Bootstrap starting.
I20251209 08:57:58.733963 22327 tablet_bootstrap.cc:492] T 18aba1ab36b74aa09b512665fa2327e2 P 2da3b1d7e30440909571db652b3bf08e: Bootstrap starting.
I20251209 08:57:58.734238 22330 tablet_bootstrap.cc:654] T 18aba1ab36b74aa09b512665fa2327e2 P 6b11fb305a694cb0bb64203a3b2ba5c8: Neither blocks nor log segments found. Creating new log.
I20251209 08:57:58.734266 22326 tablet_bootstrap.cc:492] T 18aba1ab36b74aa09b512665fa2327e2 P 4764a44e2a814f6db9d0185ea0bc2425: Bootstrap starting.
I20251209 08:57:58.734818 22327 tablet_bootstrap.cc:654] T 18aba1ab36b74aa09b512665fa2327e2 P 2da3b1d7e30440909571db652b3bf08e: Neither blocks nor log segments found. Creating new log.
I20251209 08:57:58.735267 22326 tablet_bootstrap.cc:654] T 18aba1ab36b74aa09b512665fa2327e2 P 4764a44e2a814f6db9d0185ea0bc2425: Neither blocks nor log segments found. Creating new log.
I20251209 08:57:58.735703 22330 tablet_bootstrap.cc:492] T 18aba1ab36b74aa09b512665fa2327e2 P 6b11fb305a694cb0bb64203a3b2ba5c8: No bootstrap required, opened a new log
I20251209 08:57:58.735810 22330 ts_tablet_manager.cc:1403] T 18aba1ab36b74aa09b512665fa2327e2 P 6b11fb305a694cb0bb64203a3b2ba5c8: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20251209 08:57:58.736352 22327 tablet_bootstrap.cc:492] T 18aba1ab36b74aa09b512665fa2327e2 P 2da3b1d7e30440909571db652b3bf08e: No bootstrap required, opened a new log
I20251209 08:57:58.736436 22327 ts_tablet_manager.cc:1403] T 18aba1ab36b74aa09b512665fa2327e2 P 2da3b1d7e30440909571db652b3bf08e: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20251209 08:57:58.736361 22330 raft_consensus.cc:359] T 18aba1ab36b74aa09b512665fa2327e2 P 6b11fb305a694cb0bb64203a3b2ba5c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4764a44e2a814f6db9d0185ea0bc2425" member_type: VOTER last_known_addr { host: "127.20.49.65" port: 43253 } } peers { permanent_uuid: "6b11fb305a694cb0bb64203a3b2ba5c8" member_type: VOTER last_known_addr { host: "127.20.49.66" port: 39533 } } peers { permanent_uuid: "2da3b1d7e30440909571db652b3bf08e" member_type: VOTER last_known_addr { host: "127.20.49.67" port: 35127 } }
I20251209 08:57:58.736510 22330 raft_consensus.cc:385] T 18aba1ab36b74aa09b512665fa2327e2 P 6b11fb305a694cb0bb64203a3b2ba5c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251209 08:57:58.736562 22330 raft_consensus.cc:740] T 18aba1ab36b74aa09b512665fa2327e2 P 6b11fb305a694cb0bb64203a3b2ba5c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b11fb305a694cb0bb64203a3b2ba5c8, State: Initialized, Role: FOLLOWER
I20251209 08:57:58.736708 22330 consensus_queue.cc:260] T 18aba1ab36b74aa09b512665fa2327e2 P 6b11fb305a694cb0bb64203a3b2ba5c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4764a44e2a814f6db9d0185ea0bc2425" member_type: VOTER last_known_addr { host: "127.20.49.65" port: 43253 } } peers { permanent_uuid: "6b11fb305a694cb0bb64203a3b2ba5c8" member_type: VOTER last_known_addr { host: "127.20.49.66" port: 39533 } } peers { permanent_uuid: "2da3b1d7e30440909571db652b3bf08e" member_type: VOTER last_known_addr { host: "127.20.49.67" port: 35127 } }
I20251209 08:57:58.736891 22326 tablet_bootstrap.cc:492] T 18aba1ab36b74aa09b512665fa2327e2 P 4764a44e2a814f6db9d0185ea0bc2425: No bootstrap required, opened a new log
I20251209 08:57:58.736860 22327 raft_consensus.cc:359] T 18aba1ab36b74aa09b512665fa2327e2 P 2da3b1d7e30440909571db652b3bf08e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4764a44e2a814f6db9d0185ea0bc2425" member_type: VOTER last_known_addr { host: "127.20.49.65" port: 43253 } } peers { permanent_uuid: "6b11fb305a694cb0bb64203a3b2ba5c8" member_type: VOTER last_known_addr { host: "127.20.49.66" port: 39533 } } peers { permanent_uuid: "2da3b1d7e30440909571db652b3bf08e" member_type: VOTER last_known_addr { host: "127.20.49.67" port: 35127 } }
I20251209 08:57:58.736958 22327 raft_consensus.cc:385] T 18aba1ab36b74aa09b512665fa2327e2 P 2da3b1d7e30440909571db652b3bf08e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251209 08:57:58.736976 22326 ts_tablet_manager.cc:1403] T 18aba1ab36b74aa09b512665fa2327e2 P 4764a44e2a814f6db9d0185ea0bc2425: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20251209 08:57:58.736989 22327 raft_consensus.cc:740] T 18aba1ab36b74aa09b512665fa2327e2 P 2da3b1d7e30440909571db652b3bf08e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2da3b1d7e30440909571db652b3bf08e, State: Initialized, Role: FOLLOWER
I20251209 08:57:58.736967 22356 tablet_replica.cc:442] T 18aba1ab36b74aa09b512665fa2327e2 P 6b11fb305a694cb0bb64203a3b2ba5c8: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4764a44e2a814f6db9d0185ea0bc2425" member_type: VOTER last_known_addr { host: "127.20.49.65" port: 43253 } } peers { permanent_uuid: "6b11fb305a694cb0bb64203a3b2ba5c8" member_type: VOTER last_known_addr { host: "127.20.49.66" port: 39533 } } peers { permanent_uuid: "2da3b1d7e30440909571db652b3bf08e" member_type: VOTER last_known_addr { host: "127.20.49.67" port: 35127 } } }
I20251209 08:57:58.737103 22356 tablet_replica.cc:445] T 18aba1ab36b74aa09b512665fa2327e2 P 6b11fb305a694cb0bb64203a3b2ba5c8: This TxnStatusTablet replica's current role is: FOLLOWER
I20251209 08:57:58.737128 22327 consensus_queue.cc:260] T 18aba1ab36b74aa09b512665fa2327e2 P 2da3b1d7e30440909571db652b3bf08e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4764a44e2a814f6db9d0185ea0bc2425" member_type: VOTER last_known_addr { host: "127.20.49.65" port: 43253 } } peers { permanent_uuid: "6b11fb305a694cb0bb64203a3b2ba5c8" member_type: VOTER last_known_addr { host: "127.20.49.66" port: 39533 } } peers { permanent_uuid: "2da3b1d7e30440909571db652b3bf08e" member_type: VOTER last_known_addr { host: "127.20.49.67" port: 35127 } }
I20251209 08:57:58.737352 22336 tablet_replica.cc:442] T 18aba1ab36b74aa09b512665fa2327e2 P 2da3b1d7e30440909571db652b3bf08e: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4764a44e2a814f6db9d0185ea0bc2425" member_type: VOTER last_known_addr { host: "127.20.49.65" port: 43253 } } peers { permanent_uuid: "6b11fb305a694cb0bb64203a3b2ba5c8" member_type: VOTER last_known_addr { host: "127.20.49.66" port: 39533 } } peers { permanent_uuid: "2da3b1d7e30440909571db652b3bf08e" member_type: VOTER last_known_addr { host: "127.20.49.67" port: 35127 } } }
I20251209 08:57:58.737418 22327 ts_tablet_manager.cc:1434] T 18aba1ab36b74aa09b512665fa2327e2 P 2da3b1d7e30440909571db652b3bf08e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20251209 08:57:58.737457 22336 tablet_replica.cc:445] T 18aba1ab36b74aa09b512665fa2327e2 P 2da3b1d7e30440909571db652b3bf08e: This TxnStatusTablet replica's current role is: FOLLOWER
I20251209 08:57:58.737499 22326 raft_consensus.cc:359] T 18aba1ab36b74aa09b512665fa2327e2 P 4764a44e2a814f6db9d0185ea0bc2425 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4764a44e2a814f6db9d0185ea0bc2425" member_type: VOTER last_known_addr { host: "127.20.49.65" port: 43253 } } peers { permanent_uuid: "6b11fb305a694cb0bb64203a3b2ba5c8" member_type: VOTER last_known_addr { host: "127.20.49.66" port: 39533 } } peers { permanent_uuid: "2da3b1d7e30440909571db652b3bf08e" member_type: VOTER last_known_addr { host: "127.20.49.67" port: 35127 } }
I20251209 08:57:58.737591 22326 raft_consensus.cc:385] T 18aba1ab36b74aa09b512665fa2327e2 P 4764a44e2a814f6db9d0185ea0bc2425 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251209 08:57:58.737622 22326 raft_consensus.cc:740] T 18aba1ab36b74aa09b512665fa2327e2 P 4764a44e2a814f6db9d0185ea0bc2425 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4764a44e2a814f6db9d0185ea0bc2425, State: Initialized, Role: FOLLOWER
I20251209 08:57:58.737769 22326 consensus_queue.cc:260] T 18aba1ab36b74aa09b512665fa2327e2 P 4764a44e2a814f6db9d0185ea0bc2425 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4764a44e2a814f6db9d0185ea0bc2425" member_type: VOTER last_known_addr { host: "127.20.49.65" port: 43253 } } peers { permanent_uuid: "6b11fb305a694cb0bb64203a3b2ba5c8" member_type: VOTER last_known_addr { host: "127.20.49.66" port: 39533 } } peers { permanent_uuid: "2da3b1d7e30440909571db652b3bf08e" member_type: VOTER last_known_addr { host: "127.20.49.67" port: 35127 } }
I20251209 08:57:58.738058 22326 ts_tablet_manager.cc:1434] T 18aba1ab36b74aa09b512665fa2327e2 P 4764a44e2a814f6db9d0185ea0bc2425: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20251209 08:57:58.738058 22330 ts_tablet_manager.cc:1434] T 18aba1ab36b74aa09b512665fa2327e2 P 6b11fb305a694cb0bb64203a3b2ba5c8: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20251209 08:57:58.738276 22331 tablet_replica.cc:442] T 18aba1ab36b74aa09b512665fa2327e2 P 4764a44e2a814f6db9d0185ea0bc2425: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4764a44e2a814f6db9d0185ea0bc2425" member_type: VOTER last_known_addr { host: "127.20.49.65" port: 43253 } } peers { permanent_uuid: "6b11fb305a694cb0bb64203a3b2ba5c8" member_type: VOTER last_known_addr { host: "127.20.49.66" port: 39533 } } peers { permanent_uuid: "2da3b1d7e30440909571db652b3bf08e" member_type: VOTER last_known_addr { host: "127.20.49.67" port: 35127 } } }
I20251209 08:57:58.738359 22331 tablet_replica.cc:445] T 18aba1ab36b74aa09b512665fa2327e2 P 4764a44e2a814f6db9d0185ea0bc2425: This TxnStatusTablet replica's current role is: FOLLOWER
W20251209 08:57:58.781378 22162 tablet.cc:2378] T 18aba1ab36b74aa09b512665fa2327e2 P 6b11fb305a694cb0bb64203a3b2ba5c8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20251209 08:57:58.790185 22331 raft_consensus.cc:493] T 18aba1ab36b74aa09b512665fa2327e2 P 4764a44e2a814f6db9d0185ea0bc2425 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251209 08:57:58.790318 22331 raft_consensus.cc:515] T 18aba1ab36b74aa09b512665fa2327e2 P 4764a44e2a814f6db9d0185ea0bc2425 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4764a44e2a814f6db9d0185ea0bc2425" member_type: VOTER last_known_addr { host: "127.20.49.65" port: 43253 } } peers { permanent_uuid: "6b11fb305a694cb0bb64203a3b2ba5c8" member_type: VOTER last_known_addr { host: "127.20.49.66" port: 39533 } } peers { permanent_uuid: "2da3b1d7e30440909571db652b3bf08e" member_type: VOTER last_known_addr { host: "127.20.49.67" port: 35127 } }
I20251209 08:57:58.791404 22331 leader_election.cc:290] T 18aba1ab36b74aa09b512665fa2327e2 P 4764a44e2a814f6db9d0185ea0bc2425 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6b11fb305a694cb0bb64203a3b2ba5c8 (127.20.49.66:39533), 2da3b1d7e30440909571db652b3bf08e (127.20.49.67:35127)
I20251209 08:57:58.794749 22367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:57:58.791631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.49.66:39533 (local address 127.20.49.65:57819)
1209 08:57:58.792002 (+ 371us) negotiation.cc:107] Waiting for socket to connect
1209 08:57:58.792017 (+ 15us) client_negotiation.cc:174] Beginning negotiation
1209 08:57:58.792126 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1209 08:57:58.792442 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1209 08:57:58.792505 (+ 63us) client_negotiation.cc:394] Received NEGOTIATE response from server
1209 08:57:58.792536 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1209 08:57:58.792892 (+ 356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:57:58.792904 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:57:58.793671 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1209 08:57:58.793675 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1209 08:57:58.794389 (+ 714us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:57:58.794398 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:57:58.794495 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:57:58.794512 (+ 17us) client_negotiation.cc:769] Sending connection context
1209 08:57:58.794549 (+ 37us) client_negotiation.cc:240] Negotiation successful
1209 08:57:58.794591 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"spinlock_wait_cycles":3968,"thread_start_us":130,"threads_started":1}
I20251209 08:57:58.794935 22368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:57:58.792061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.49.67:35127 (local address 127.20.49.65:41773)
1209 08:57:58.792476 (+ 415us) negotiation.cc:107] Waiting for socket to connect
1209 08:57:58.792490 (+ 14us) client_negotiation.cc:174] Beginning negotiation
1209 08:57:58.792606 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1209 08:57:58.792854 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1209 08:57:58.792856 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1209 08:57:58.792867 (+ 11us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1209 08:57:58.793133 (+ 266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:57:58.793142 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:57:58.793921 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1209 08:57:58.793925 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1209 08:57:58.794547 (+ 622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:57:58.794556 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:57:58.794668 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:57:58.794686 (+ 18us) client_negotiation.cc:769] Sending connection context
1209 08:57:58.794735 (+ 49us) client_negotiation.cc:240] Negotiation successful
1209 08:57:58.794788 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":282,"spinlock_wait_cycles":19072,"thread_start_us":104,"threads_started":1}
I20251209 08:57:58.795198 22323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:57:58.791738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.49.65:57819 (local address 127.20.49.66:39533)
1209 08:57:58.791902 (+ 164us) server_negotiation.cc:207] Beginning negotiation
1209 08:57:58.791906 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1209 08:57:58.792088 (+ 182us) server_negotiation.cc:408] Connection header received
1209 08:57:58.792251 (+ 163us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1209 08:57:58.792254 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1209 08:57:58.792317 (+ 63us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1209 08:57:58.792418 (+ 101us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1209 08:57:58.793016 (+ 598us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:57:58.793569 (+ 553us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1209 08:57:58.794512 (+ 943us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:57:58.794962 (+ 450us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:57:58.795002 (+ 40us) server_negotiation.cc:1036] Waiting for connection context
1209 08:57:58.795053 (+ 51us) server_negotiation.cc:300] Negotiation successful
1209 08:57:58.795107 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20251209 08:57:58.795434 22324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:57:58.792220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.49.65:41773 (local address 127.20.49.67:35127)
1209 08:57:58.792377 (+ 157us) server_negotiation.cc:207] Beginning negotiation
1209 08:57:58.792382 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1209 08:57:58.792549 (+ 167us) server_negotiation.cc:408] Connection header received
1209 08:57:58.792709 (+ 160us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1209 08:57:58.792715 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
1209 08:57:58.792766 (+ 51us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1209 08:57:58.792845 (+ 79us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1209 08:57:58.793255 (+ 410us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:57:58.793801 (+ 546us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1209 08:57:58.794678 (+ 877us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:57:58.795168 (+ 490us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:57:58.795199 (+ 31us) server_negotiation.cc:1036] Waiting for connection context
1209 08:57:58.795247 (+ 48us) server_negotiation.cc:300] Negotiation successful
1209 08:57:58.795300 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20251209 08:57:58.795755 22114 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "18aba1ab36b74aa09b512665fa2327e2" candidate_uuid: "4764a44e2a814f6db9d0185ea0bc2425" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b11fb305a694cb0bb64203a3b2ba5c8" is_pre_election: true
I20251209 08:57:58.795822 22257 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "18aba1ab36b74aa09b512665fa2327e2" candidate_uuid: "4764a44e2a814f6db9d0185ea0bc2425" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2da3b1d7e30440909571db652b3bf08e" is_pre_election: true
I20251209 08:57:58.795894 22114 raft_consensus.cc:2468] T 18aba1ab36b74aa09b512665fa2327e2 P 6b11fb305a694cb0bb64203a3b2ba5c8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4764a44e2a814f6db9d0185ea0bc2425 in term 0.
I20251209 08:57:58.795958 22257 raft_consensus.cc:2468] T 18aba1ab36b74aa09b512665fa2327e2 P 2da3b1d7e30440909571db652b3bf08e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4764a44e2a814f6db9d0185ea0bc2425 in term 0.
I20251209 08:57:58.796270 21904 leader_election.cc:304] T 18aba1ab36b74aa09b512665fa2327e2 P 4764a44e2a814f6db9d0185ea0bc2425 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4764a44e2a814f6db9d0185ea0bc2425, 6b11fb305a694cb0bb64203a3b2ba5c8; no voters:
I20251209 08:57:58.796471 22331 raft_consensus.cc:2804] T 18aba1ab36b74aa09b512665fa2327e2 P 4764a44e2a814f6db9d0185ea0bc2425 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251209 08:57:58.796533 22331 raft_consensus.cc:493] T 18aba1ab36b74aa09b512665fa2327e2 P 4764a44e2a814f6db9d0185ea0bc2425 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251209 08:57:58.796589 22331 raft_consensus.cc:3060] T 18aba1ab36b74aa09b512665fa2327e2 P 4764a44e2a814f6db9d0185ea0bc2425 [term 0 FOLLOWER]: Advancing to term 1
I20251209 08:57:58.797483 22331 raft_consensus.cc:515] T 18aba1ab36b74aa09b512665fa2327e2 P 4764a44e2a814f6db9d0185ea0bc2425 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4764a44e2a814f6db9d0185ea0bc2425" member_type: VOTER last_known_addr { host: "127.20.49.65" port: 43253 } } peers { permanent_uuid: "6b11fb305a694cb0bb64203a3b2ba5c8" member_type: VOTER last_known_addr { host: "127.20.49.66" port: 39533 } } peers { permanent_uuid: "2da3b1d7e30440909571db652b3bf08e" member_type: VOTER last_known_addr { host: "127.20.49.67" port: 35127 } }
I20251209 08:57:58.797851 22331 leader_election.cc:290] T 18aba1ab36b74aa09b512665fa2327e2 P 4764a44e2a814f6db9d0185ea0bc2425 [CANDIDATE]: Term 1 election: Requested vote from peers 6b11fb305a694cb0bb64203a3b2ba5c8 (127.20.49.66:39533), 2da3b1d7e30440909571db652b3bf08e (127.20.49.67:35127)
I20251209 08:57:58.798187 22114 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "18aba1ab36b74aa09b512665fa2327e2" candidate_uuid: "4764a44e2a814f6db9d0185ea0bc2425" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b11fb305a694cb0bb64203a3b2ba5c8"
I20251209 08:57:58.798218 22257 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "18aba1ab36b74aa09b512665fa2327e2" candidate_uuid: "4764a44e2a814f6db9d0185ea0bc2425" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2da3b1d7e30440909571db652b3bf08e"
I20251209 08:57:58.798290 22114 raft_consensus.cc:3060] T 18aba1ab36b74aa09b512665fa2327e2 P 6b11fb305a694cb0bb64203a3b2ba5c8 [term 0 FOLLOWER]: Advancing to term 1
I20251209 08:57:58.798327 22257 raft_consensus.cc:3060] T 18aba1ab36b74aa09b512665fa2327e2 P 2da3b1d7e30440909571db652b3bf08e [term 0 FOLLOWER]: Advancing to term 1
I20251209 08:57:58.799135 22114 raft_consensus.cc:2468] T 18aba1ab36b74aa09b512665fa2327e2 P 6b11fb305a694cb0bb64203a3b2ba5c8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4764a44e2a814f6db9d0185ea0bc2425 in term 1.
I20251209 08:57:58.799227 22257 raft_consensus.cc:2468] T 18aba1ab36b74aa09b512665fa2327e2 P 2da3b1d7e30440909571db652b3bf08e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4764a44e2a814f6db9d0185ea0bc2425 in term 1.
I20251209 08:57:58.799386 21904 leader_election.cc:304] T 18aba1ab36b74aa09b512665fa2327e2 P 4764a44e2a814f6db9d0185ea0bc2425 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4764a44e2a814f6db9d0185ea0bc2425, 6b11fb305a694cb0bb64203a3b2ba5c8; no voters:
I20251209 08:57:58.799569 22331 raft_consensus.cc:2804] T 18aba1ab36b74aa09b512665fa2327e2 P 4764a44e2a814f6db9d0185ea0bc2425 [term 1 FOLLOWER]: Leader election won for term 1
I20251209 08:57:58.799758 22331 raft_consensus.cc:697] T 18aba1ab36b74aa09b512665fa2327e2 P 4764a44e2a814f6db9d0185ea0bc2425 [term 1 LEADER]: Becoming Leader. State: Replica: 4764a44e2a814f6db9d0185ea0bc2425, State: Running, Role: LEADER
I20251209 08:57:58.799993 22331 consensus_queue.cc:237] T 18aba1ab36b74aa09b512665fa2327e2 P 4764a44e2a814f6db9d0185ea0bc2425 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4764a44e2a814f6db9d0185ea0bc2425" member_type: VOTER last_known_addr { host: "127.20.49.65" port: 43253 } } peers { permanent_uuid: "6b11fb305a694cb0bb64203a3b2ba5c8" member_type: VOTER last_known_addr { host: "127.20.49.66" port: 39533 } } peers { permanent_uuid: "2da3b1d7e30440909571db652b3bf08e" member_type: VOTER last_known_addr { host: "127.20.49.67" port: 35127 } }
I20251209 08:57:58.801100 22369 tablet_replica.cc:442] T 18aba1ab36b74aa09b512665fa2327e2 P 4764a44e2a814f6db9d0185ea0bc2425: TxnStatusTablet state changed. Reason: New leader 4764a44e2a814f6db9d0185ea0bc2425. Latest consensus state: current_term: 1 leader_uuid: "4764a44e2a814f6db9d0185ea0bc2425" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4764a44e2a814f6db9d0185ea0bc2425" member_type: VOTER last_known_addr { host: "127.20.49.65" port: 43253 } } peers { permanent_uuid: "6b11fb305a694cb0bb64203a3b2ba5c8" member_type: VOTER last_known_addr { host: "127.20.49.66" port: 39533 } } peers { permanent_uuid: "2da3b1d7e30440909571db652b3bf08e" member_type: VOTER last_known_addr { host: "127.20.49.67" port: 35127 } } }
I20251209 08:57:58.801224 22369 tablet_replica.cc:445] T 18aba1ab36b74aa09b512665fa2327e2 P 4764a44e2a814f6db9d0185ea0bc2425: This TxnStatusTablet replica's current role is: LEADER
I20251209 08:57:58.801499 22371 txn_status_manager.cc:874] Waiting until node catch up with all replicated operations in previous term...
I20251209 08:57:58.801703 22371 txn_status_manager.cc:930] Loading transaction status metadata into memory...
I20251209 08:57:58.802354 21813 catalog_manager.cc:5654] T 18aba1ab36b74aa09b512665fa2327e2 P 4764a44e2a814f6db9d0185ea0bc2425 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4764a44e2a814f6db9d0185ea0bc2425 (127.20.49.65). New cstate: current_term: 1 leader_uuid: "4764a44e2a814f6db9d0185ea0bc2425" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4764a44e2a814f6db9d0185ea0bc2425" member_type: VOTER last_known_addr { host: "127.20.49.65" port: 43253 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6b11fb305a694cb0bb64203a3b2ba5c8" member_type: VOTER last_known_addr { host: "127.20.49.66" port: 39533 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2da3b1d7e30440909571db652b3bf08e" member_type: VOTER last_known_addr { host: "127.20.49.67" port: 35127 } health_report { overall_health: UNKNOWN } } }
W20251209 08:57:58.805836 22020 tablet.cc:2378] T 18aba1ab36b74aa09b512665fa2327e2 P 4764a44e2a814f6db9d0185ea0bc2425: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20251209 08:57:58.816973 22325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:57:58.813302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40426 (local address 127.20.49.65:43253)
1209 08:57:58.813418 (+ 116us) server_negotiation.cc:207] Beginning negotiation
1209 08:57:58.813423 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1209 08:57:58.813438 (+ 15us) server_negotiation.cc:408] Connection header received
1209 08:57:58.813634 (+ 196us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1209 08:57:58.813637 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1209 08:57:58.813720 (+ 83us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1209 08:57:58.813808 (+ 88us) server_negotiation.cc:227] Negotiated authn=TOKEN
1209 08:57:58.814235 (+ 427us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:57:58.814940 (+ 705us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1209 08:57:58.815846 (+ 906us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:57:58.816025 (+ 179us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:57:58.816078 (+ 53us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1209 08:57:58.816575 (+ 497us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1209 08:57:58.816687 (+ 112us) server_negotiation.cc:1036] Waiting for connection context
1209 08:57:58.816798 (+ 111us) server_negotiation.cc:300] Negotiation successful
1209 08:57:58.816850 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20251209 08:57:58.816990 22363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:57:58.813023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.49.65:43253 (local address 127.0.0.1:40426)
1209 08:57:58.813364 (+ 341us) negotiation.cc:107] Waiting for socket to connect
1209 08:57:58.813380 (+ 16us) client_negotiation.cc:174] Beginning negotiation
1209 08:57:58.813502 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1209 08:57:58.813848 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1209 08:57:58.813852 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
1209 08:57:58.813861 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
1209 08:57:58.814118 (+ 257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:57:58.814127 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:57:58.815074 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1209 08:57:58.815078 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1209 08:57:58.815734 (+ 656us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:57:58.815744 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:57:58.815862 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:57:58.815908 (+ 46us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
1209 08:57:58.816681 (+ 773us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
1209 08:57:58.816697 (+ 16us) client_negotiation.cc:769] Sending connection context
1209 08:57:58.816781 (+ 84us) client_negotiation.cc:240] Negotiation successful
1209 08:57:58.816859 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":221}
I20251209 08:57:58.819979 22114 raft_consensus.cc:1275] T 18aba1ab36b74aa09b512665fa2327e2 P 6b11fb305a694cb0bb64203a3b2ba5c8 [term 1 FOLLOWER]: Refusing update from remote peer 4764a44e2a814f6db9d0185ea0bc2425: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251209 08:57:58.820183 22257 raft_consensus.cc:1275] T 18aba1ab36b74aa09b512665fa2327e2 P 2da3b1d7e30440909571db652b3bf08e [term 1 FOLLOWER]: Refusing update from remote peer 4764a44e2a814f6db9d0185ea0bc2425: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251209 08:57:58.820420 22331 consensus_queue.cc:1048] T 18aba1ab36b74aa09b512665fa2327e2 P 4764a44e2a814f6db9d0185ea0bc2425 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b11fb305a694cb0bb64203a3b2ba5c8" member_type: VOTER last_known_addr { host: "127.20.49.66" port: 39533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251209 08:57:58.820744 22369 consensus_queue.cc:1048] T 18aba1ab36b74aa09b512665fa2327e2 P 4764a44e2a814f6db9d0185ea0bc2425 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2da3b1d7e30440909571db652b3bf08e" member_type: VOTER last_known_addr { host: "127.20.49.67" port: 35127 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251209 08:57:58.822511 22336 tablet_replica.cc:442] T 18aba1ab36b74aa09b512665fa2327e2 P 2da3b1d7e30440909571db652b3bf08e: TxnStatusTablet state changed. Reason: New leader 4764a44e2a814f6db9d0185ea0bc2425. Latest consensus state: current_term: 1 leader_uuid: "4764a44e2a814f6db9d0185ea0bc2425" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4764a44e2a814f6db9d0185ea0bc2425" member_type: VOTER last_known_addr { host: "127.20.49.65" port: 43253 } } peers { permanent_uuid: "6b11fb305a694cb0bb64203a3b2ba5c8" member_type: VOTER last_known_addr { host: "127.20.49.66" port: 39533 } } peers { permanent_uuid: "2da3b1d7e30440909571db652b3bf08e" member_type: VOTER last_known_addr { host: "127.20.49.67" port: 35127 } } }
I20251209 08:57:58.822629 22336 tablet_replica.cc:445] T 18aba1ab36b74aa09b512665fa2327e2 P 2da3b1d7e30440909571db652b3bf08e: This TxnStatusTablet replica's current role is: FOLLOWER
I20251209 08:57:58.822796 22356 tablet_replica.cc:442] T 18aba1ab36b74aa09b512665fa2327e2 P 6b11fb305a694cb0bb64203a3b2ba5c8: TxnStatusTablet state changed. Reason: New leader 4764a44e2a814f6db9d0185ea0bc2425. Latest consensus state: current_term: 1 leader_uuid: "4764a44e2a814f6db9d0185ea0bc2425" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4764a44e2a814f6db9d0185ea0bc2425" member_type: VOTER last_known_addr { host: "127.20.49.65" port: 43253 } } peers { permanent_uuid: "6b11fb305a694cb0bb64203a3b2ba5c8" member_type: VOTER last_known_addr { host: "127.20.49.66" port: 39533 } } peers { permanent_uuid: "2da3b1d7e30440909571db652b3bf08e" member_type: VOTER last_known_addr { host: "127.20.49.67" port: 35127 } } }
I20251209 08:57:58.822906 22356 tablet_replica.cc:445] T 18aba1ab36b74aa09b512665fa2327e2 P 6b11fb305a694cb0bb64203a3b2ba5c8: This TxnStatusTablet replica's current role is: FOLLOWER
I20251209 08:57:58.824637 22331 tablet_replica.cc:442] T 18aba1ab36b74aa09b512665fa2327e2 P 4764a44e2a814f6db9d0185ea0bc2425: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4764a44e2a814f6db9d0185ea0bc2425" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4764a44e2a814f6db9d0185ea0bc2425" member_type: VOTER last_known_addr { host: "127.20.49.65" port: 43253 } } peers { permanent_uuid: "6b11fb305a694cb0bb64203a3b2ba5c8" member_type: VOTER last_known_addr { host: "127.20.49.66" port: 39533 } } peers { permanent_uuid: "2da3b1d7e30440909571db652b3bf08e" member_type: VOTER last_known_addr { host: "127.20.49.67" port: 35127 } } }
I20251209 08:57:58.824734 22331 tablet_replica.cc:445] T 18aba1ab36b74aa09b512665fa2327e2 P 4764a44e2a814f6db9d0185ea0bc2425: This TxnStatusTablet replica's current role is: LEADER
I20251209 08:57:58.825305 22369 tablet_replica.cc:442] T 18aba1ab36b74aa09b512665fa2327e2 P 4764a44e2a814f6db9d0185ea0bc2425: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4764a44e2a814f6db9d0185ea0bc2425" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4764a44e2a814f6db9d0185ea0bc2425" member_type: VOTER last_known_addr { host: "127.20.49.65" port: 43253 } } peers { permanent_uuid: "6b11fb305a694cb0bb64203a3b2ba5c8" member_type: VOTER last_known_addr { host: "127.20.49.66" port: 39533 } } peers { permanent_uuid: "2da3b1d7e30440909571db652b3bf08e" member_type: VOTER last_known_addr { host: "127.20.49.67" port: 35127 } } }
I20251209 08:57:58.825397 22369 tablet_replica.cc:445] T 18aba1ab36b74aa09b512665fa2327e2 P 4764a44e2a814f6db9d0185ea0bc2425: This TxnStatusTablet replica's current role is: LEADER
I20251209 08:57:58.826900 22356 tablet_replica.cc:442] T 18aba1ab36b74aa09b512665fa2327e2 P 6b11fb305a694cb0bb64203a3b2ba5c8: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4764a44e2a814f6db9d0185ea0bc2425" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4764a44e2a814f6db9d0185ea0bc2425" member_type: VOTER last_known_addr { host: "127.20.49.65" port: 43253 } } peers { permanent_uuid: "6b11fb305a694cb0bb64203a3b2ba5c8" member_type: VOTER last_known_addr { host: "127.20.49.66" port: 39533 } } peers { permanent_uuid: "2da3b1d7e30440909571db652b3bf08e" member_type: VOTER last_known_addr { host: "127.20.49.67" port: 35127 } } }
I20251209 08:57:58.826999 22356 tablet_replica.cc:445] T 18aba1ab36b74aa09b512665fa2327e2 P 6b11fb305a694cb0bb64203a3b2ba5c8: This TxnStatusTablet replica's current role is: FOLLOWER
I20251209 08:57:58.828454 22336 tablet_replica.cc:442] T 18aba1ab36b74aa09b512665fa2327e2 P 2da3b1d7e30440909571db652b3bf08e: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4764a44e2a814f6db9d0185ea0bc2425" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4764a44e2a814f6db9d0185ea0bc2425" member_type: VOTER last_known_addr { host: "127.20.49.65" port: 43253 } } peers { permanent_uuid: "6b11fb305a694cb0bb64203a3b2ba5c8" member_type: VOTER last_known_addr { host: "127.20.49.66" port: 39533 } } peers { permanent_uuid: "2da3b1d7e30440909571db652b3bf08e" member_type: VOTER last_known_addr { host: "127.20.49.67" port: 35127 } } }
I20251209 08:57:58.828563 22336 tablet_replica.cc:445] T 18aba1ab36b74aa09b512665fa2327e2 P 2da3b1d7e30440909571db652b3bf08e: This TxnStatusTablet replica's current role is: FOLLOWER
I20251209 08:57:58.833123 22323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:57:58.830336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41794 (local address 127.20.49.66:39533)
1209 08:57:58.830459 (+ 123us) server_negotiation.cc:207] Beginning negotiation
1209 08:57:58.830463 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1209 08:57:58.830475 (+ 12us) server_negotiation.cc:408] Connection header received
1209 08:57:58.830551 (+ 76us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1209 08:57:58.830554 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1209 08:57:58.830613 (+ 59us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1209 08:57:58.830706 (+ 93us) server_negotiation.cc:227] Negotiated authn=TOKEN
1209 08:57:58.831032 (+ 326us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:57:58.831508 (+ 476us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1209 08:57:58.832109 (+ 601us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:57:58.832287 (+ 178us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:57:58.832338 (+ 51us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1209 08:57:58.832754 (+ 416us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1209 08:57:58.832866 (+ 112us) server_negotiation.cc:1036] Waiting for connection context
1209 08:57:58.832951 (+ 85us) server_negotiation.cc:300] Negotiation successful
1209 08:57:58.832996 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20251209 08:57:58.842796 22325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:57:58.840006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.49.66:42457 (local address 127.20.49.65:43253)
1209 08:57:58.840134 (+ 128us) server_negotiation.cc:207] Beginning negotiation
1209 08:57:58.840139 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1209 08:57:58.840310 (+ 171us) server_negotiation.cc:408] Connection header received
1209 08:57:58.840451 (+ 141us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1209 08:57:58.840453 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1209 08:57:58.840500 (+ 47us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1209 08:57:58.840570 (+ 70us) server_negotiation.cc:227] Negotiated authn=TOKEN
1209 08:57:58.840949 (+ 379us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:57:58.841421 (+ 472us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1209 08:57:58.842125 (+ 704us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:57:58.842319 (+ 194us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:57:58.842361 (+ 42us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1209 08:57:58.842433 (+ 72us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1209 08:57:58.842501 (+ 68us) server_negotiation.cc:1036] Waiting for connection context
1209 08:57:58.842618 (+ 117us) server_negotiation.cc:300] Negotiation successful
1209 08:57:58.842665 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20251209 08:57:58.842813 22378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:57:58.839918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.49.65:43253 (local address 127.20.49.66:42457)
1209 08:57:58.840230 (+ 312us) negotiation.cc:107] Waiting for socket to connect
1209 08:57:58.840242 (+ 12us) client_negotiation.cc:174] Beginning negotiation
1209 08:57:58.840346 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1209 08:57:58.840586 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1209 08:57:58.840589 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1209 08:57:58.840597 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
1209 08:57:58.840836 (+ 239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:57:58.840842 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:57:58.841529 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1209 08:57:58.841532 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1209 08:57:58.842009 (+ 477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:57:58.842017 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:57:58.842135 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:57:58.842181 (+ 46us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
1209 08:57:58.842536 (+ 355us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
1209 08:57:58.842541 (+ 5us) client_negotiation.cc:769] Sending connection context
1209 08:57:58.842602 (+ 61us) client_negotiation.cc:240] Negotiation successful
1209 08:57:58.842668 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":110,"threads_started":1}
I20251209 08:57:58.849431 22257 raft_consensus.cc:1275] T 83a821ea6f9946628a9ca70d835606c0 P 2da3b1d7e30440909571db652b3bf08e [term 1 FOLLOWER]: Refusing update from remote peer 6b11fb305a694cb0bb64203a3b2ba5c8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251209 08:57:58.849570 21973 raft_consensus.cc:1275] T 83a821ea6f9946628a9ca70d835606c0 P 4764a44e2a814f6db9d0185ea0bc2425 [term 1 FOLLOWER]: Refusing update from remote peer 6b11fb305a694cb0bb64203a3b2ba5c8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251209 08:57:58.849957 22356 consensus_queue.cc:1048] T 83a821ea6f9946628a9ca70d835606c0 P 6b11fb305a694cb0bb64203a3b2ba5c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2da3b1d7e30440909571db652b3bf08e" member_type: VOTER last_known_addr { host: "127.20.49.67" port: 35127 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251209 08:57:58.850102 22334 consensus_queue.cc:1048] T 83a821ea6f9946628a9ca70d835606c0 P 6b11fb305a694cb0bb64203a3b2ba5c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4764a44e2a814f6db9d0185ea0bc2425" member_type: VOTER last_known_addr { host: "127.20.49.65" port: 43253 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251209 08:57:58.874864 22384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:57:58.871169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.49.66:39533 (local address 127.20.49.65:46929)
1209 08:57:58.871533 (+ 364us) negotiation.cc:107] Waiting for socket to connect
1209 08:57:58.871548 (+ 15us) client_negotiation.cc:174] Beginning negotiation
1209 08:57:58.871664 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1209 08:57:58.871915 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1209 08:57:58.871917 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1209 08:57:58.871924 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
1209 08:57:58.872161 (+ 237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:57:58.872166 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:57:58.873210 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1209 08:57:58.873214 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1209 08:57:58.873787 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:57:58.873797 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:57:58.874128 (+ 331us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:57:58.874180 (+ 52us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
1209 08:57:58.874536 (+ 356us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
1209 08:57:58.874541 (+ 5us) client_negotiation.cc:769] Sending connection context
1209 08:57:58.874606 (+ 65us) client_negotiation.cc:240] Negotiation successful
1209 08:57:58.874691 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":85,"threads_started":1}
I20251209 08:57:58.875095 22323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:57:58.871225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.49.65:46929 (local address 127.20.49.66:39533)
1209 08:57:58.871376 (+ 151us) server_negotiation.cc:207] Beginning negotiation
1209 08:57:58.871382 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1209 08:57:58.871633 (+ 251us) server_negotiation.cc:408] Connection header received
1209 08:57:58.871763 (+ 130us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1209 08:57:58.871767 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1209 08:57:58.871824 (+ 57us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1209 08:57:58.871899 (+ 75us) server_negotiation.cc:227] Negotiated authn=TOKEN
1209 08:57:58.872249 (+ 350us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:57:58.872992 (+ 743us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1209 08:57:58.873897 (+ 905us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:57:58.874101 (+ 204us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:57:58.874312 (+ 211us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1209 08:57:58.874441 (+ 129us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1209 08:57:58.874523 (+ 82us) server_negotiation.cc:1036] Waiting for connection context
1209 08:57:58.874906 (+ 383us) server_negotiation.cc:300] Negotiation successful
1209 08:57:58.874959 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20251209 08:57:58.903154 21813 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42554:
table { table_name: "test-table" } modify_external_catalogs: true
I20251209 08:57:58.903337 21813 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42554:
table { table_name: "test-table" } modify_external_catalogs: true
I20251209 08:57:58.905860 21813 catalog_manager.cc:5941] T 00000000000000000000000000000000 P fc08086ef48c4a79a51eac09446fa095: Sending DeleteTablet for 3 replicas of tablet 83a821ea6f9946628a9ca70d835606c0
I20251209 08:57:58.906533 21953 tablet_service.cc:1552] Processing DeleteTablet for tablet 83a821ea6f9946628a9ca70d835606c0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-09 08:57:58 UTC) from {username='kudu', principal='kudu/127.20.49.126@KRBTEST.COM'} at 127.0.0.1:40412
I20251209 08:57:58.906709 22237 tablet_service.cc:1552] Processing DeleteTablet for tablet 83a821ea6f9946628a9ca70d835606c0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-09 08:57:58 UTC) from {username='kudu', principal='kudu/127.20.49.126@KRBTEST.COM'} at 127.0.0.1:40448
I20251209 08:57:58.906760 22349 tablet_replica.cc:333] T 83a821ea6f9946628a9ca70d835606c0 P 4764a44e2a814f6db9d0185ea0bc2425: stopping tablet replica
I20251209 08:57:58.906888 22349 raft_consensus.cc:2243] T 83a821ea6f9946628a9ca70d835606c0 P 4764a44e2a814f6db9d0185ea0bc2425 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251209 08:57:58.907025 22349 raft_consensus.cc:2272] T 83a821ea6f9946628a9ca70d835606c0 P 4764a44e2a814f6db9d0185ea0bc2425 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251209 08:57:58.907094 22095 tablet_service.cc:1552] Processing DeleteTablet for tablet 83a821ea6f9946628a9ca70d835606c0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-09 08:57:58 UTC) from {username='kudu', principal='kudu/127.20.49.126@KRBTEST.COM'} at 127.0.0.1:41788
I20251209 08:57:58.907238 22350 tablet_replica.cc:333] T 83a821ea6f9946628a9ca70d835606c0 P 2da3b1d7e30440909571db652b3bf08e: stopping tablet replica
I20251209 08:57:58.907369 22350 raft_consensus.cc:2243] T 83a821ea6f9946628a9ca70d835606c0 P 2da3b1d7e30440909571db652b3bf08e [term 1 FOLLOWER]: Raft consensus shutting down.
I20251209 08:57:58.907388 22348 tablet_replica.cc:333] T 83a821ea6f9946628a9ca70d835606c0 P 6b11fb305a694cb0bb64203a3b2ba5c8: stopping tablet replica
I20251209 08:57:58.907481 22348 raft_consensus.cc:2243] T 83a821ea6f9946628a9ca70d835606c0 P 6b11fb305a694cb0bb64203a3b2ba5c8 [term 1 LEADER]: Raft consensus shutting down.
I20251209 08:57:58.907496 22350 raft_consensus.cc:2272] T 83a821ea6f9946628a9ca70d835606c0 P 2da3b1d7e30440909571db652b3bf08e [term 1 FOLLOWER]: Raft consensus is shut down!
I20251209 08:57:58.907774 22348 raft_consensus.cc:2272] T 83a821ea6f9946628a9ca70d835606c0 P 6b11fb305a694cb0bb64203a3b2ba5c8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251209 08:57:58.907927 22349 ts_tablet_manager.cc:1916] T 83a821ea6f9946628a9ca70d835606c0 P 4764a44e2a814f6db9d0185ea0bc2425: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251209 08:57:58.908284 22350 ts_tablet_manager.cc:1916] T 83a821ea6f9946628a9ca70d835606c0 P 2da3b1d7e30440909571db652b3bf08e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251209 08:57:58.910868 22350 ts_tablet_manager.cc:1929] T 83a821ea6f9946628a9ca70d835606c0 P 2da3b1d7e30440909571db652b3bf08e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20251209 08:57:58.910948 22350 log.cc:1199] T 83a821ea6f9946628a9ca70d835606c0 P 2da3b1d7e30440909571db652b3bf08e: Deleting WAL directory at /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765270668420570-20677-0/minicluster-data/ts-2/wal/wals/83a821ea6f9946628a9ca70d835606c0
I20251209 08:57:58.910964 22349 ts_tablet_manager.cc:1929] T 83a821ea6f9946628a9ca70d835606c0 P 4764a44e2a814f6db9d0185ea0bc2425: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20251209 08:57:58.911029 22349 log.cc:1199] T 83a821ea6f9946628a9ca70d835606c0 P 4764a44e2a814f6db9d0185ea0bc2425: Deleting WAL directory at /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765270668420570-20677-0/minicluster-data/ts-0/wal/wals/83a821ea6f9946628a9ca70d835606c0
I20251209 08:57:58.911222 22350 ts_tablet_manager.cc:1950] T 83a821ea6f9946628a9ca70d835606c0 P 2da3b1d7e30440909571db652b3bf08e: Deleting consensus metadata
I20251209 08:57:58.911319 22349 ts_tablet_manager.cc:1950] T 83a821ea6f9946628a9ca70d835606c0 P 4764a44e2a814f6db9d0185ea0bc2425: Deleting consensus metadata
I20251209 08:57:58.911984 21801 catalog_manager.cc:4985] TS 4764a44e2a814f6db9d0185ea0bc2425 (127.20.49.65:43253): tablet 83a821ea6f9946628a9ca70d835606c0 (table test-table [id=e6a5e9b717c540c1998ff3a64857c633]) successfully deleted
I20251209 08:57:58.912014 21798 catalog_manager.cc:4985] TS 2da3b1d7e30440909571db652b3bf08e (127.20.49.67:35127): tablet 83a821ea6f9946628a9ca70d835606c0 (table test-table [id=e6a5e9b717c540c1998ff3a64857c633]) successfully deleted
I20251209 08:57:58.912686 22348 ts_tablet_manager.cc:1916] T 83a821ea6f9946628a9ca70d835606c0 P 6b11fb305a694cb0bb64203a3b2ba5c8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251209 08:57:58.915303 22348 ts_tablet_manager.cc:1929] T 83a821ea6f9946628a9ca70d835606c0 P 6b11fb305a694cb0bb64203a3b2ba5c8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20251209 08:57:58.915382 22348 log.cc:1199] T 83a821ea6f9946628a9ca70d835606c0 P 6b11fb305a694cb0bb64203a3b2ba5c8: Deleting WAL directory at /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1765270668420570-20677-0/minicluster-data/ts-1/wal/wals/83a821ea6f9946628a9ca70d835606c0
Dec 09 08:57:58 dist-test-slave-8w6c krb5kdc[21766](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765270678, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.20.49.65@KRBTEST.COM
I20251209 08:57:58.915627 22348 ts_tablet_manager.cc:1950] T 83a821ea6f9946628a9ca70d835606c0 P 6b11fb305a694cb0bb64203a3b2ba5c8: Deleting consensus metadata
I20251209 08:57:58.916463 21798 catalog_manager.cc:4985] TS 6b11fb305a694cb0bb64203a3b2ba5c8 (127.20.49.66:39533): tablet 83a821ea6f9946628a9ca70d835606c0 (table test-table [id=e6a5e9b717c540c1998ff3a64857c633]) successfully deleted
I20251209 08:57:58.921028 22325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:57:58.908702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40442 (local address 127.20.49.65:43253)
1209 08:57:58.908813 (+ 111us) server_negotiation.cc:207] Beginning negotiation
1209 08:57:58.908818 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1209 08:57:58.909332 (+ 514us) server_negotiation.cc:408] Connection header received
1209 08:57:58.909539 (+ 207us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1209 08:57:58.909545 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
1209 08:57:58.909603 (+ 58us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1209 08:57:58.909660 (+ 57us) server_negotiation.cc:227] Negotiated authn=SASL
1209 08:57:58.910820 (+ 1160us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:57:58.911526 (+ 706us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1209 08:57:58.912622 (+ 1096us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:57:58.912812 (+ 190us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:57:58.916164 (+ 3352us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1209 08:57:58.916193 (+ 29us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1209 08:57:58.916207 (+ 14us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1209 08:57:58.916239 (+ 32us) server_negotiation.cc:923] calling sasl_server_start()
1209 08:57:58.919147 (+ 2908us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1209 08:57:58.919656 (+ 509us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1209 08:57:58.919663 (+ 7us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1209 08:57:58.919666 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
1209 08:57:58.919738 (+ 72us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1209 08:57:58.919970 (+ 232us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1209 08:57:58.919973 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1209 08:57:58.919975 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1209 08:57:58.920205 (+ 230us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1209 08:57:58.920411 (+ 206us) server_negotiation.cc:1036] Waiting for connection context
1209 08:57:58.920578 (+ 167us) server_negotiation.cc:300] Negotiation successful
1209 08:57:58.920721 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
W20251209 08:57:58.921468 22013 server_base.cc:1129] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40442
I20251209 08:57:58.932303 21863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:57:58.923585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42558 (local address 127.20.49.126:35129)
1209 08:57:58.923711 (+ 126us) server_negotiation.cc:207] Beginning negotiation
1209 08:57:58.923716 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1209 08:57:58.923824 (+ 108us) server_negotiation.cc:408] Connection header received
1209 08:57:58.923922 (+ 98us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1209 08:57:58.923926 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1209 08:57:58.923979 (+ 53us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1209 08:57:58.924061 (+ 82us) server_negotiation.cc:227] Negotiated authn=SASL
1209 08:57:58.925260 (+ 1199us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:57:58.926093 (+ 833us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1209 08:57:58.927034 (+ 941us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:57:58.927237 (+ 203us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:57:58.928876 (+ 1639us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1209 08:57:58.928898 (+ 22us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1209 08:57:58.928902 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1209 08:57:58.928933 (+ 31us) server_negotiation.cc:923] calling sasl_server_start()
1209 08:57:58.930775 (+ 1842us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1209 08:57:58.931215 (+ 440us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1209 08:57:58.931220 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1209 08:57:58.931222 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1209 08:57:58.931272 (+ 50us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1209 08:57:58.931495 (+ 223us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1209 08:57:58.931499 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1209 08:57:58.931501 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1209 08:57:58.931677 (+ 176us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1209 08:57:58.931804 (+ 127us) server_negotiation.cc:1036] Waiting for connection context
1209 08:57:58.932006 (+ 202us) server_negotiation.cc:300] Negotiation successful
1209 08:57:58.932133 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
W20251209 08:57:58.933303 21813 server_base.cc:1129] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42558
I20251209 08:57:58.935009 20677 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task8XuDZV/build/debug/bin/kudu with pid 21888
I20251209 08:57:58.943038 20677 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task8XuDZV/build/debug/bin/kudu with pid 22030
I20251209 08:57:58.949743 20677 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task8XuDZV/build/debug/bin/kudu with pid 22172
I20251209 08:57:58.956848 20677 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task8XuDZV/build/debug/bin/kudu with pid 21782
2025-12-09T08:57:58Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (3766 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765270668420570-20677-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Dec 09 08:57:59 dist-test-slave-8w6c krb5kdc[22393](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Dec 09 08:57:59 dist-test-slave-8w6c krb5kdc[22393](info): set up 2 sockets
Dec 09 08:57:59 dist-test-slave-8w6c krb5kdc[22393](info): commencing operation
krb5kdc: starting...
W20251209 08:58:00.962356 20677 mini_kdc.cc:121] Time spent starting KDC: real 1.982s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765270668420570-20677-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765270668420570-20677-0/krb5kdc/test-user.keytab.
Dec 09 08:58:01 dist-test-slave-8w6c krb5kdc[22393](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765270681, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-12-09T08:58:01Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-12-09T08:58:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.49.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.49.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.49.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765270668420570-20677-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.20.49.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765270668420570-20677-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.20.49.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.20.49.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.20.49.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765270668420570-20677-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.20.49.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765270668420570-20677-0/krb5kdc/kudu.keytab.
I20251209 08:58:01.112389 20677 external_mini_cluster.cc:1366] Running /tmp/dist-test-task8XuDZV/build/debug/bin/kudu
/tmp/dist-test-task8XuDZV/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765270668420570-20677-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765270668420570-20677-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765270668420570-20677-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765270668420570-20677-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.49.126:39233
--webserver_interface=127.20.49.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765270668420570-20677-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.49.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.20.49.84:32905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.49.126:39233
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765270668420570-20677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765270668420570-20677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765270668420570-20677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251209 08:58:01.229807 22409 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251209 08:58:01.230075 22409 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251209 08:58:01.230136 22409 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251209 08:58:01.233444 22409 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251209 08:58:01.233512 22409 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20251209 08:58:01.233531 22409 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251209 08:58:01.233552 22409 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251209 08:58:01.233570 22409 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251209 08:58:01.233588 22409 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251209 08:58:01.237947 22409 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.49.84:32905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765270668420570-20677-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765270668420570-20677-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.49.126:39233
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765270668420570-20677-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.49.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.49.126:39233
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765270668420570-20677-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.49.126
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22409
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765270668420570-20677-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 468897755969d686fb08386f42f7835685e7953a
build type DEBUG
built by None at 09 Dec 2025 08:43:15 UTC on 24a791456cd2
build id 9426
I20251209 08:58:01.239015 22409 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251209 08:58:01.239904 22409 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251209 08:58:01.245666 22417 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251209 08:58:01.245668 22415 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251209 08:58:01.245667 22414 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251209 08:58:01.246073 22409 server_base.cc:1047] running on GCE node
I20251209 08:58:01.246592 22409 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251209 08:58:01.247534 22409 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251209 08:58:01.248695 22409 hybrid_clock.cc:648] HybridClock initialized: now 1765270681248661 us; error 58 us; skew 500 ppm
Dec 09 08:58:01 dist-test-slave-8w6c krb5kdc[22393](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765270681, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.49.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251209 08:58:01.251363 22409 init.cc:377] Logged in from keytab as kudu/127.20.49.126@KRBTEST.COM (short username kudu)
I20251209 08:58:01.252449 22409 webserver.cc:492] Webserver started at http://127.20.49.126:33021/ using document root <none> and password file <none>
I20251209 08:58:01.253003 22409 fs_manager.cc:362] Metadata directory not provided
I20251209 08:58:01.253059 22409 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251209 08:58:01.253332 22409 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251209 08:58:01.255317 22409 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765270668420570-20677-0/minicluster-data/master-0/data/instance:
uuid: "39341759984c48989ceba01b8acb2da9"
format_stamp: "Formatted at 2025-12-09 08:58:01 on dist-test-slave-8w6c"
server_key: "ec251f94197a91a042cfdf1dda2d92ed"
server_key_iv: "84f6e5aa1b444ca83ef99cffdd593c8b"
server_key_version: "encryptionkey@0"
I20251209 08:58:01.255829 22409 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765270668420570-20677-0/minicluster-data/master-0/wal/instance:
uuid: "39341759984c48989ceba01b8acb2da9"
format_stamp: "Formatted at 2025-12-09 08:58:01 on dist-test-slave-8w6c"
server_key: "ec251f94197a91a042cfdf1dda2d92ed"
server_key_iv: "84f6e5aa1b444ca83ef99cffdd593c8b"
server_key_version: "encryptionkey@0"
I20251209 08:58:01.259459 22409 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20251209 08:58:01.261656 22424 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251209 08:58:01.262693 22409 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20251209 08:58:01.262804 22409 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765270668420570-20677-0/minicluster-data/master-0/data,/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765270668420570-20677-0/minicluster-data/master-0/wal
uuid: "39341759984c48989ceba01b8acb2da9"
format_stamp: "Formatted at 2025-12-09 08:58:01 on dist-test-slave-8w6c"
server_key: "ec251f94197a91a042cfdf1dda2d92ed"
server_key_iv: "84f6e5aa1b444ca83ef99cffdd593c8b"
server_key_version: "encryptionkey@0"
I20251209 08:58:01.262897 22409 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765270668420570-20677-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765270668420570-20677-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765270668420570-20677-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251209 08:58:01.292320 22409 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251209 08:58:01.295626 22409 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251209 08:58:01.295825 22409 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251209 08:58:01.304500 22409 rpc_server.cc:307] RPC server started. Bound to: 127.20.49.126:39233
I20251209 08:58:01.304744 22486 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.49.126:39233 every 8 connection(s)
I20251209 08:58:01.305574 22409 server_base.cc:1179] Dumped server information to /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765270668420570-20677-0/minicluster-data/master-0/data/info.pb
I20251209 08:58:01.308387 22487 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251209 08:58:01.309198 20677 external_mini_cluster.cc:1428] Started /tmp/dist-test-task8XuDZV/build/debug/bin/kudu as pid 22409
I20251209 08:58:01.309320 20677 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765270668420570-20677-0/minicluster-data/master-0/wal/instance
I20251209 08:58:01.309582 20677 external_mini_cluster.cc:1468] Setting key c60f35be3350bb8a68e5f537f007b8c7
I20251209 08:58:01.314128 22487 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 39341759984c48989ceba01b8acb2da9: Bootstrap starting.
I20251209 08:58:01.316344 22487 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 39341759984c48989ceba01b8acb2da9: Neither blocks nor log segments found. Creating new log.
I20251209 08:58:01.317139 22487 log.cc:826] T 00000000000000000000000000000000 P 39341759984c48989ceba01b8acb2da9: Log is configured to *not* fsync() on all Append() calls
Dec 09 08:58:01 dist-test-slave-8w6c krb5kdc[22393](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765270681, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.49.126@KRBTEST.COM
I20251209 08:58:01.319185 22487 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 39341759984c48989ceba01b8acb2da9: No bootstrap required, opened a new log
I20251209 08:58:01.322449 22487 raft_consensus.cc:359] T 00000000000000000000000000000000 P 39341759984c48989ceba01b8acb2da9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39341759984c48989ceba01b8acb2da9" member_type: VOTER last_known_addr { host: "127.20.49.126" port: 39233 } }
I20251209 08:58:01.322650 22487 raft_consensus.cc:385] T 00000000000000000000000000000000 P 39341759984c48989ceba01b8acb2da9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251209 08:58:01.322691 22487 raft_consensus.cc:740] T 00000000000000000000000000000000 P 39341759984c48989ceba01b8acb2da9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39341759984c48989ceba01b8acb2da9, State: Initialized, Role: FOLLOWER
I20251209 08:58:01.322801 22490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:58:01.311068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42086 (local address 127.20.49.126:39233)
1209 08:58:01.311559 (+ 491us) server_negotiation.cc:207] Beginning negotiation
1209 08:58:01.311570 (+ 11us) server_negotiation.cc:400] Waiting for connection header
1209 08:58:01.311608 (+ 38us) server_negotiation.cc:408] Connection header received
1209 08:58:01.312126 (+ 518us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1209 08:58:01.312149 (+ 23us) server_negotiation.cc:462] Received NEGOTIATE request from client
1209 08:58:01.312466 (+ 317us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1209 08:58:01.312811 (+ 345us) server_negotiation.cc:227] Negotiated authn=SASL
1209 08:58:01.313824 (+ 1013us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:58:01.314615 (+ 791us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1209 08:58:01.315269 (+ 654us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:58:01.315592 (+ 323us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:58:01.318177 (+ 2585us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1209 08:58:01.318197 (+ 20us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1209 08:58:01.318208 (+ 11us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1209 08:58:01.318243 (+ 35us) server_negotiation.cc:923] calling sasl_server_start()
1209 08:58:01.320175 (+ 1932us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1209 08:58:01.320612 (+ 437us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1209 08:58:01.320616 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1209 08:58:01.320621 (+ 5us) server_negotiation.cc:968] calling sasl_server_step()
1209 08:58:01.320694 (+ 73us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1209 08:58:01.320918 (+ 224us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1209 08:58:01.320922 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1209 08:58:01.320924 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1209 08:58:01.321258 (+ 334us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1209 08:58:01.321464 (+ 206us) server_negotiation.cc:1036] Waiting for connection context
1209 08:58:01.321790 (+ 326us) server_negotiation.cc:300] Negotiation successful
1209 08:58:01.322093 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":353,"thread_start_us":158,"threads_started":1}
I20251209 08:58:01.323290 22487 consensus_queue.cc:260] T 00000000000000000000000000000000 P 39341759984c48989ceba01b8acb2da9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39341759984c48989ceba01b8acb2da9" member_type: VOTER last_known_addr { host: "127.20.49.126" port: 39233 } }
I20251209 08:58:01.323437 22487 raft_consensus.cc:399] T 00000000000000000000000000000000 P 39341759984c48989ceba01b8acb2da9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251209 08:58:01.323506 22487 raft_consensus.cc:493] T 00000000000000000000000000000000 P 39341759984c48989ceba01b8acb2da9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251209 08:58:01.323603 22487 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 39341759984c48989ceba01b8acb2da9 [term 0 FOLLOWER]: Advancing to term 1
I20251209 08:58:01.324465 22487 raft_consensus.cc:515] T 00000000000000000000000000000000 P 39341759984c48989ceba01b8acb2da9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39341759984c48989ceba01b8acb2da9" member_type: VOTER last_known_addr { host: "127.20.49.126" port: 39233 } }
I20251209 08:58:01.324837 22487 leader_election.cc:304] T 00000000000000000000000000000000 P 39341759984c48989ceba01b8acb2da9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 39341759984c48989ceba01b8acb2da9; no voters:
I20251209 08:58:01.325147 22487 leader_election.cc:290] T 00000000000000000000000000000000 P 39341759984c48989ceba01b8acb2da9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20251209 08:58:01.325284 22492 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 39341759984c48989ceba01b8acb2da9 [term 1 FOLLOWER]: Leader election won for term 1
I20251209 08:58:01.325497 22492 raft_consensus.cc:697] T 00000000000000000000000000000000 P 39341759984c48989ceba01b8acb2da9 [term 1 LEADER]: Becoming Leader. State: Replica: 39341759984c48989ceba01b8acb2da9, State: Running, Role: LEADER
I20251209 08:58:01.325801 22492 consensus_queue.cc:237] T 00000000000000000000000000000000 P 39341759984c48989ceba01b8acb2da9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39341759984c48989ceba01b8acb2da9" member_type: VOTER last_known_addr { host: "127.20.49.126" port: 39233 } }
I20251209 08:58:01.326278 22487 sys_catalog.cc:565] T 00000000000000000000000000000000 P 39341759984c48989ceba01b8acb2da9 [sys.catalog]: configured and running, proceeding with master startup.
I20251209 08:58:01.327459 22493 sys_catalog.cc:455] T 00000000000000000000000000000000 P 39341759984c48989ceba01b8acb2da9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "39341759984c48989ceba01b8acb2da9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39341759984c48989ceba01b8acb2da9" member_type: VOTER last_known_addr { host: "127.20.49.126" port: 39233 } } }
I20251209 08:58:01.327461 22494 sys_catalog.cc:455] T 00000000000000000000000000000000 P 39341759984c48989ceba01b8acb2da9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 39341759984c48989ceba01b8acb2da9. Latest consensus state: current_term: 1 leader_uuid: "39341759984c48989ceba01b8acb2da9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39341759984c48989ceba01b8acb2da9" member_type: VOTER last_known_addr { host: "127.20.49.126" port: 39233 } } }
I20251209 08:58:01.327577 22493 sys_catalog.cc:458] T 00000000000000000000000000000000 P 39341759984c48989ceba01b8acb2da9 [sys.catalog]: This master's current role is: LEADER
I20251209 08:58:01.327597 22494 sys_catalog.cc:458] T 00000000000000000000000000000000 P 39341759984c48989ceba01b8acb2da9 [sys.catalog]: This master's current role is: LEADER
I20251209 08:58:01.327921 22501 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251209 08:58:01.331262 22501 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251209 08:58:01.337025 22501 catalog_manager.cc:1357] Generated new cluster ID: a3d418fce182494d840ebf678d9c44d3
I20251209 08:58:01.337101 22501 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251209 08:58:01.351830 22501 catalog_manager.cc:1380] Generated new certificate authority record
I20251209 08:58:01.353094 22501 catalog_manager.cc:1514] Loading token signing keys...
I20251209 08:58:01.368310 22501 catalog_manager.cc:6027] T 00000000000000000000000000000000 P 39341759984c48989ceba01b8acb2da9: Generated new TSK 0
I20251209 08:58:01.369091 22501 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.20.49.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.49.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.49.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765270668420570-20677-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.20.49.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765270668420570-20677-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.20.49.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.20.49.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.20.49.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765270668420570-20677-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.20.49.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765270668420570-20677-0/krb5kdc/kudu.keytab.
I20251209 08:58:01.433553 20677 external_mini_cluster.cc:1366] Running /tmp/dist-test-task8XuDZV/build/debug/bin/kudu
/tmp/dist-test-task8XuDZV/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765270668420570-20677-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765270668420570-20677-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765270668420570-20677-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765270668420570-20677-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.49.65:0
--local_ip_for_outbound_sockets=127.20.49.65
--webserver_interface=127.20.49.65
--webserver_port=0
--tserver_master_addrs=127.20.49.126:39233
--keytab_file=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765270668420570-20677-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.49.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.20.49.84:32905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765270668420570-20677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765270668420570-20677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765270668420570-20677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251209 08:58:01.550958 22515 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251209 08:58:01.551280 22515 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251209 08:58:01.551409 22515 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251209 08:58:01.555413 22515 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251209 08:58:01.555506 22515 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251209 08:58:01.555598 22515 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20251209 08:58:01.555629 22515 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20251209 08:58:01.555708 22515 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.49.65
I20251209 08:58:01.560415 22515 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.49.84:32905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765270668420570-20677-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765270668420570-20677-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765270668420570-20677-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.49.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.49.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765270668420570-20677-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.49.65
--webserver_port=0
--webserver_require_spnego=true
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.20.49.126:39233
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22515
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.49.65
--log_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765270668420570-20677-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 468897755969d686fb08386f42f7835685e7953a
build type DEBUG
built by None at 09 Dec 2025 08:43:15 UTC on 24a791456cd2
build id 9426
I20251209 08:58:01.561659 22515 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251209 08:58:01.562574 22515 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251209 08:58:01.569476 22523 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251209 08:58:01.569617 22521 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251209 08:58:01.569738 22520 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251209 08:58:01.569787 22515 server_base.cc:1047] running on GCE node
I20251209 08:58:01.570343 22515 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251209 08:58:01.570914 22515 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251209 08:58:01.572077 22515 hybrid_clock.cc:648] HybridClock initialized: now 1765270681572053 us; error 38 us; skew 500 ppm
Dec 09 08:58:01 dist-test-slave-8w6c krb5kdc[22393](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765270681, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.49.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251209 08:58:01.574895 22515 init.cc:377] Logged in from keytab as kudu/127.20.49.65@KRBTEST.COM (short username kudu)
I20251209 08:58:01.575996 22515 webserver.cc:492] Webserver started at http://127.20.49.65:43953/ using document root <none> and password file <none>
I20251209 08:58:01.576550 22515 fs_manager.cc:362] Metadata directory not provided
I20251209 08:58:01.576607 22515 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251209 08:58:01.576808 22515 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251209 08:58:01.578617 22515 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765270668420570-20677-0/minicluster-data/ts-0/data/instance:
uuid: "da42139421e54188b827cd6256e162b0"
format_stamp: "Formatted at 2025-12-09 08:58:01 on dist-test-slave-8w6c"
server_key: "f66ed7bd7ddfa9c1d40cfc94bcd7d0e4"
server_key_iv: "a854d6296bb81c97ffe8055ed340f3a6"
server_key_version: "encryptionkey@0"
I20251209 08:58:01.579098 22515 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765270668420570-20677-0/minicluster-data/ts-0/wal/instance:
uuid: "da42139421e54188b827cd6256e162b0"
format_stamp: "Formatted at 2025-12-09 08:58:01 on dist-test-slave-8w6c"
server_key: "f66ed7bd7ddfa9c1d40cfc94bcd7d0e4"
server_key_iv: "a854d6296bb81c97ffe8055ed340f3a6"
server_key_version: "encryptionkey@0"
I20251209 08:58:01.582818 22515 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20251209 08:58:01.585141 22530 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251209 08:58:01.586372 22515 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20251209 08:58:01.586491 22515 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765270668420570-20677-0/minicluster-data/ts-0/data,/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765270668420570-20677-0/minicluster-data/ts-0/wal
uuid: "da42139421e54188b827cd6256e162b0"
format_stamp: "Formatted at 2025-12-09 08:58:01 on dist-test-slave-8w6c"
server_key: "f66ed7bd7ddfa9c1d40cfc94bcd7d0e4"
server_key_iv: "a854d6296bb81c97ffe8055ed340f3a6"
server_key_version: "encryptionkey@0"
I20251209 08:58:01.586596 22515 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765270668420570-20677-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765270668420570-20677-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765270668420570-20677-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251209 08:58:01.622115 22515 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251209 08:58:01.629417 22515 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251209 08:58:01.629608 22515 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251209 08:58:01.631361 22515 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251209 08:58:01.631431 22515 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251209 08:58:01.631491 22515 ts_tablet_manager.cc:616] Registered 0 tablets
I20251209 08:58:01.631536 22515 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Dec 09 08:58:01 dist-test-slave-8w6c krb5kdc[22393](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765270681, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.49.65@KRBTEST.COM for kudu/127.20.49.126@KRBTEST.COM
I20251209 08:58:01.643808 22515 rpc_server.cc:307] RPC server started. Bound to: 127.20.49.65:46509
I20251209 08:58:01.643833 22645 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.49.65:46509 every 8 connection(s)
I20251209 08:58:01.645049 22515 server_base.cc:1179] Dumped server information to /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765270668420570-20677-0/minicluster-data/ts-0/data/info.pb
I20251209 08:58:01.646696 22539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:58:01.632590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.49.126:39233 (local address 127.20.49.65:50331)
1209 08:58:01.633265 (+ 675us) negotiation.cc:107] Waiting for socket to connect
1209 08:58:01.633310 (+ 45us) client_negotiation.cc:174] Beginning negotiation
1209 08:58:01.634200 (+ 890us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1209 08:58:01.634796 (+ 596us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1209 08:58:01.634808 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
1209 08:58:01.635442 (+ 634us) client_negotiation.cc:189] Negotiated authn=SASL
1209 08:58:01.636322 (+ 880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:58:01.636342 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:58:01.637769 (+ 1427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1209 08:58:01.637774 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1209 08:58:01.638502 (+ 728us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:58:01.638513 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:58:01.638776 (+ 263us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:58:01.639614 (+ 838us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1209 08:58:01.639660 (+ 46us) client_negotiation.cc:656] calling sasl_client_start()
1209 08:58:01.641789 (+ 2129us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1209 08:58:01.643990 (+ 2201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1209 08:58:01.643996 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1209 08:58:01.644008 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
1209 08:58:01.644313 (+ 305us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1209 08:58:01.644612 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1209 08:58:01.644616 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1209 08:58:01.644618 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1209 08:58:01.644750 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1209 08:58:01.645250 (+ 500us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1209 08:58:01.645257 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1209 08:58:01.645571 (+ 314us) client_negotiation.cc:769] Sending connection context
1209 08:58:01.645839 (+ 268us) client_negotiation.cc:240] Negotiation successful
1209 08:58:01.646085 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":379,"thread_start_us":160,"threads_started":1}
I20251209 08:58:01.647329 22490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:58:01.632211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.49.65:50331 (local address 127.20.49.126:39233)
1209 08:58:01.632361 (+ 150us) server_negotiation.cc:207] Beginning negotiation
1209 08:58:01.632365 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1209 08:58:01.633441 (+ 1076us) server_negotiation.cc:408] Connection header received
1209 08:58:01.634429 (+ 988us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1209 08:58:01.634439 (+ 10us) server_negotiation.cc:462] Received NEGOTIATE request from client
1209 08:58:01.634510 (+ 71us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1209 08:58:01.634591 (+ 81us) server_negotiation.cc:227] Negotiated authn=SASL
1209 08:58:01.636571 (+ 1980us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:58:01.637561 (+ 990us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1209 08:58:01.638656 (+ 1095us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:58:01.638849 (+ 193us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:58:01.641964 (+ 3115us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1209 08:58:01.641990 (+ 26us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1209 08:58:01.641992 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1209 08:58:01.642024 (+ 32us) server_negotiation.cc:923] calling sasl_server_start()
1209 08:58:01.643825 (+ 1801us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1209 08:58:01.644453 (+ 628us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1209 08:58:01.644456 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1209 08:58:01.644458 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1209 08:58:01.644510 (+ 52us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1209 08:58:01.644890 (+ 380us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1209 08:58:01.644895 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1209 08:58:01.644897 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1209 08:58:01.645132 (+ 235us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1209 08:58:01.645252 (+ 120us) server_negotiation.cc:1036] Waiting for connection context
1209 08:58:01.647057 (+ 1805us) server_negotiation.cc:300] Negotiation successful
1209 08:58:01.647217 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20251209 08:58:01.648397 22646 heartbeater.cc:344] Connected to a master server at 127.20.49.126:39233
I20251209 08:58:01.648564 22646 heartbeater.cc:461] Registering TS with master...
I20251209 08:58:01.649235 22646 heartbeater.cc:507] Master 127.20.49.126:39233 requested a full tablet report, sending...
I20251209 08:58:01.650880 22440 ts_manager.cc:194] Registered new tserver with Master: da42139421e54188b827cd6256e162b0 (127.20.49.65:46509)
I20251209 08:58:01.651216 20677 external_mini_cluster.cc:1428] Started /tmp/dist-test-task8XuDZV/build/debug/bin/kudu as pid 22515
I20251209 08:58:01.651333 20677 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765270668420570-20677-0/minicluster-data/ts-0/wal/instance
I20251209 08:58:01.651572 20677 external_mini_cluster.cc:1468] Setting key dc44fd9757f583ebfe26d6be96fdface
I20251209 08:58:01.652412 22440 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.49.65@KRBTEST.COM'} at 127.20.49.65:50331
I20251209 08:58:01.658855 22490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:58:01.650800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.49.65:56041 (local address 127.20.49.126:39233)
1209 08:58:01.650970 (+ 170us) server_negotiation.cc:207] Beginning negotiation
1209 08:58:01.650975 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1209 08:58:01.651045 (+ 70us) server_negotiation.cc:408] Connection header received
1209 08:58:01.651273 (+ 228us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1209 08:58:01.651276 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1209 08:58:01.651325 (+ 49us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1209 08:58:01.651426 (+ 101us) server_negotiation.cc:227] Negotiated authn=SASL
1209 08:58:01.652042 (+ 616us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:58:01.652681 (+ 639us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1209 08:58:01.653588 (+ 907us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:58:01.653844 (+ 256us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:58:01.655307 (+ 1463us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1209 08:58:01.655330 (+ 23us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1209 08:58:01.655337 (+ 7us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1209 08:58:01.655370 (+ 33us) server_negotiation.cc:923] calling sasl_server_start()
1209 08:58:01.656994 (+ 1624us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1209 08:58:01.657573 (+ 579us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1209 08:58:01.657576 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1209 08:58:01.657577 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1209 08:58:01.657624 (+ 47us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1209 08:58:01.658029 (+ 405us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1209 08:58:01.658031 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1209 08:58:01.658033 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1209 08:58:01.658238 (+ 205us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1209 08:58:01.658318 (+ 80us) server_negotiation.cc:1036] Waiting for connection context
1209 08:58:01.658636 (+ 318us) server_negotiation.cc:300] Negotiation successful
1209 08:58:01.658733 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20251209 08:58:01.659075 22652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:58:01.650661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.49.126:39233 (local address 127.20.49.65:56041)
1209 08:58:01.650954 (+ 293us) negotiation.cc:107] Waiting for socket to connect
1209 08:58:01.650970 (+ 16us) client_negotiation.cc:174] Beginning negotiation
1209 08:58:01.651088 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1209 08:58:01.651482 (+ 394us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1209 08:58:01.651486 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
1209 08:58:01.651675 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
1209 08:58:01.651887 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:58:01.651894 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:58:01.652826 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1209 08:58:01.652829 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1209 08:58:01.653418 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:58:01.653429 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:58:01.653983 (+ 554us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:58:01.654666 (+ 683us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1209 08:58:01.654688 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
1209 08:58:01.655156 (+ 468us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1209 08:58:01.657140 (+ 1984us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1209 08:58:01.657145 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1209 08:58:01.657147 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1209 08:58:01.657422 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1209 08:58:01.657835 (+ 413us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1209 08:58:01.657839 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1209 08:58:01.657842 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
1209 08:58:01.657938 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1209 08:58:01.658338 (+ 400us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1209 08:58:01.658343 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1209 08:58:01.658470 (+ 127us) client_negotiation.cc:769] Sending connection context
1209 08:58:01.658605 (+ 135us) client_negotiation.cc:240] Negotiation successful
1209 08:58:01.658777 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":102,"threads_started":1}
WARNING: no policy specified for kudu/127.20.49.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.49.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.49.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765270668420570-20677-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.20.49.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765270668420570-20677-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.20.49.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.20.49.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.20.49.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765270668420570-20677-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.20.49.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765270668420570-20677-0/krb5kdc/kudu.keytab.
I20251209 08:58:01.705199 20677 external_mini_cluster.cc:1366] Running /tmp/dist-test-task8XuDZV/build/debug/bin/kudu
/tmp/dist-test-task8XuDZV/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765270668420570-20677-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765270668420570-20677-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765270668420570-20677-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765270668420570-20677-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.49.66:0
--local_ip_for_outbound_sockets=127.20.49.66
--webserver_interface=127.20.49.66
--webserver_port=0
--tserver_master_addrs=127.20.49.126:39233
--keytab_file=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765270668420570-20677-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.49.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.20.49.84:32905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765270668420570-20677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765270668420570-20677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765270668420570-20677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251209 08:58:01.819250 22657 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251209 08:58:01.819523 22657 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251209 08:58:01.819633 22657 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251209 08:58:01.823328 22657 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251209 08:58:01.823414 22657 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251209 08:58:01.823509 22657 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20251209 08:58:01.823539 22657 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20251209 08:58:01.823606 22657 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.49.66
I20251209 08:58:01.828411 22657 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.49.84:32905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765270668420570-20677-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765270668420570-20677-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765270668420570-20677-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.49.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.49.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765270668420570-20677-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.49.66
--webserver_port=0
--webserver_require_spnego=true
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.20.49.126:39233
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22657
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.49.66
--log_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765270668420570-20677-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 468897755969d686fb08386f42f7835685e7953a
build type DEBUG
built by None at 09 Dec 2025 08:43:15 UTC on 24a791456cd2
build id 9426
I20251209 08:58:01.829595 22657 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251209 08:58:01.830530 22657 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251209 08:58:01.837793 22665 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251209 08:58:01.837779 22662 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251209 08:58:01.837842 22657 server_base.cc:1047] running on GCE node
W20251209 08:58:01.837779 22663 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251209 08:58:01.838439 22657 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251209 08:58:01.839027 22657 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251209 08:58:01.840267 22657 hybrid_clock.cc:648] HybridClock initialized: now 1765270681840245 us; error 37 us; skew 500 ppm
Dec 09 08:58:01 dist-test-slave-8w6c krb5kdc[22393](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765270681, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.49.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251209 08:58:01.843313 22657 init.cc:377] Logged in from keytab as kudu/127.20.49.66@KRBTEST.COM (short username kudu)
I20251209 08:58:01.844426 22657 webserver.cc:492] Webserver started at http://127.20.49.66:35937/ using document root <none> and password file <none>
I20251209 08:58:01.844986 22657 fs_manager.cc:362] Metadata directory not provided
I20251209 08:58:01.845041 22657 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251209 08:58:01.845237 22657 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251209 08:58:01.847081 22657 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765270668420570-20677-0/minicluster-data/ts-1/data/instance:
uuid: "15ab8f217086450b933f077d59555f07"
format_stamp: "Formatted at 2025-12-09 08:58:01 on dist-test-slave-8w6c"
server_key: "41351860d2e23c9f22eeeabe79915cfd"
server_key_iv: "d3f565ecb8e4a6bd6efed1d2aa40588c"
server_key_version: "encryptionkey@0"
I20251209 08:58:01.847554 22657 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765270668420570-20677-0/minicluster-data/ts-1/wal/instance:
uuid: "15ab8f217086450b933f077d59555f07"
format_stamp: "Formatted at 2025-12-09 08:58:01 on dist-test-slave-8w6c"
server_key: "41351860d2e23c9f22eeeabe79915cfd"
server_key_iv: "d3f565ecb8e4a6bd6efed1d2aa40588c"
server_key_version: "encryptionkey@0"
I20251209 08:58:01.851315 22657 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20251209 08:58:01.853883 22672 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251209 08:58:01.855023 22657 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20251209 08:58:01.855137 22657 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765270668420570-20677-0/minicluster-data/ts-1/data,/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765270668420570-20677-0/minicluster-data/ts-1/wal
uuid: "15ab8f217086450b933f077d59555f07"
format_stamp: "Formatted at 2025-12-09 08:58:01 on dist-test-slave-8w6c"
server_key: "41351860d2e23c9f22eeeabe79915cfd"
server_key_iv: "d3f565ecb8e4a6bd6efed1d2aa40588c"
server_key_version: "encryptionkey@0"
I20251209 08:58:01.855273 22657 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765270668420570-20677-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765270668420570-20677-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765270668420570-20677-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251209 08:58:01.890321 22657 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251209 08:58:01.893734 22657 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251209 08:58:01.893940 22657 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251209 08:58:01.895543 22657 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251209 08:58:01.895598 22657 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251209 08:58:01.895642 22657 ts_tablet_manager.cc:616] Registered 0 tablets
I20251209 08:58:01.895677 22657 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Dec 09 08:58:01 dist-test-slave-8w6c krb5kdc[22393](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765270681, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.49.66@KRBTEST.COM for kudu/127.20.49.126@KRBTEST.COM
I20251209 08:58:01.906988 22657 rpc_server.cc:307] RPC server started. Bound to: 127.20.49.66:45423
I20251209 08:58:01.907022 22787 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.49.66:45423 every 8 connection(s)
I20251209 08:58:01.908303 22657 server_base.cc:1179] Dumped server information to /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765270668420570-20677-0/minicluster-data/ts-1/data/info.pb
I20251209 08:58:01.909421 22490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:58:01.896289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.49.66:48845 (local address 127.20.49.126:39233)
1209 08:58:01.896455 (+ 166us) server_negotiation.cc:207] Beginning negotiation
1209 08:58:01.896463 (+ 8us) server_negotiation.cc:400] Waiting for connection header
1209 08:58:01.897649 (+ 1186us) server_negotiation.cc:408] Connection header received
1209 08:58:01.898674 (+ 1025us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1209 08:58:01.898679 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1209 08:58:01.898751 (+ 72us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1209 08:58:01.898865 (+ 114us) server_negotiation.cc:227] Negotiated authn=SASL
1209 08:58:01.900573 (+ 1708us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:58:01.901396 (+ 823us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1209 08:58:01.902425 (+ 1029us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:58:01.902685 (+ 260us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:58:01.905554 (+ 2869us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1209 08:58:01.905585 (+ 31us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1209 08:58:01.905589 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1209 08:58:01.905629 (+ 40us) server_negotiation.cc:923] calling sasl_server_start()
1209 08:58:01.907473 (+ 1844us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1209 08:58:01.907962 (+ 489us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1209 08:58:01.907968 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1209 08:58:01.907969 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1209 08:58:01.908026 (+ 57us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1209 08:58:01.908286 (+ 260us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1209 08:58:01.908289 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1209 08:58:01.908291 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1209 08:58:01.908510 (+ 219us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1209 08:58:01.908623 (+ 113us) server_negotiation.cc:1036] Waiting for connection context
1209 08:58:01.909176 (+ 553us) server_negotiation.cc:300] Negotiation successful
1209 08:58:01.909299 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20251209 08:58:01.910131 22681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:58:01.896592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.49.126:39233 (local address 127.20.49.66:48845)
1209 08:58:01.897483 (+ 891us) negotiation.cc:107] Waiting for socket to connect
1209 08:58:01.897528 (+ 45us) client_negotiation.cc:174] Beginning negotiation
1209 08:58:01.898476 (+ 948us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1209 08:58:01.899086 (+ 610us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1209 08:58:01.899099 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
1209 08:58:01.899629 (+ 530us) client_negotiation.cc:189] Negotiated authn=SASL
1209 08:58:01.900405 (+ 776us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:58:01.900424 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:58:01.901562 (+ 1138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1209 08:58:01.901568 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1209 08:58:01.902288 (+ 720us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:58:01.902300 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:58:01.902589 (+ 289us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:58:01.903356 (+ 767us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1209 08:58:01.903394 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
1209 08:58:01.905366 (+ 1972us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1209 08:58:01.907601 (+ 2235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1209 08:58:01.907606 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1209 08:58:01.907615 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
1209 08:58:01.907852 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1209 08:58:01.908118 (+ 266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1209 08:58:01.908121 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1209 08:58:01.908122 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1209 08:58:01.908196 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1209 08:58:01.908612 (+ 416us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1209 08:58:01.908617 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1209 08:58:01.908897 (+ 280us) client_negotiation.cc:769] Sending connection context
1209 08:58:01.909127 (+ 230us) client_negotiation.cc:240] Negotiation successful
1209 08:58:01.909397 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":691,"thread_start_us":559,"threads_started":1}
I20251209 08:58:01.911331 22788 heartbeater.cc:344] Connected to a master server at 127.20.49.126:39233
I20251209 08:58:01.911495 22788 heartbeater.cc:461] Registering TS with master...
I20251209 08:58:01.912098 22788 heartbeater.cc:507] Master 127.20.49.126:39233 requested a full tablet report, sending...
I20251209 08:58:01.912918 20677 external_mini_cluster.cc:1428] Started /tmp/dist-test-task8XuDZV/build/debug/bin/kudu as pid 22657
I20251209 08:58:01.913017 20677 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765270668420570-20677-0/minicluster-data/ts-1/wal/instance
I20251209 08:58:01.913280 20677 external_mini_cluster.cc:1468] Setting key 6b1f324af8c816b508c4c09453bb76d7
I20251209 08:58:01.913273 22440 ts_manager.cc:194] Registered new tserver with Master: 15ab8f217086450b933f077d59555f07 (127.20.49.66:45423)
I20251209 08:58:01.913983 22440 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.49.66@KRBTEST.COM'} at 127.20.49.66:48845
I20251209 08:58:01.921494 22490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:58:01.914373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.49.66:49777 (local address 127.20.49.126:39233)
1209 08:58:01.914485 (+ 112us) server_negotiation.cc:207] Beginning negotiation
1209 08:58:01.914490 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1209 08:58:01.914504 (+ 14us) server_negotiation.cc:408] Connection header received
1209 08:58:01.914583 (+ 79us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1209 08:58:01.914589 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
1209 08:58:01.914631 (+ 42us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1209 08:58:01.914711 (+ 80us) server_negotiation.cc:227] Negotiated authn=SASL
1209 08:58:01.915401 (+ 690us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:58:01.916093 (+ 692us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1209 08:58:01.916942 (+ 849us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:58:01.917166 (+ 224us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:58:01.918126 (+ 960us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1209 08:58:01.918146 (+ 20us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1209 08:58:01.918149 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1209 08:58:01.918174 (+ 25us) server_negotiation.cc:923] calling sasl_server_start()
1209 08:58:01.919829 (+ 1655us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1209 08:58:01.920349 (+ 520us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1209 08:58:01.920353 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1209 08:58:01.920354 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1209 08:58:01.920410 (+ 56us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1209 08:58:01.920677 (+ 267us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1209 08:58:01.920680 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1209 08:58:01.920681 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1209 08:58:01.920848 (+ 167us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1209 08:58:01.920927 (+ 79us) server_negotiation.cc:1036] Waiting for connection context
1209 08:58:01.921262 (+ 335us) server_negotiation.cc:300] Negotiation successful
1209 08:58:01.921365 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20251209 08:58:01.921548 22794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:58:01.913959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.49.126:39233 (local address 127.20.49.66:49777)
1209 08:58:01.914404 (+ 445us) negotiation.cc:107] Waiting for socket to connect
1209 08:58:01.914420 (+ 16us) client_negotiation.cc:174] Beginning negotiation
1209 08:58:01.914502 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1209 08:58:01.914743 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1209 08:58:01.914747 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
1209 08:58:01.914957 (+ 210us) client_negotiation.cc:189] Negotiated authn=SASL
1209 08:58:01.915249 (+ 292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:58:01.915262 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:58:01.916225 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1209 08:58:01.916232 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1209 08:58:01.916809 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:58:01.916818 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:58:01.916913 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:58:01.917498 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1209 08:58:01.917520 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
1209 08:58:01.917999 (+ 479us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1209 08:58:01.919960 (+ 1961us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1209 08:58:01.919968 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1209 08:58:01.919973 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
1209 08:58:01.920225 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1209 08:58:01.920512 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1209 08:58:01.920518 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1209 08:58:01.920522 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
1209 08:58:01.920591 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1209 08:58:01.920936 (+ 345us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1209 08:58:01.920941 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1209 08:58:01.921046 (+ 105us) client_negotiation.cc:769] Sending connection context
1209 08:58:01.921171 (+ 125us) client_negotiation.cc:240] Negotiation successful
1209 08:58:01.921321 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":356,"thread_start_us":153,"threads_started":1}
WARNING: no policy specified for kudu/127.20.49.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.49.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.49.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765270668420570-20677-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.20.49.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765270668420570-20677-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.20.49.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.20.49.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.20.49.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765270668420570-20677-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.20.49.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765270668420570-20677-0/krb5kdc/kudu.keytab.
I20251209 08:58:01.965287 20677 external_mini_cluster.cc:1366] Running /tmp/dist-test-task8XuDZV/build/debug/bin/kudu
/tmp/dist-test-task8XuDZV/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765270668420570-20677-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765270668420570-20677-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765270668420570-20677-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765270668420570-20677-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.49.67:0
--local_ip_for_outbound_sockets=127.20.49.67
--webserver_interface=127.20.49.67
--webserver_port=0
--tserver_master_addrs=127.20.49.126:39233
--keytab_file=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765270668420570-20677-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.49.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.20.49.84:32905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765270668420570-20677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765270668420570-20677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765270668420570-20677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251209 08:58:02.079165 22799 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251209 08:58:02.079421 22799 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251209 08:58:02.079521 22799 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251209 08:58:02.082978 22799 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251209 08:58:02.083060 22799 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251209 08:58:02.083144 22799 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20251209 08:58:02.083180 22799 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20251209 08:58:02.083266 22799 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.49.67
I20251209 08:58:02.087930 22799 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.49.84:32905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765270668420570-20677-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765270668420570-20677-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765270668420570-20677-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.49.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.49.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765270668420570-20677-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.49.67
--webserver_port=0
--webserver_require_spnego=true
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.20.49.126:39233
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22799
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.49.67
--log_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765270668420570-20677-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 468897755969d686fb08386f42f7835685e7953a
build type DEBUG
built by None at 09 Dec 2025 08:43:15 UTC on 24a791456cd2
build id 9426
I20251209 08:58:02.089063 22799 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251209 08:58:02.089982 22799 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251209 08:58:02.096957 22805 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251209 08:58:02.097128 22804 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251209 08:58:02.097128 22807 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251209 08:58:02.097421 22799 server_base.cc:1047] running on GCE node
I20251209 08:58:02.097913 22799 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251209 08:58:02.098558 22799 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251209 08:58:02.099735 22799 hybrid_clock.cc:648] HybridClock initialized: now 1765270682099692 us; error 59 us; skew 500 ppm
Dec 09 08:58:02 dist-test-slave-8w6c krb5kdc[22393](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765270682, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.49.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251209 08:58:02.103015 22799 init.cc:377] Logged in from keytab as kudu/127.20.49.67@KRBTEST.COM (short username kudu)
I20251209 08:58:02.104146 22799 webserver.cc:492] Webserver started at http://127.20.49.67:44933/ using document root <none> and password file <none>
I20251209 08:58:02.104707 22799 fs_manager.cc:362] Metadata directory not provided
I20251209 08:58:02.104755 22799 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251209 08:58:02.104918 22799 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251209 08:58:02.106770 22799 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765270668420570-20677-0/minicluster-data/ts-2/data/instance:
uuid: "f2190f825aa04a0cb6b89cac0e770d12"
format_stamp: "Formatted at 2025-12-09 08:58:02 on dist-test-slave-8w6c"
server_key: "aaa3279c4db206b594aaff2238c987d5"
server_key_iv: "5ea8741a4af7ea711f834a6002cd07a1"
server_key_version: "encryptionkey@0"
I20251209 08:58:02.107244 22799 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765270668420570-20677-0/minicluster-data/ts-2/wal/instance:
uuid: "f2190f825aa04a0cb6b89cac0e770d12"
format_stamp: "Formatted at 2025-12-09 08:58:02 on dist-test-slave-8w6c"
server_key: "aaa3279c4db206b594aaff2238c987d5"
server_key_iv: "5ea8741a4af7ea711f834a6002cd07a1"
server_key_version: "encryptionkey@0"
I20251209 08:58:02.110901 22799 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.001s sys 0.003s
I20251209 08:58:02.113461 22814 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251209 08:58:02.114645 22799 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20251209 08:58:02.114768 22799 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765270668420570-20677-0/minicluster-data/ts-2/data,/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765270668420570-20677-0/minicluster-data/ts-2/wal
uuid: "f2190f825aa04a0cb6b89cac0e770d12"
format_stamp: "Formatted at 2025-12-09 08:58:02 on dist-test-slave-8w6c"
server_key: "aaa3279c4db206b594aaff2238c987d5"
server_key_iv: "5ea8741a4af7ea711f834a6002cd07a1"
server_key_version: "encryptionkey@0"
I20251209 08:58:02.114882 22799 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765270668420570-20677-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765270668420570-20677-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765270668420570-20677-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251209 08:58:02.140077 22799 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251209 08:58:02.143440 22799 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251209 08:58:02.143638 22799 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251209 08:58:02.145118 22799 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251209 08:58:02.145176 22799 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251209 08:58:02.145246 22799 ts_tablet_manager.cc:616] Registered 0 tablets
I20251209 08:58:02.145278 22799 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Dec 09 08:58:02 dist-test-slave-8w6c krb5kdc[22393](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765270682, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.49.67@KRBTEST.COM for kudu/127.20.49.126@KRBTEST.COM
I20251209 08:58:02.155930 22799 rpc_server.cc:307] RPC server started. Bound to: 127.20.49.67:42299
I20251209 08:58:02.155998 22929 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.49.67:42299 every 8 connection(s)
I20251209 08:58:02.156889 22799 server_base.cc:1179] Dumped server information to /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765270668420570-20677-0/minicluster-data/ts-2/data/info.pb
I20251209 08:58:02.159230 22490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:58:02.145934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.49.67:54625 (local address 127.20.49.126:39233)
1209 08:58:02.146065 (+ 131us) server_negotiation.cc:207] Beginning negotiation
1209 08:58:02.146086 (+ 21us) server_negotiation.cc:400] Waiting for connection header
1209 08:58:02.146992 (+ 906us) server_negotiation.cc:408] Connection header received
1209 08:58:02.147956 (+ 964us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1209 08:58:02.147960 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1209 08:58:02.148022 (+ 62us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1209 08:58:02.148108 (+ 86us) server_negotiation.cc:227] Negotiated authn=SASL
1209 08:58:02.150096 (+ 1988us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:58:02.150982 (+ 886us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1209 08:58:02.152027 (+ 1045us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:58:02.152301 (+ 274us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:58:02.155407 (+ 3106us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1209 08:58:02.155430 (+ 23us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1209 08:58:02.155432 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1209 08:58:02.155461 (+ 29us) server_negotiation.cc:923] calling sasl_server_start()
1209 08:58:02.157155 (+ 1694us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1209 08:58:02.157806 (+ 651us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1209 08:58:02.157810 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1209 08:58:02.157812 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1209 08:58:02.157872 (+ 60us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1209 08:58:02.158211 (+ 339us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1209 08:58:02.158216 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1209 08:58:02.158218 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1209 08:58:02.158406 (+ 188us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1209 08:58:02.158516 (+ 110us) server_negotiation.cc:1036] Waiting for connection context
1209 08:58:02.158992 (+ 476us) server_negotiation.cc:300] Negotiation successful
1209 08:58:02.159107 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20251209 08:58:02.159991 22823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:58:02.146297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.49.126:39233 (local address 127.20.49.67:54625)
1209 08:58:02.146826 (+ 529us) negotiation.cc:107] Waiting for socket to connect
1209 08:58:02.146874 (+ 48us) client_negotiation.cc:174] Beginning negotiation
1209 08:58:02.147726 (+ 852us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1209 08:58:02.148295 (+ 569us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1209 08:58:02.148308 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
1209 08:58:02.149047 (+ 739us) client_negotiation.cc:189] Negotiated authn=SASL
1209 08:58:02.149880 (+ 833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:58:02.149901 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:58:02.151167 (+ 1266us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1209 08:58:02.151174 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1209 08:58:02.151871 (+ 697us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:58:02.151886 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:58:02.152144 (+ 258us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:58:02.152863 (+ 719us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1209 08:58:02.152887 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
1209 08:58:02.155241 (+ 2354us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1209 08:58:02.157304 (+ 2063us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1209 08:58:02.157309 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1209 08:58:02.157320 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
1209 08:58:02.157618 (+ 298us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1209 08:58:02.157982 (+ 364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1209 08:58:02.157985 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1209 08:58:02.157986 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1209 08:58:02.158071 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1209 08:58:02.158551 (+ 480us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1209 08:58:02.158556 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1209 08:58:02.158779 (+ 223us) client_negotiation.cc:769] Sending connection context
1209 08:58:02.158952 (+ 173us) client_negotiation.cc:240] Negotiation successful
1209 08:58:02.159259 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":349,"thread_start_us":201,"threads_started":1}
I20251209 08:58:02.161167 22930 heartbeater.cc:344] Connected to a master server at 127.20.49.126:39233
I20251209 08:58:02.161345 22930 heartbeater.cc:461] Registering TS with master...
I20251209 08:58:02.161898 22930 heartbeater.cc:507] Master 127.20.49.126:39233 requested a full tablet report, sending...
I20251209 08:58:02.162106 20677 external_mini_cluster.cc:1428] Started /tmp/dist-test-task8XuDZV/build/debug/bin/kudu as pid 22799
I20251209 08:58:02.162259 20677 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765270668420570-20677-0/minicluster-data/ts-2/wal/instance
I20251209 08:58:02.162555 20677 external_mini_cluster.cc:1468] Setting key 80890db667982c9fbe80d50812e3adff
I20251209 08:58:02.162971 22440 ts_manager.cc:194] Registered new tserver with Master: f2190f825aa04a0cb6b89cac0e770d12 (127.20.49.67:42299)
I20251209 08:58:02.163658 22440 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.49.67@KRBTEST.COM'} at 127.20.49.67:54625
I20251209 08:58:02.164558 20677 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20251209 08:58:02.170676 22490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:58:02.163627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.49.67:60375 (local address 127.20.49.126:39233)
1209 08:58:02.163749 (+ 122us) server_negotiation.cc:207] Beginning negotiation
1209 08:58:02.163753 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1209 08:58:02.163987 (+ 234us) server_negotiation.cc:408] Connection header received
1209 08:58:02.164159 (+ 172us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1209 08:58:02.164163 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1209 08:58:02.164218 (+ 55us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1209 08:58:02.164301 (+ 83us) server_negotiation.cc:227] Negotiated authn=SASL
1209 08:58:02.164914 (+ 613us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:58:02.165401 (+ 487us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1209 08:58:02.166072 (+ 671us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:58:02.166297 (+ 225us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:58:02.167325 (+ 1028us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1209 08:58:02.167340 (+ 15us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1209 08:58:02.167342 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1209 08:58:02.167366 (+ 24us) server_negotiation.cc:923] calling sasl_server_start()
1209 08:58:02.169246 (+ 1880us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1209 08:58:02.169647 (+ 401us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1209 08:58:02.169651 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1209 08:58:02.169652 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1209 08:58:02.169728 (+ 76us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1209 08:58:02.169952 (+ 224us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1209 08:58:02.169955 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1209 08:58:02.169957 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1209 08:58:02.170095 (+ 138us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1209 08:58:02.170184 (+ 89us) server_negotiation.cc:1036] Waiting for connection context
1209 08:58:02.170421 (+ 237us) server_negotiation.cc:300] Negotiation successful
1209 08:58:02.170527 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20251209 08:58:02.170694 22936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:58:02.163523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.49.126:39233 (local address 127.20.49.67:60375)
1209 08:58:02.163900 (+ 377us) negotiation.cc:107] Waiting for socket to connect
1209 08:58:02.163915 (+ 15us) client_negotiation.cc:174] Beginning negotiation
1209 08:58:02.164051 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1209 08:58:02.164340 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1209 08:58:02.164345 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
1209 08:58:02.164555 (+ 210us) client_negotiation.cc:189] Negotiated authn=SASL
1209 08:58:02.164801 (+ 246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:58:02.164808 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:58:02.165512 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1209 08:58:02.165516 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1209 08:58:02.165961 (+ 445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:58:02.165969 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:58:02.166080 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:58:02.166703 (+ 623us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1209 08:58:02.166742 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
1209 08:58:02.167200 (+ 458us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1209 08:58:02.169355 (+ 2155us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1209 08:58:02.169358 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1209 08:58:02.169360 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1209 08:58:02.169551 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1209 08:58:02.169824 (+ 273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1209 08:58:02.169827 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1209 08:58:02.169828 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1209 08:58:02.169872 (+ 44us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1209 08:58:02.170183 (+ 311us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1209 08:58:02.170186 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1209 08:58:02.170270 (+ 84us) client_negotiation.cc:769] Sending connection context
1209 08:58:02.170361 (+ 91us) client_negotiation.cc:240] Negotiation successful
1209 08:58:02.170509 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":299,"thread_start_us":87,"threads_started":1}
Dec 09 08:58:02 dist-test-slave-8w6c krb5kdc[22393](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765270682, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Dec 09 08:58:02 dist-test-slave-8w6c krb5kdc[22393](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765270682, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.49.126@KRBTEST.COM
I20251209 08:58:02.191035 22490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:58:02.183080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42090 (local address 127.20.49.126:39233)
1209 08:58:02.183277 (+ 197us) server_negotiation.cc:207] Beginning negotiation
1209 08:58:02.183282 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1209 08:58:02.183302 (+ 20us) server_negotiation.cc:408] Connection header received
1209 08:58:02.183349 (+ 47us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1209 08:58:02.183351 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1209 08:58:02.183394 (+ 43us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1209 08:58:02.183495 (+ 101us) server_negotiation.cc:227] Negotiated authn=SASL
1209 08:58:02.184355 (+ 860us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:58:02.184852 (+ 497us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1209 08:58:02.185497 (+ 645us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:58:02.185746 (+ 249us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:58:02.187881 (+ 2135us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1209 08:58:02.187901 (+ 20us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1209 08:58:02.187903 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1209 08:58:02.187930 (+ 27us) server_negotiation.cc:923] calling sasl_server_start()
1209 08:58:02.189409 (+ 1479us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1209 08:58:02.189867 (+ 458us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1209 08:58:02.189873 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1209 08:58:02.189875 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1209 08:58:02.189929 (+ 54us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1209 08:58:02.190153 (+ 224us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1209 08:58:02.190157 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1209 08:58:02.190158 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1209 08:58:02.190370 (+ 212us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1209 08:58:02.190511 (+ 141us) server_negotiation.cc:1036] Waiting for connection context
1209 08:58:02.190742 (+ 231us) server_negotiation.cc:300] Negotiation successful
1209 08:58:02.190868 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20251209 08:58:02.194200 22441 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42090:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251209 08:58:02.196677 22441 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251209 08:58:02.210888 22947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:58:02.205898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.49.65:46509 (local address 127.0.0.1:44186)
1209 08:58:02.206429 (+ 531us) negotiation.cc:107] Waiting for socket to connect
1209 08:58:02.206473 (+ 44us) client_negotiation.cc:174] Beginning negotiation
1209 08:58:02.206627 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1209 08:58:02.207437 (+ 810us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1209 08:58:02.207444 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
1209 08:58:02.207573 (+ 129us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1209 08:58:02.207927 (+ 354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:58:02.207937 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:58:02.209438 (+ 1501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1209 08:58:02.209441 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1209 08:58:02.210329 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:58:02.210339 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:58:02.210438 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:58:02.210492 (+ 54us) client_negotiation.cc:769] Sending connection context
1209 08:58:02.210591 (+ 99us) client_negotiation.cc:240] Negotiation successful
1209 08:58:02.210696 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":90,"threads_started":1}
I20251209 08:58:02.211115 22950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:58:02.206370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.49.66:45423 (local address 127.0.0.1:33058)
1209 08:58:02.206882 (+ 512us) negotiation.cc:107] Waiting for socket to connect
1209 08:58:02.206893 (+ 11us) client_negotiation.cc:174] Beginning negotiation
1209 08:58:02.206979 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1209 08:58:02.208240 (+ 1261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1209 08:58:02.208243 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1209 08:58:02.208261 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1209 08:58:02.208495 (+ 234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:58:02.208505 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:58:02.209856 (+ 1351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1209 08:58:02.209862 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1209 08:58:02.210736 (+ 874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:58:02.210745 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:58:02.210855 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:58:02.210872 (+ 17us) client_negotiation.cc:769] Sending connection context
1209 08:58:02.210922 (+ 50us) client_negotiation.cc:240] Negotiation successful
1209 08:58:02.210981 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":430,"thread_start_us":109,"threads_started":1}
I20251209 08:58:02.211385 22949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:58:02.206623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.49.67:42299 (local address 127.0.0.1:57140)
1209 08:58:02.207285 (+ 662us) negotiation.cc:107] Waiting for socket to connect
1209 08:58:02.207301 (+ 16us) client_negotiation.cc:174] Beginning negotiation
1209 08:58:02.207389 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1209 08:58:02.207955 (+ 566us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1209 08:58:02.207958 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1209 08:58:02.207976 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1209 08:58:02.208241 (+ 265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:58:02.208248 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:58:02.209310 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1209 08:58:02.209313 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1209 08:58:02.210284 (+ 971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:58:02.210294 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:58:02.210426 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:58:02.210492 (+ 66us) client_negotiation.cc:769] Sending connection context
1209 08:58:02.210591 (+ 99us) client_negotiation.cc:240] Negotiation successful
1209 08:58:02.210696 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":518,"thread_start_us":120,"threads_started":1}
I20251209 08:58:02.211555 22948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:58:02.206051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44186 (local address 127.20.49.65:46509)
1209 08:58:02.206653 (+ 602us) server_negotiation.cc:207] Beginning negotiation
1209 08:58:02.206661 (+ 8us) server_negotiation.cc:400] Waiting for connection header
1209 08:58:02.206690 (+ 29us) server_negotiation.cc:408] Connection header received
1209 08:58:02.207040 (+ 350us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1209 08:58:02.207048 (+ 8us) server_negotiation.cc:462] Received NEGOTIATE request from client
1209 08:58:02.207255 (+ 207us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1209 08:58:02.208092 (+ 837us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1209 08:58:02.208250 (+ 158us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:58:02.209311 (+ 1061us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1209 08:58:02.210532 (+ 1221us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:58:02.211089 (+ 557us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:58:02.211225 (+ 136us) server_negotiation.cc:1036] Waiting for connection context
1209 08:58:02.211299 (+ 74us) server_negotiation.cc:300] Negotiation successful
1209 08:58:02.211419 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":493,"thread_start_us":112,"threads_started":1}
I20251209 08:58:02.212479 22952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:58:02.207189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33058 (local address 127.20.49.66:45423)
1209 08:58:02.207705 (+ 516us) server_negotiation.cc:207] Beginning negotiation
1209 08:58:02.207715 (+ 10us) server_negotiation.cc:400] Waiting for connection header
1209 08:58:02.207740 (+ 25us) server_negotiation.cc:408] Connection header received
1209 08:58:02.207823 (+ 83us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1209 08:58:02.207828 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1209 08:58:02.208010 (+ 182us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1209 08:58:02.208259 (+ 249us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1209 08:58:02.208633 (+ 374us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:58:02.209673 (+ 1040us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1209 08:58:02.211434 (+ 1761us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:58:02.211934 (+ 500us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:58:02.212024 (+ 90us) server_negotiation.cc:1036] Waiting for connection context
1209 08:58:02.212075 (+ 51us) server_negotiation.cc:300] Negotiation successful
1209 08:58:02.212141 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":414,"thread_start_us":100,"threads_started":1}
I20251209 08:58:02.214931 22580 tablet_service.cc:1505] Processing CreateTablet for tablet e98c48f6c144444c97e5450ffe99615d (DEFAULT_TABLE table=test-table [id=fd588d506ef541619d07881a13274efb]), partition=RANGE (key) PARTITION UNBOUNDED
I20251209 08:58:02.215282 22722 tablet_service.cc:1505] Processing CreateTablet for tablet e98c48f6c144444c97e5450ffe99615d (DEFAULT_TABLE table=test-table [id=fd588d506ef541619d07881a13274efb]), partition=RANGE (key) PARTITION UNBOUNDED
I20251209 08:58:02.216102 22580 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e98c48f6c144444c97e5450ffe99615d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251209 08:58:02.216101 22722 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e98c48f6c144444c97e5450ffe99615d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251209 08:58:02.217504 22951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:58:02.207033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57140 (local address 127.20.49.67:42299)
1209 08:58:02.207589 (+ 556us) server_negotiation.cc:207] Beginning negotiation
1209 08:58:02.207594 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1209 08:58:02.207615 (+ 21us) server_negotiation.cc:408] Connection header received
1209 08:58:02.207685 (+ 70us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1209 08:58:02.207689 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1209 08:58:02.207821 (+ 132us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1209 08:58:02.207956 (+ 135us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1209 08:58:02.208390 (+ 434us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:58:02.209192 (+ 802us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1209 08:58:02.210791 (+ 1599us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:58:02.212087 (+ 1296us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:58:02.212260 (+ 173us) server_negotiation.cc:1036] Waiting for connection context
1209 08:58:02.212329 (+ 69us) server_negotiation.cc:300] Negotiation successful
1209 08:58:02.212420 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":437,"thread_start_us":106,"threads_started":1}
I20251209 08:58:02.220369 22864 tablet_service.cc:1505] Processing CreateTablet for tablet e98c48f6c144444c97e5450ffe99615d (DEFAULT_TABLE table=test-table [id=fd588d506ef541619d07881a13274efb]), partition=RANGE (key) PARTITION UNBOUNDED
I20251209 08:58:02.221239 22864 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e98c48f6c144444c97e5450ffe99615d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251209 08:58:02.223374 22953 tablet_bootstrap.cc:492] T e98c48f6c144444c97e5450ffe99615d P da42139421e54188b827cd6256e162b0: Bootstrap starting.
I20251209 08:58:02.223794 22954 tablet_bootstrap.cc:492] T e98c48f6c144444c97e5450ffe99615d P 15ab8f217086450b933f077d59555f07: Bootstrap starting.
I20251209 08:58:02.226361 22953 tablet_bootstrap.cc:654] T e98c48f6c144444c97e5450ffe99615d P da42139421e54188b827cd6256e162b0: Neither blocks nor log segments found. Creating new log.
I20251209 08:58:02.227314 22953 log.cc:826] T e98c48f6c144444c97e5450ffe99615d P da42139421e54188b827cd6256e162b0: Log is configured to *not* fsync() on all Append() calls
I20251209 08:58:02.227828 22954 tablet_bootstrap.cc:654] T e98c48f6c144444c97e5450ffe99615d P 15ab8f217086450b933f077d59555f07: Neither blocks nor log segments found. Creating new log.
I20251209 08:58:02.229099 22954 log.cc:826] T e98c48f6c144444c97e5450ffe99615d P 15ab8f217086450b933f077d59555f07: Log is configured to *not* fsync() on all Append() calls
I20251209 08:58:02.229796 22953 tablet_bootstrap.cc:492] T e98c48f6c144444c97e5450ffe99615d P da42139421e54188b827cd6256e162b0: No bootstrap required, opened a new log
I20251209 08:58:02.229982 22953 ts_tablet_manager.cc:1403] T e98c48f6c144444c97e5450ffe99615d P da42139421e54188b827cd6256e162b0: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20251209 08:58:02.230726 22954 tablet_bootstrap.cc:492] T e98c48f6c144444c97e5450ffe99615d P 15ab8f217086450b933f077d59555f07: No bootstrap required, opened a new log
I20251209 08:58:02.230891 22954 ts_tablet_manager.cc:1403] T e98c48f6c144444c97e5450ffe99615d P 15ab8f217086450b933f077d59555f07: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20251209 08:58:02.232432 22953 raft_consensus.cc:359] T e98c48f6c144444c97e5450ffe99615d P da42139421e54188b827cd6256e162b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da42139421e54188b827cd6256e162b0" member_type: VOTER last_known_addr { host: "127.20.49.65" port: 46509 } } peers { permanent_uuid: "f2190f825aa04a0cb6b89cac0e770d12" member_type: VOTER last_known_addr { host: "127.20.49.67" port: 42299 } } peers { permanent_uuid: "15ab8f217086450b933f077d59555f07" member_type: VOTER last_known_addr { host: "127.20.49.66" port: 45423 } }
I20251209 08:58:02.232625 22953 raft_consensus.cc:385] T e98c48f6c144444c97e5450ffe99615d P da42139421e54188b827cd6256e162b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251209 08:58:02.232674 22953 raft_consensus.cc:740] T e98c48f6c144444c97e5450ffe99615d P da42139421e54188b827cd6256e162b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: da42139421e54188b827cd6256e162b0, State: Initialized, Role: FOLLOWER
I20251209 08:58:02.233081 22953 consensus_queue.cc:260] T e98c48f6c144444c97e5450ffe99615d P da42139421e54188b827cd6256e162b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da42139421e54188b827cd6256e162b0" member_type: VOTER last_known_addr { host: "127.20.49.65" port: 46509 } } peers { permanent_uuid: "f2190f825aa04a0cb6b89cac0e770d12" member_type: VOTER last_known_addr { host: "127.20.49.67" port: 42299 } } peers { permanent_uuid: "15ab8f217086450b933f077d59555f07" member_type: VOTER last_known_addr { host: "127.20.49.66" port: 45423 } }
I20251209 08:58:02.233309 22954 raft_consensus.cc:359] T e98c48f6c144444c97e5450ffe99615d P 15ab8f217086450b933f077d59555f07 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da42139421e54188b827cd6256e162b0" member_type: VOTER last_known_addr { host: "127.20.49.65" port: 46509 } } peers { permanent_uuid: "f2190f825aa04a0cb6b89cac0e770d12" member_type: VOTER last_known_addr { host: "127.20.49.67" port: 42299 } } peers { permanent_uuid: "15ab8f217086450b933f077d59555f07" member_type: VOTER last_known_addr { host: "127.20.49.66" port: 45423 } }
I20251209 08:58:02.233495 22954 raft_consensus.cc:385] T e98c48f6c144444c97e5450ffe99615d P 15ab8f217086450b933f077d59555f07 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251209 08:58:02.233556 22954 raft_consensus.cc:740] T e98c48f6c144444c97e5450ffe99615d P 15ab8f217086450b933f077d59555f07 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 15ab8f217086450b933f077d59555f07, State: Initialized, Role: FOLLOWER
I20251209 08:58:02.233798 22646 heartbeater.cc:499] Master 127.20.49.126:39233 was elected leader, sending a full tablet report...
I20251209 08:58:02.234009 22953 ts_tablet_manager.cc:1434] T e98c48f6c144444c97e5450ffe99615d P da42139421e54188b827cd6256e162b0: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20251209 08:58:02.234421 22956 tablet_bootstrap.cc:492] T e98c48f6c144444c97e5450ffe99615d P f2190f825aa04a0cb6b89cac0e770d12: Bootstrap starting.
I20251209 08:58:02.235033 22954 consensus_queue.cc:260] T e98c48f6c144444c97e5450ffe99615d P 15ab8f217086450b933f077d59555f07 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da42139421e54188b827cd6256e162b0" member_type: VOTER last_known_addr { host: "127.20.49.65" port: 46509 } } peers { permanent_uuid: "f2190f825aa04a0cb6b89cac0e770d12" member_type: VOTER last_known_addr { host: "127.20.49.67" port: 42299 } } peers { permanent_uuid: "15ab8f217086450b933f077d59555f07" member_type: VOTER last_known_addr { host: "127.20.49.66" port: 45423 } }
I20251209 08:58:02.235911 22954 ts_tablet_manager.cc:1434] T e98c48f6c144444c97e5450ffe99615d P 15ab8f217086450b933f077d59555f07: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20251209 08:58:02.236294 22788 heartbeater.cc:499] Master 127.20.49.126:39233 was elected leader, sending a full tablet report...
I20251209 08:58:02.236840 22956 tablet_bootstrap.cc:654] T e98c48f6c144444c97e5450ffe99615d P f2190f825aa04a0cb6b89cac0e770d12: Neither blocks nor log segments found. Creating new log.
I20251209 08:58:02.238241 22956 log.cc:826] T e98c48f6c144444c97e5450ffe99615d P f2190f825aa04a0cb6b89cac0e770d12: Log is configured to *not* fsync() on all Append() calls
I20251209 08:58:02.240268 22956 tablet_bootstrap.cc:492] T e98c48f6c144444c97e5450ffe99615d P f2190f825aa04a0cb6b89cac0e770d12: No bootstrap required, opened a new log
I20251209 08:58:02.240461 22956 ts_tablet_manager.cc:1403] T e98c48f6c144444c97e5450ffe99615d P f2190f825aa04a0cb6b89cac0e770d12: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20251209 08:58:02.242887 22956 raft_consensus.cc:359] T e98c48f6c144444c97e5450ffe99615d P f2190f825aa04a0cb6b89cac0e770d12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da42139421e54188b827cd6256e162b0" member_type: VOTER last_known_addr { host: "127.20.49.65" port: 46509 } } peers { permanent_uuid: "f2190f825aa04a0cb6b89cac0e770d12" member_type: VOTER last_known_addr { host: "127.20.49.67" port: 42299 } } peers { permanent_uuid: "15ab8f217086450b933f077d59555f07" member_type: VOTER last_known_addr { host: "127.20.49.66" port: 45423 } }
I20251209 08:58:02.243081 22956 raft_consensus.cc:385] T e98c48f6c144444c97e5450ffe99615d P f2190f825aa04a0cb6b89cac0e770d12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251209 08:58:02.243120 22956 raft_consensus.cc:740] T e98c48f6c144444c97e5450ffe99615d P f2190f825aa04a0cb6b89cac0e770d12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f2190f825aa04a0cb6b89cac0e770d12, State: Initialized, Role: FOLLOWER
I20251209 08:58:02.243531 22956 consensus_queue.cc:260] T e98c48f6c144444c97e5450ffe99615d P f2190f825aa04a0cb6b89cac0e770d12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da42139421e54188b827cd6256e162b0" member_type: VOTER last_known_addr { host: "127.20.49.65" port: 46509 } } peers { permanent_uuid: "f2190f825aa04a0cb6b89cac0e770d12" member_type: VOTER last_known_addr { host: "127.20.49.67" port: 42299 } } peers { permanent_uuid: "15ab8f217086450b933f077d59555f07" member_type: VOTER last_known_addr { host: "127.20.49.66" port: 45423 } }
I20251209 08:58:02.244194 22930 heartbeater.cc:499] Master 127.20.49.126:39233 was elected leader, sending a full tablet report...
I20251209 08:58:02.244405 22956 ts_tablet_manager.cc:1434] T e98c48f6c144444c97e5450ffe99615d P f2190f825aa04a0cb6b89cac0e770d12: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
W20251209 08:58:02.396513 22647 tablet.cc:2378] T e98c48f6c144444c97e5450ffe99615d P da42139421e54188b827cd6256e162b0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20251209 08:58:02.400501 22959 raft_consensus.cc:493] T e98c48f6c144444c97e5450ffe99615d P 15ab8f217086450b933f077d59555f07 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251209 08:58:02.400682 22959 raft_consensus.cc:515] T e98c48f6c144444c97e5450ffe99615d P 15ab8f217086450b933f077d59555f07 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da42139421e54188b827cd6256e162b0" member_type: VOTER last_known_addr { host: "127.20.49.65" port: 46509 } } peers { permanent_uuid: "f2190f825aa04a0cb6b89cac0e770d12" member_type: VOTER last_known_addr { host: "127.20.49.67" port: 42299 } } peers { permanent_uuid: "15ab8f217086450b933f077d59555f07" member_type: VOTER last_known_addr { host: "127.20.49.66" port: 45423 } }
I20251209 08:58:02.401901 22959 leader_election.cc:290] T e98c48f6c144444c97e5450ffe99615d P 15ab8f217086450b933f077d59555f07 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers da42139421e54188b827cd6256e162b0 (127.20.49.65:46509), f2190f825aa04a0cb6b89cac0e770d12 (127.20.49.67:42299)
I20251209 08:58:02.405236 22681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:58:02.402146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.49.65:46509 (local address 127.20.49.66:52029)
1209 08:58:02.402325 (+ 179us) negotiation.cc:107] Waiting for socket to connect
1209 08:58:02.402339 (+ 14us) client_negotiation.cc:174] Beginning negotiation
1209 08:58:02.402471 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1209 08:58:02.402803 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1209 08:58:02.402806 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1209 08:58:02.402826 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1209 08:58:02.403056 (+ 230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:58:02.403062 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:58:02.404208 (+ 1146us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1209 08:58:02.404214 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1209 08:58:02.404805 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:58:02.404812 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:58:02.404925 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:58:02.404948 (+ 23us) client_negotiation.cc:769] Sending connection context
1209 08:58:02.404999 (+ 51us) client_negotiation.cc:240] Negotiation successful
1209 08:58:02.405058 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":64,"spinlock_wait_cycles":1408}
I20251209 08:58:02.405397 22962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:58:02.402458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.49.67:42299 (local address 127.20.49.66:49297)
1209 08:58:02.402736 (+ 278us) negotiation.cc:107] Waiting for socket to connect
1209 08:58:02.402751 (+ 15us) client_negotiation.cc:174] Beginning negotiation
1209 08:58:02.402882 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1209 08:58:02.403122 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1209 08:58:02.403126 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
1209 08:58:02.403142 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1209 08:58:02.403463 (+ 321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:58:02.403473 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:58:02.404254 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1209 08:58:02.404258 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1209 08:58:02.404826 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:58:02.404834 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:58:02.404949 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:58:02.404969 (+ 20us) client_negotiation.cc:769] Sending connection context
1209 08:58:02.405021 (+ 52us) client_negotiation.cc:240] Negotiation successful
1209 08:58:02.405075 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":219,"spinlock_wait_cycles":9600,"thread_start_us":116,"threads_started":1}
I20251209 08:58:02.405731 22948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:58:02.402345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.49.66:52029 (local address 127.20.49.65:46509)
1209 08:58:02.402531 (+ 186us) server_negotiation.cc:207] Beginning negotiation
1209 08:58:02.402538 (+ 7us) server_negotiation.cc:400] Waiting for connection header
1209 08:58:02.402558 (+ 20us) server_negotiation.cc:408] Connection header received
1209 08:58:02.402619 (+ 61us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1209 08:58:02.402623 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1209 08:58:02.402677 (+ 54us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1209 08:58:02.402777 (+ 100us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1209 08:58:02.403191 (+ 414us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:58:02.404082 (+ 891us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1209 08:58:02.404934 (+ 852us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:58:02.405428 (+ 494us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:58:02.405478 (+ 50us) server_negotiation.cc:1036] Waiting for connection context
1209 08:58:02.405529 (+ 51us) server_negotiation.cc:300] Negotiation successful
1209 08:58:02.405585 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99}
I20251209 08:58:02.405743 22951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:58:02.402593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.49.66:49297 (local address 127.20.49.67:42299)
1209 08:58:02.402729 (+ 136us) server_negotiation.cc:207] Beginning negotiation
1209 08:58:02.402734 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1209 08:58:02.402826 (+ 92us) server_negotiation.cc:408] Connection header received
1209 08:58:02.402985 (+ 159us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1209 08:58:02.402988 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1209 08:58:02.403031 (+ 43us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1209 08:58:02.403103 (+ 72us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1209 08:58:02.403622 (+ 519us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:58:02.404134 (+ 512us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1209 08:58:02.404956 (+ 822us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:58:02.405458 (+ 502us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:58:02.405504 (+ 46us) server_negotiation.cc:1036] Waiting for connection context
1209 08:58:02.405558 (+ 54us) server_negotiation.cc:300] Negotiation successful
1209 08:58:02.405625 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20251209 08:58:02.406440 22600 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e98c48f6c144444c97e5450ffe99615d" candidate_uuid: "15ab8f217086450b933f077d59555f07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "da42139421e54188b827cd6256e162b0" is_pre_election: true
I20251209 08:58:02.406447 22884 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e98c48f6c144444c97e5450ffe99615d" candidate_uuid: "15ab8f217086450b933f077d59555f07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f2190f825aa04a0cb6b89cac0e770d12" is_pre_election: true
I20251209 08:58:02.406750 22884 raft_consensus.cc:2468] T e98c48f6c144444c97e5450ffe99615d P f2190f825aa04a0cb6b89cac0e770d12 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 15ab8f217086450b933f077d59555f07 in term 0.
I20251209 08:58:02.406750 22600 raft_consensus.cc:2468] T e98c48f6c144444c97e5450ffe99615d P da42139421e54188b827cd6256e162b0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 15ab8f217086450b933f077d59555f07 in term 0.
I20251209 08:58:02.407302 22674 leader_election.cc:304] T e98c48f6c144444c97e5450ffe99615d P 15ab8f217086450b933f077d59555f07 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 15ab8f217086450b933f077d59555f07, f2190f825aa04a0cb6b89cac0e770d12; no voters:
I20251209 08:58:02.407591 22959 raft_consensus.cc:2804] T e98c48f6c144444c97e5450ffe99615d P 15ab8f217086450b933f077d59555f07 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251209 08:58:02.407675 22959 raft_consensus.cc:493] T e98c48f6c144444c97e5450ffe99615d P 15ab8f217086450b933f077d59555f07 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251209 08:58:02.407749 22959 raft_consensus.cc:3060] T e98c48f6c144444c97e5450ffe99615d P 15ab8f217086450b933f077d59555f07 [term 0 FOLLOWER]: Advancing to term 1
W20251209 08:58:02.407860 22931 tablet.cc:2378] T e98c48f6c144444c97e5450ffe99615d P f2190f825aa04a0cb6b89cac0e770d12: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20251209 08:58:02.408782 22959 raft_consensus.cc:515] T e98c48f6c144444c97e5450ffe99615d P 15ab8f217086450b933f077d59555f07 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da42139421e54188b827cd6256e162b0" member_type: VOTER last_known_addr { host: "127.20.49.65" port: 46509 } } peers { permanent_uuid: "f2190f825aa04a0cb6b89cac0e770d12" member_type: VOTER last_known_addr { host: "127.20.49.67" port: 42299 } } peers { permanent_uuid: "15ab8f217086450b933f077d59555f07" member_type: VOTER last_known_addr { host: "127.20.49.66" port: 45423 } }
I20251209 08:58:02.409158 22959 leader_election.cc:290] T e98c48f6c144444c97e5450ffe99615d P 15ab8f217086450b933f077d59555f07 [CANDIDATE]: Term 1 election: Requested vote from peers da42139421e54188b827cd6256e162b0 (127.20.49.65:46509), f2190f825aa04a0cb6b89cac0e770d12 (127.20.49.67:42299)
W20251209 08:58:02.409546 22789 tablet.cc:2378] T e98c48f6c144444c97e5450ffe99615d P 15ab8f217086450b933f077d59555f07: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20251209 08:58:02.409606 22600 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e98c48f6c144444c97e5450ffe99615d" candidate_uuid: "15ab8f217086450b933f077d59555f07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "da42139421e54188b827cd6256e162b0"
I20251209 08:58:02.409705 22884 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e98c48f6c144444c97e5450ffe99615d" candidate_uuid: "15ab8f217086450b933f077d59555f07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f2190f825aa04a0cb6b89cac0e770d12"
I20251209 08:58:02.409829 22884 raft_consensus.cc:3060] T e98c48f6c144444c97e5450ffe99615d P f2190f825aa04a0cb6b89cac0e770d12 [term 0 FOLLOWER]: Advancing to term 1
I20251209 08:58:02.409795 22600 raft_consensus.cc:3060] T e98c48f6c144444c97e5450ffe99615d P da42139421e54188b827cd6256e162b0 [term 0 FOLLOWER]: Advancing to term 1
I20251209 08:58:02.410833 22884 raft_consensus.cc:2468] T e98c48f6c144444c97e5450ffe99615d P f2190f825aa04a0cb6b89cac0e770d12 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 15ab8f217086450b933f077d59555f07 in term 1.
I20251209 08:58:02.410938 22600 raft_consensus.cc:2468] T e98c48f6c144444c97e5450ffe99615d P da42139421e54188b827cd6256e162b0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 15ab8f217086450b933f077d59555f07 in term 1.
I20251209 08:58:02.411265 22674 leader_election.cc:304] T e98c48f6c144444c97e5450ffe99615d P 15ab8f217086450b933f077d59555f07 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 15ab8f217086450b933f077d59555f07, f2190f825aa04a0cb6b89cac0e770d12; no voters:
I20251209 08:58:02.411484 22959 raft_consensus.cc:2804] T e98c48f6c144444c97e5450ffe99615d P 15ab8f217086450b933f077d59555f07 [term 1 FOLLOWER]: Leader election won for term 1
I20251209 08:58:02.411710 22959 raft_consensus.cc:697] T e98c48f6c144444c97e5450ffe99615d P 15ab8f217086450b933f077d59555f07 [term 1 LEADER]: Becoming Leader. State: Replica: 15ab8f217086450b933f077d59555f07, State: Running, Role: LEADER
I20251209 08:58:02.412036 22959 consensus_queue.cc:237] T e98c48f6c144444c97e5450ffe99615d P 15ab8f217086450b933f077d59555f07 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da42139421e54188b827cd6256e162b0" member_type: VOTER last_known_addr { host: "127.20.49.65" port: 46509 } } peers { permanent_uuid: "f2190f825aa04a0cb6b89cac0e770d12" member_type: VOTER last_known_addr { host: "127.20.49.67" port: 42299 } } peers { permanent_uuid: "15ab8f217086450b933f077d59555f07" member_type: VOTER last_known_addr { host: "127.20.49.66" port: 45423 } }
I20251209 08:58:02.415314 22440 catalog_manager.cc:5654] T e98c48f6c144444c97e5450ffe99615d P 15ab8f217086450b933f077d59555f07 reported cstate change: term changed from 0 to 1, leader changed from <none> to 15ab8f217086450b933f077d59555f07 (127.20.49.66). New cstate: current_term: 1 leader_uuid: "15ab8f217086450b933f077d59555f07" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da42139421e54188b827cd6256e162b0" member_type: VOTER last_known_addr { host: "127.20.49.65" port: 46509 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f2190f825aa04a0cb6b89cac0e770d12" member_type: VOTER last_known_addr { host: "127.20.49.67" port: 42299 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "15ab8f217086450b933f077d59555f07" member_type: VOTER last_known_addr { host: "127.20.49.66" port: 45423 } health_report { overall_health: HEALTHY } } }
Dec 09 08:58:02 dist-test-slave-8w6c krb5kdc[22393](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765270681, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.49.126@KRBTEST.COM for kudu/127.20.49.126@KRBTEST.COM
I20251209 08:58:02.476192 22970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:58:02.467654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.49.126:39233 (local address 127.0.0.1:42092)
1209 08:58:02.467990 (+ 336us) negotiation.cc:107] Waiting for socket to connect
1209 08:58:02.468005 (+ 15us) client_negotiation.cc:174] Beginning negotiation
1209 08:58:02.468115 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1209 08:58:02.468357 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1209 08:58:02.468360 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1209 08:58:02.468799 (+ 439us) client_negotiation.cc:189] Negotiated authn=SASL
1209 08:58:02.469001 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:58:02.469008 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:58:02.469805 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1209 08:58:02.469809 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1209 08:58:02.470242 (+ 433us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:58:02.470249 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:58:02.470356 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:58:02.471027 (+ 671us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1209 08:58:02.471042 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
1209 08:58:02.472610 (+ 1568us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1209 08:58:02.474704 (+ 2094us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1209 08:58:02.474713 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1209 08:58:02.474718 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
1209 08:58:02.474946 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1209 08:58:02.475139 (+ 193us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1209 08:58:02.475141 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1209 08:58:02.475143 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1209 08:58:02.475186 (+ 43us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1209 08:58:02.475640 (+ 454us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1209 08:58:02.475647 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1209 08:58:02.475757 (+ 110us) client_negotiation.cc:769] Sending connection context
1209 08:58:02.475872 (+ 115us) client_negotiation.cc:240] Negotiation successful
1209 08:58:02.476000 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":136,"threads_started":1}
I20251209 08:58:02.476191 22490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:58:02.467706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42092 (local address 127.20.49.126:39233)
1209 08:58:02.467864 (+ 158us) server_negotiation.cc:207] Beginning negotiation
1209 08:58:02.467873 (+ 9us) server_negotiation.cc:400] Waiting for connection header
1209 08:58:02.468096 (+ 223us) server_negotiation.cc:408] Connection header received
1209 08:58:02.468194 (+ 98us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1209 08:58:02.468196 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1209 08:58:02.468246 (+ 50us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1209 08:58:02.468330 (+ 84us) server_negotiation.cc:227] Negotiated authn=SASL
1209 08:58:02.469128 (+ 798us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:58:02.469660 (+ 532us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1209 08:58:02.470357 (+ 697us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:58:02.470507 (+ 150us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:58:02.472767 (+ 2260us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1209 08:58:02.472787 (+ 20us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1209 08:58:02.472790 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1209 08:58:02.472813 (+ 23us) server_negotiation.cc:923] calling sasl_server_start()
1209 08:58:02.474579 (+ 1766us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1209 08:58:02.475035 (+ 456us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1209 08:58:02.475037 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1209 08:58:02.475039 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1209 08:58:02.475076 (+ 37us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1209 08:58:02.475299 (+ 223us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1209 08:58:02.475303 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1209 08:58:02.475305 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1209 08:58:02.475494 (+ 189us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1209 08:58:02.475594 (+ 100us) server_negotiation.cc:1036] Waiting for connection context
1209 08:58:02.475959 (+ 365us) server_negotiation.cc:300] Negotiation successful
1209 08:58:02.476064 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20251209 08:58:02.480496 22440 catalog_manager.cc:2257] Servicing CreateTable request from {username='kudu', principal='kudu/127.20.49.126@KRBTEST.COM'} at 127.0.0.1:42092:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20251209 08:58:02.481287 22440 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251209 08:58:02.486408 22580 tablet_service.cc:1505] Processing CreateTablet for tablet b805b1032118471f8277b20cefe39dd8 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=82fd900e3a8d43369f2da0862e366fac]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20251209 08:58:02.486649 22722 tablet_service.cc:1505] Processing CreateTablet for tablet b805b1032118471f8277b20cefe39dd8 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=82fd900e3a8d43369f2da0862e366fac]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20251209 08:58:02.486707 22864 tablet_service.cc:1505] Processing CreateTablet for tablet b805b1032118471f8277b20cefe39dd8 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=82fd900e3a8d43369f2da0862e366fac]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20251209 08:58:02.487188 22580 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b805b1032118471f8277b20cefe39dd8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251209 08:58:02.487243 22722 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b805b1032118471f8277b20cefe39dd8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251209 08:58:02.487246 22864 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b805b1032118471f8277b20cefe39dd8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251209 08:58:02.489714 22953 tablet_bootstrap.cc:492] T b805b1032118471f8277b20cefe39dd8 P da42139421e54188b827cd6256e162b0: Bootstrap starting.
I20251209 08:58:02.490006 22956 tablet_bootstrap.cc:492] T b805b1032118471f8277b20cefe39dd8 P f2190f825aa04a0cb6b89cac0e770d12: Bootstrap starting.
I20251209 08:58:02.490281 22954 tablet_bootstrap.cc:492] T b805b1032118471f8277b20cefe39dd8 P 15ab8f217086450b933f077d59555f07: Bootstrap starting.
I20251209 08:58:02.490914 22953 tablet_bootstrap.cc:654] T b805b1032118471f8277b20cefe39dd8 P da42139421e54188b827cd6256e162b0: Neither blocks nor log segments found. Creating new log.
I20251209 08:58:02.491104 22956 tablet_bootstrap.cc:654] T b805b1032118471f8277b20cefe39dd8 P f2190f825aa04a0cb6b89cac0e770d12: Neither blocks nor log segments found. Creating new log.
I20251209 08:58:02.491164 22954 tablet_bootstrap.cc:654] T b805b1032118471f8277b20cefe39dd8 P 15ab8f217086450b933f077d59555f07: Neither blocks nor log segments found. Creating new log.
I20251209 08:58:02.492489 22953 tablet_bootstrap.cc:492] T b805b1032118471f8277b20cefe39dd8 P da42139421e54188b827cd6256e162b0: No bootstrap required, opened a new log
I20251209 08:58:02.492584 22953 ts_tablet_manager.cc:1403] T b805b1032118471f8277b20cefe39dd8 P da42139421e54188b827cd6256e162b0: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20251209 08:58:02.492645 22954 tablet_bootstrap.cc:492] T b805b1032118471f8277b20cefe39dd8 P 15ab8f217086450b933f077d59555f07: No bootstrap required, opened a new log
I20251209 08:58:02.492748 22954 ts_tablet_manager.cc:1403] T b805b1032118471f8277b20cefe39dd8 P 15ab8f217086450b933f077d59555f07: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20251209 08:58:02.493072 22953 raft_consensus.cc:359] T b805b1032118471f8277b20cefe39dd8 P da42139421e54188b827cd6256e162b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da42139421e54188b827cd6256e162b0" member_type: VOTER last_known_addr { host: "127.20.49.65" port: 46509 } } peers { permanent_uuid: "15ab8f217086450b933f077d59555f07" member_type: VOTER last_known_addr { host: "127.20.49.66" port: 45423 } } peers { permanent_uuid: "f2190f825aa04a0cb6b89cac0e770d12" member_type: VOTER last_known_addr { host: "127.20.49.67" port: 42299 } }
I20251209 08:58:02.493180 22953 raft_consensus.cc:385] T b805b1032118471f8277b20cefe39dd8 P da42139421e54188b827cd6256e162b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251209 08:58:02.493213 22953 raft_consensus.cc:740] T b805b1032118471f8277b20cefe39dd8 P da42139421e54188b827cd6256e162b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: da42139421e54188b827cd6256e162b0, State: Initialized, Role: FOLLOWER
I20251209 08:58:02.493256 22954 raft_consensus.cc:359] T b805b1032118471f8277b20cefe39dd8 P 15ab8f217086450b933f077d59555f07 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da42139421e54188b827cd6256e162b0" member_type: VOTER last_known_addr { host: "127.20.49.65" port: 46509 } } peers { permanent_uuid: "15ab8f217086450b933f077d59555f07" member_type: VOTER last_known_addr { host: "127.20.49.66" port: 45423 } } peers { permanent_uuid: "f2190f825aa04a0cb6b89cac0e770d12" member_type: VOTER last_known_addr { host: "127.20.49.67" port: 42299 } }
I20251209 08:58:02.493376 22954 raft_consensus.cc:385] T b805b1032118471f8277b20cefe39dd8 P 15ab8f217086450b933f077d59555f07 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251209 08:58:02.493433 22954 raft_consensus.cc:740] T b805b1032118471f8277b20cefe39dd8 P 15ab8f217086450b933f077d59555f07 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 15ab8f217086450b933f077d59555f07, State: Initialized, Role: FOLLOWER
I20251209 08:58:02.493386 22953 consensus_queue.cc:260] T b805b1032118471f8277b20cefe39dd8 P da42139421e54188b827cd6256e162b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da42139421e54188b827cd6256e162b0" member_type: VOTER last_known_addr { host: "127.20.49.65" port: 46509 } } peers { permanent_uuid: "15ab8f217086450b933f077d59555f07" member_type: VOTER last_known_addr { host: "127.20.49.66" port: 45423 } } peers { permanent_uuid: "f2190f825aa04a0cb6b89cac0e770d12" member_type: VOTER last_known_addr { host: "127.20.49.67" port: 42299 } }
I20251209 08:58:02.493563 22954 consensus_queue.cc:260] T b805b1032118471f8277b20cefe39dd8 P 15ab8f217086450b933f077d59555f07 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da42139421e54188b827cd6256e162b0" member_type: VOTER last_known_addr { host: "127.20.49.65" port: 46509 } } peers { permanent_uuid: "15ab8f217086450b933f077d59555f07" member_type: VOTER last_known_addr { host: "127.20.49.66" port: 45423 } } peers { permanent_uuid: "f2190f825aa04a0cb6b89cac0e770d12" member_type: VOTER last_known_addr { host: "127.20.49.67" port: 42299 } }
I20251209 08:58:02.493661 22958 tablet_replica.cc:442] T b805b1032118471f8277b20cefe39dd8 P da42139421e54188b827cd6256e162b0: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da42139421e54188b827cd6256e162b0" member_type: VOTER last_known_addr { host: "127.20.49.65" port: 46509 } } peers { permanent_uuid: "15ab8f217086450b933f077d59555f07" member_type: VOTER last_known_addr { host: "127.20.49.66" port: 45423 } } peers { permanent_uuid: "f2190f825aa04a0cb6b89cac0e770d12" member_type: VOTER last_known_addr { host: "127.20.49.67" port: 42299 } } }
I20251209 08:58:02.493811 22958 tablet_replica.cc:445] T b805b1032118471f8277b20cefe39dd8 P da42139421e54188b827cd6256e162b0: This TxnStatusTablet replica's current role is: FOLLOWER
I20251209 08:58:02.493853 22959 tablet_replica.cc:442] T b805b1032118471f8277b20cefe39dd8 P 15ab8f217086450b933f077d59555f07: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da42139421e54188b827cd6256e162b0" member_type: VOTER last_known_addr { host: "127.20.49.65" port: 46509 } } peers { permanent_uuid: "15ab8f217086450b933f077d59555f07" member_type: VOTER last_known_addr { host: "127.20.49.66" port: 45423 } } peers { permanent_uuid: "f2190f825aa04a0cb6b89cac0e770d12" member_type: VOTER last_known_addr { host: "127.20.49.67" port: 42299 } } }
I20251209 08:58:02.493999 22959 tablet_replica.cc:445] T b805b1032118471f8277b20cefe39dd8 P 15ab8f217086450b933f077d59555f07: This TxnStatusTablet replica's current role is: FOLLOWER
I20251209 08:58:02.493672 22953 ts_tablet_manager.cc:1434] T b805b1032118471f8277b20cefe39dd8 P da42139421e54188b827cd6256e162b0: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20251209 08:58:02.493850 22954 ts_tablet_manager.cc:1434] T b805b1032118471f8277b20cefe39dd8 P 15ab8f217086450b933f077d59555f07: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20251209 08:58:02.495443 22956 tablet_bootstrap.cc:492] T b805b1032118471f8277b20cefe39dd8 P f2190f825aa04a0cb6b89cac0e770d12: No bootstrap required, opened a new log
I20251209 08:58:02.495551 22956 ts_tablet_manager.cc:1403] T b805b1032118471f8277b20cefe39dd8 P f2190f825aa04a0cb6b89cac0e770d12: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20251209 08:58:02.496047 22956 raft_consensus.cc:359] T b805b1032118471f8277b20cefe39dd8 P f2190f825aa04a0cb6b89cac0e770d12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da42139421e54188b827cd6256e162b0" member_type: VOTER last_known_addr { host: "127.20.49.65" port: 46509 } } peers { permanent_uuid: "15ab8f217086450b933f077d59555f07" member_type: VOTER last_known_addr { host: "127.20.49.66" port: 45423 } } peers { permanent_uuid: "f2190f825aa04a0cb6b89cac0e770d12" member_type: VOTER last_known_addr { host: "127.20.49.67" port: 42299 } }
I20251209 08:58:02.496199 22956 raft_consensus.cc:385] T b805b1032118471f8277b20cefe39dd8 P f2190f825aa04a0cb6b89cac0e770d12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251209 08:58:02.496250 22956 raft_consensus.cc:740] T b805b1032118471f8277b20cefe39dd8 P f2190f825aa04a0cb6b89cac0e770d12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f2190f825aa04a0cb6b89cac0e770d12, State: Initialized, Role: FOLLOWER
I20251209 08:58:02.496380 22956 consensus_queue.cc:260] T b805b1032118471f8277b20cefe39dd8 P f2190f825aa04a0cb6b89cac0e770d12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da42139421e54188b827cd6256e162b0" member_type: VOTER last_known_addr { host: "127.20.49.65" port: 46509 } } peers { permanent_uuid: "15ab8f217086450b933f077d59555f07" member_type: VOTER last_known_addr { host: "127.20.49.66" port: 45423 } } peers { permanent_uuid: "f2190f825aa04a0cb6b89cac0e770d12" member_type: VOTER last_known_addr { host: "127.20.49.67" port: 42299 } }
I20251209 08:58:02.496645 22956 ts_tablet_manager.cc:1434] T b805b1032118471f8277b20cefe39dd8 P f2190f825aa04a0cb6b89cac0e770d12: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20251209 08:58:02.497052 22961 tablet_replica.cc:442] T b805b1032118471f8277b20cefe39dd8 P f2190f825aa04a0cb6b89cac0e770d12: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da42139421e54188b827cd6256e162b0" member_type: VOTER last_known_addr { host: "127.20.49.65" port: 46509 } } peers { permanent_uuid: "15ab8f217086450b933f077d59555f07" member_type: VOTER last_known_addr { host: "127.20.49.66" port: 45423 } } peers { permanent_uuid: "f2190f825aa04a0cb6b89cac0e770d12" member_type: VOTER last_known_addr { host: "127.20.49.67" port: 42299 } } }
I20251209 08:58:02.497200 22961 tablet_replica.cc:445] T b805b1032118471f8277b20cefe39dd8 P f2190f825aa04a0cb6b89cac0e770d12: This TxnStatusTablet replica's current role is: FOLLOWER
I20251209 08:58:02.796311 22958 raft_consensus.cc:493] T b805b1032118471f8277b20cefe39dd8 P da42139421e54188b827cd6256e162b0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251209 08:58:02.796481 22958 raft_consensus.cc:515] T b805b1032118471f8277b20cefe39dd8 P da42139421e54188b827cd6256e162b0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da42139421e54188b827cd6256e162b0" member_type: VOTER last_known_addr { host: "127.20.49.65" port: 46509 } } peers { permanent_uuid: "15ab8f217086450b933f077d59555f07" member_type: VOTER last_known_addr { host: "127.20.49.66" port: 45423 } } peers { permanent_uuid: "f2190f825aa04a0cb6b89cac0e770d12" member_type: VOTER last_known_addr { host: "127.20.49.67" port: 42299 } }
I20251209 08:58:02.797822 22958 leader_election.cc:290] T b805b1032118471f8277b20cefe39dd8 P da42139421e54188b827cd6256e162b0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 15ab8f217086450b933f077d59555f07 (127.20.49.66:45423), f2190f825aa04a0cb6b89cac0e770d12 (127.20.49.67:42299)
I20251209 08:58:02.801855 22975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:58:02.798044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.49.67:42299 (local address 127.20.49.65:49771)
1209 08:58:02.798458 (+ 414us) negotiation.cc:107] Waiting for socket to connect
1209 08:58:02.798473 (+ 15us) client_negotiation.cc:174] Beginning negotiation
1209 08:58:02.798585 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1209 08:58:02.798840 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1209 08:58:02.798843 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1209 08:58:02.798860 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1209 08:58:02.799133 (+ 273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:58:02.799141 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:58:02.800235 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1209 08:58:02.800240 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1209 08:58:02.801287 (+ 1047us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:58:02.801302 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:58:02.801447 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:58:02.801487 (+ 40us) client_negotiation.cc:769] Sending connection context
1209 08:58:02.801542 (+ 55us) client_negotiation.cc:240] Negotiation successful
1209 08:58:02.801626 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":346,"mutex_wait_us":12,"spinlock_wait_cycles":71168,"thread_start_us":188,"threads_started":1}
I20251209 08:58:02.802099 22951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:58:02.798225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.49.65:49771 (local address 127.20.49.67:42299)
1209 08:58:02.798455 (+ 230us) server_negotiation.cc:207] Beginning negotiation
1209 08:58:02.798460 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1209 08:58:02.798552 (+ 92us) server_negotiation.cc:408] Connection header received
1209 08:58:02.798693 (+ 141us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1209 08:58:02.798695 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1209 08:58:02.798739 (+ 44us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1209 08:58:02.798803 (+ 64us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1209 08:58:02.799289 (+ 486us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:58:02.800045 (+ 756us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1209 08:58:02.801438 (+ 1393us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:58:02.801821 (+ 383us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:58:02.801859 (+ 38us) server_negotiation.cc:1036] Waiting for connection context
1209 08:58:02.801914 (+ 55us) server_negotiation.cc:300] Negotiation successful
1209 08:58:02.801974 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106}
I20251209 08:58:02.802228 22974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:58:02.798044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.49.66:45423 (local address 127.20.49.65:33845)
1209 08:58:02.798355 (+ 311us) negotiation.cc:107] Waiting for socket to connect
1209 08:58:02.798371 (+ 16us) client_negotiation.cc:174] Beginning negotiation
1209 08:58:02.798496 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1209 08:58:02.798865 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1209 08:58:02.798868 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1209 08:58:02.798898 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1209 08:58:02.799099 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:58:02.799105 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:58:02.800260 (+ 1155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1209 08:58:02.800263 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1209 08:58:02.801788 (+ 1525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:58:02.801797 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:58:02.801927 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:58:02.801948 (+ 21us) client_negotiation.cc:769] Sending connection context
1209 08:58:02.802004 (+ 56us) client_negotiation.cc:240] Negotiation successful
1209 08:58:02.802067 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":122,"threads_started":1}
I20251209 08:58:02.802556 22884 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b805b1032118471f8277b20cefe39dd8" candidate_uuid: "da42139421e54188b827cd6256e162b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f2190f825aa04a0cb6b89cac0e770d12" is_pre_election: true
I20251209 08:58:02.802677 22976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:58:02.798367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.49.65:33845 (local address 127.20.49.66:45423)
1209 08:58:02.798625 (+ 258us) server_negotiation.cc:207] Beginning negotiation
1209 08:58:02.798631 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1209 08:58:02.798649 (+ 18us) server_negotiation.cc:408] Connection header received
1209 08:58:02.798708 (+ 59us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1209 08:58:02.798711 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1209 08:58:02.798752 (+ 41us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1209 08:58:02.798837 (+ 85us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1209 08:58:02.799241 (+ 404us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:58:02.800052 (+ 811us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1209 08:58:02.801910 (+ 1858us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:58:02.802384 (+ 474us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:58:02.802421 (+ 37us) server_negotiation.cc:1036] Waiting for connection context
1209 08:58:02.802481 (+ 60us) server_negotiation.cc:300] Negotiation successful
1209 08:58:02.802541 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":173,"thread_start_us":84,"threads_started":1}
I20251209 08:58:02.802717 22884 raft_consensus.cc:2468] T b805b1032118471f8277b20cefe39dd8 P f2190f825aa04a0cb6b89cac0e770d12 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate da42139421e54188b827cd6256e162b0 in term 0.
I20251209 08:58:02.803117 22532 leader_election.cc:304] T b805b1032118471f8277b20cefe39dd8 P da42139421e54188b827cd6256e162b0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: da42139421e54188b827cd6256e162b0, f2190f825aa04a0cb6b89cac0e770d12; no voters:
I20251209 08:58:02.803158 22742 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b805b1032118471f8277b20cefe39dd8" candidate_uuid: "da42139421e54188b827cd6256e162b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15ab8f217086450b933f077d59555f07" is_pre_election: true
I20251209 08:58:02.803357 22958 raft_consensus.cc:2804] T b805b1032118471f8277b20cefe39dd8 P da42139421e54188b827cd6256e162b0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251209 08:58:02.803417 22958 raft_consensus.cc:493] T b805b1032118471f8277b20cefe39dd8 P da42139421e54188b827cd6256e162b0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251209 08:58:02.803413 22742 raft_consensus.cc:2468] T b805b1032118471f8277b20cefe39dd8 P 15ab8f217086450b933f077d59555f07 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate da42139421e54188b827cd6256e162b0 in term 0.
I20251209 08:58:02.803484 22958 raft_consensus.cc:3060] T b805b1032118471f8277b20cefe39dd8 P da42139421e54188b827cd6256e162b0 [term 0 FOLLOWER]: Advancing to term 1
I20251209 08:58:02.804499 22958 raft_consensus.cc:515] T b805b1032118471f8277b20cefe39dd8 P da42139421e54188b827cd6256e162b0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da42139421e54188b827cd6256e162b0" member_type: VOTER last_known_addr { host: "127.20.49.65" port: 46509 } } peers { permanent_uuid: "15ab8f217086450b933f077d59555f07" member_type: VOTER last_known_addr { host: "127.20.49.66" port: 45423 } } peers { permanent_uuid: "f2190f825aa04a0cb6b89cac0e770d12" member_type: VOTER last_known_addr { host: "127.20.49.67" port: 42299 } }
I20251209 08:58:02.804839 22958 leader_election.cc:290] T b805b1032118471f8277b20cefe39dd8 P da42139421e54188b827cd6256e162b0 [CANDIDATE]: Term 1 election: Requested vote from peers 15ab8f217086450b933f077d59555f07 (127.20.49.66:45423), f2190f825aa04a0cb6b89cac0e770d12 (127.20.49.67:42299)
I20251209 08:58:02.805240 22884 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b805b1032118471f8277b20cefe39dd8" candidate_uuid: "da42139421e54188b827cd6256e162b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f2190f825aa04a0cb6b89cac0e770d12"
I20251209 08:58:02.805231 22742 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b805b1032118471f8277b20cefe39dd8" candidate_uuid: "da42139421e54188b827cd6256e162b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15ab8f217086450b933f077d59555f07"
I20251209 08:58:02.805359 22742 raft_consensus.cc:3060] T b805b1032118471f8277b20cefe39dd8 P 15ab8f217086450b933f077d59555f07 [term 0 FOLLOWER]: Advancing to term 1
I20251209 08:58:02.805359 22884 raft_consensus.cc:3060] T b805b1032118471f8277b20cefe39dd8 P f2190f825aa04a0cb6b89cac0e770d12 [term 0 FOLLOWER]: Advancing to term 1
I20251209 08:58:02.806689 22884 raft_consensus.cc:2468] T b805b1032118471f8277b20cefe39dd8 P f2190f825aa04a0cb6b89cac0e770d12 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate da42139421e54188b827cd6256e162b0 in term 1.
I20251209 08:58:02.807000 22532 leader_election.cc:304] T b805b1032118471f8277b20cefe39dd8 P da42139421e54188b827cd6256e162b0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: da42139421e54188b827cd6256e162b0, f2190f825aa04a0cb6b89cac0e770d12; no voters:
I20251209 08:58:02.807036 22742 raft_consensus.cc:2468] T b805b1032118471f8277b20cefe39dd8 P 15ab8f217086450b933f077d59555f07 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate da42139421e54188b827cd6256e162b0 in term 1.
I20251209 08:58:02.807232 22958 raft_consensus.cc:2804] T b805b1032118471f8277b20cefe39dd8 P da42139421e54188b827cd6256e162b0 [term 1 FOLLOWER]: Leader election won for term 1
I20251209 08:58:02.807456 22958 raft_consensus.cc:697] T b805b1032118471f8277b20cefe39dd8 P da42139421e54188b827cd6256e162b0 [term 1 LEADER]: Becoming Leader. State: Replica: da42139421e54188b827cd6256e162b0, State: Running, Role: LEADER
I20251209 08:58:02.807766 22958 consensus_queue.cc:237] T b805b1032118471f8277b20cefe39dd8 P da42139421e54188b827cd6256e162b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da42139421e54188b827cd6256e162b0" member_type: VOTER last_known_addr { host: "127.20.49.65" port: 46509 } } peers { permanent_uuid: "15ab8f217086450b933f077d59555f07" member_type: VOTER last_known_addr { host: "127.20.49.66" port: 45423 } } peers { permanent_uuid: "f2190f825aa04a0cb6b89cac0e770d12" member_type: VOTER last_known_addr { host: "127.20.49.67" port: 42299 } }
I20251209 08:58:02.809995 22977 tablet_replica.cc:442] T b805b1032118471f8277b20cefe39dd8 P da42139421e54188b827cd6256e162b0: TxnStatusTablet state changed. Reason: New leader da42139421e54188b827cd6256e162b0. Latest consensus state: current_term: 1 leader_uuid: "da42139421e54188b827cd6256e162b0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da42139421e54188b827cd6256e162b0" member_type: VOTER last_known_addr { host: "127.20.49.65" port: 46509 } } peers { permanent_uuid: "15ab8f217086450b933f077d59555f07" member_type: VOTER last_known_addr { host: "127.20.49.66" port: 45423 } } peers { permanent_uuid: "f2190f825aa04a0cb6b89cac0e770d12" member_type: VOTER last_known_addr { host: "127.20.49.67" port: 42299 } } }
I20251209 08:58:02.810184 22977 tablet_replica.cc:445] T b805b1032118471f8277b20cefe39dd8 P da42139421e54188b827cd6256e162b0: This TxnStatusTablet replica's current role is: LEADER
I20251209 08:58:02.810492 22959 consensus_queue.cc:1048] T e98c48f6c144444c97e5450ffe99615d P 15ab8f217086450b933f077d59555f07 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f2190f825aa04a0cb6b89cac0e770d12" member_type: VOTER last_known_addr { host: "127.20.49.67" port: 42299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251209 08:58:02.810699 22979 txn_status_manager.cc:874] Waiting until node catch up with all replicated operations in previous term...
I20251209 08:58:02.810865 22979 txn_status_manager.cc:930] Loading transaction status metadata into memory...
I20251209 08:58:02.811549 22441 catalog_manager.cc:5654] T b805b1032118471f8277b20cefe39dd8 P da42139421e54188b827cd6256e162b0 reported cstate change: term changed from 0 to 1, leader changed from <none> to da42139421e54188b827cd6256e162b0 (127.20.49.65). New cstate: current_term: 1 leader_uuid: "da42139421e54188b827cd6256e162b0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da42139421e54188b827cd6256e162b0" member_type: VOTER last_known_addr { host: "127.20.49.65" port: 46509 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "15ab8f217086450b933f077d59555f07" member_type: VOTER last_known_addr { host: "127.20.49.66" port: 45423 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f2190f825aa04a0cb6b89cac0e770d12" member_type: VOTER last_known_addr { host: "127.20.49.67" port: 42299 } health_report { overall_health: UNKNOWN } } }
I20251209 08:58:02.817235 22963 consensus_queue.cc:1048] T e98c48f6c144444c97e5450ffe99615d P 15ab8f217086450b933f077d59555f07 [LEADER]: Connected to new peer: Peer: permanent_uuid: "da42139421e54188b827cd6256e162b0" member_type: VOTER last_known_addr { host: "127.20.49.65" port: 46509 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251209 08:58:02.858947 22948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:58:02.855567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44200 (local address 127.20.49.65:46509)
1209 08:58:02.855703 (+ 136us) server_negotiation.cc:207] Beginning negotiation
1209 08:58:02.855710 (+ 7us) server_negotiation.cc:400] Waiting for connection header
1209 08:58:02.855726 (+ 16us) server_negotiation.cc:408] Connection header received
1209 08:58:02.855972 (+ 246us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1209 08:58:02.855975 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1209 08:58:02.856044 (+ 69us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1209 08:58:02.856132 (+ 88us) server_negotiation.cc:227] Negotiated authn=TOKEN
1209 08:58:02.856516 (+ 384us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:58:02.856984 (+ 468us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1209 08:58:02.857719 (+ 735us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:58:02.857957 (+ 238us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:58:02.858014 (+ 57us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1209 08:58:02.858515 (+ 501us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1209 08:58:02.858631 (+ 116us) server_negotiation.cc:1036] Waiting for connection context
1209 08:58:02.858737 (+ 106us) server_negotiation.cc:300] Negotiation successful
1209 08:58:02.858796 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20251209 08:58:02.858963 22970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:58:02.855452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.49.65:46509 (local address 127.0.0.1:44200)
1209 08:58:02.855656 (+ 204us) negotiation.cc:107] Waiting for socket to connect
1209 08:58:02.855674 (+ 18us) client_negotiation.cc:174] Beginning negotiation
1209 08:58:02.855812 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1209 08:58:02.856153 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1209 08:58:02.856156 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1209 08:58:02.856165 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
1209 08:58:02.856393 (+ 228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:58:02.856401 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:58:02.857116 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1209 08:58:02.857119 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1209 08:58:02.857565 (+ 446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:58:02.857574 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:58:02.857725 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:58:02.857789 (+ 64us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
1209 08:58:02.858619 (+ 830us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
1209 08:58:02.858644 (+ 25us) client_negotiation.cc:769] Sending connection context
1209 08:58:02.858720 (+ 76us) client_negotiation.cc:240] Negotiation successful
1209 08:58:02.858807 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":66}
I20251209 08:58:02.864771 22742 raft_consensus.cc:1275] T b805b1032118471f8277b20cefe39dd8 P 15ab8f217086450b933f077d59555f07 [term 1 FOLLOWER]: Refusing update from remote peer da42139421e54188b827cd6256e162b0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251209 08:58:02.865136 22884 raft_consensus.cc:1275] T b805b1032118471f8277b20cefe39dd8 P f2190f825aa04a0cb6b89cac0e770d12 [term 1 FOLLOWER]: Refusing update from remote peer da42139421e54188b827cd6256e162b0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251209 08:58:02.865389 22977 consensus_queue.cc:1048] T b805b1032118471f8277b20cefe39dd8 P da42139421e54188b827cd6256e162b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "15ab8f217086450b933f077d59555f07" member_type: VOTER last_known_addr { host: "127.20.49.66" port: 45423 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251209 08:58:02.865608 22958 consensus_queue.cc:1048] T b805b1032118471f8277b20cefe39dd8 P da42139421e54188b827cd6256e162b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f2190f825aa04a0cb6b89cac0e770d12" member_type: VOTER last_known_addr { host: "127.20.49.67" port: 42299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251209 08:58:02.868845 22961 tablet_replica.cc:442] T b805b1032118471f8277b20cefe39dd8 P f2190f825aa04a0cb6b89cac0e770d12: TxnStatusTablet state changed. Reason: New leader da42139421e54188b827cd6256e162b0. Latest consensus state: current_term: 1 leader_uuid: "da42139421e54188b827cd6256e162b0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da42139421e54188b827cd6256e162b0" member_type: VOTER last_known_addr { host: "127.20.49.65" port: 46509 } } peers { permanent_uuid: "15ab8f217086450b933f077d59555f07" member_type: VOTER last_known_addr { host: "127.20.49.66" port: 45423 } } peers { permanent_uuid: "f2190f825aa04a0cb6b89cac0e770d12" member_type: VOTER last_known_addr { host: "127.20.49.67" port: 42299 } } }
I20251209 08:58:02.868841 22959 tablet_replica.cc:442] T b805b1032118471f8277b20cefe39dd8 P 15ab8f217086450b933f077d59555f07: TxnStatusTablet state changed. Reason: New leader da42139421e54188b827cd6256e162b0. Latest consensus state: current_term: 1 leader_uuid: "da42139421e54188b827cd6256e162b0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da42139421e54188b827cd6256e162b0" member_type: VOTER last_known_addr { host: "127.20.49.65" port: 46509 } } peers { permanent_uuid: "15ab8f217086450b933f077d59555f07" member_type: VOTER last_known_addr { host: "127.20.49.66" port: 45423 } } peers { permanent_uuid: "f2190f825aa04a0cb6b89cac0e770d12" member_type: VOTER last_known_addr { host: "127.20.49.67" port: 42299 } } }
I20251209 08:58:02.868965 22959 tablet_replica.cc:445] T b805b1032118471f8277b20cefe39dd8 P 15ab8f217086450b933f077d59555f07: This TxnStatusTablet replica's current role is: FOLLOWER
I20251209 08:58:02.868965 22961 tablet_replica.cc:445] T b805b1032118471f8277b20cefe39dd8 P f2190f825aa04a0cb6b89cac0e770d12: This TxnStatusTablet replica's current role is: FOLLOWER
I20251209 08:58:02.871503 22985 mvcc.cc:204] Tried to move back new op lower bound from 7230548717007089664 to 7230548716783308800. Current Snapshot: MvccSnapshot[applied={T|T < 7230548717007089664}]
I20251209 08:58:02.871659 22977 tablet_replica.cc:442] T b805b1032118471f8277b20cefe39dd8 P da42139421e54188b827cd6256e162b0: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "da42139421e54188b827cd6256e162b0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da42139421e54188b827cd6256e162b0" member_type: VOTER last_known_addr { host: "127.20.49.65" port: 46509 } } peers { permanent_uuid: "15ab8f217086450b933f077d59555f07" member_type: VOTER last_known_addr { host: "127.20.49.66" port: 45423 } } peers { permanent_uuid: "f2190f825aa04a0cb6b89cac0e770d12" member_type: VOTER last_known_addr { host: "127.20.49.67" port: 42299 } } }
I20251209 08:58:02.871773 22977 tablet_replica.cc:445] T b805b1032118471f8277b20cefe39dd8 P da42139421e54188b827cd6256e162b0: This TxnStatusTablet replica's current role is: LEADER
I20251209 08:58:02.872165 22982 mvcc.cc:204] Tried to move back new op lower bound from 7230548717007089664 to 7230548716783308800. Current Snapshot: MvccSnapshot[applied={T|T < 7230548717007089664}]
I20251209 08:58:02.872350 22959 tablet_replica.cc:442] T b805b1032118471f8277b20cefe39dd8 P 15ab8f217086450b933f077d59555f07: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "da42139421e54188b827cd6256e162b0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da42139421e54188b827cd6256e162b0" member_type: VOTER last_known_addr { host: "127.20.49.65" port: 46509 } } peers { permanent_uuid: "15ab8f217086450b933f077d59555f07" member_type: VOTER last_known_addr { host: "127.20.49.66" port: 45423 } } peers { permanent_uuid: "f2190f825aa04a0cb6b89cac0e770d12" member_type: VOTER last_known_addr { host: "127.20.49.67" port: 42299 } } }
I20251209 08:58:02.872454 22959 tablet_replica.cc:445] T b805b1032118471f8277b20cefe39dd8 P 15ab8f217086450b933f077d59555f07: This TxnStatusTablet replica's current role is: FOLLOWER
I20251209 08:58:02.874503 22958 tablet_replica.cc:442] T b805b1032118471f8277b20cefe39dd8 P da42139421e54188b827cd6256e162b0: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "da42139421e54188b827cd6256e162b0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da42139421e54188b827cd6256e162b0" member_type: VOTER last_known_addr { host: "127.20.49.65" port: 46509 } } peers { permanent_uuid: "15ab8f217086450b933f077d59555f07" member_type: VOTER last_known_addr { host: "127.20.49.66" port: 45423 } } peers { permanent_uuid: "f2190f825aa04a0cb6b89cac0e770d12" member_type: VOTER last_known_addr { host: "127.20.49.67" port: 42299 } } }
I20251209 08:58:02.874647 22958 tablet_replica.cc:445] T b805b1032118471f8277b20cefe39dd8 P da42139421e54188b827cd6256e162b0: This TxnStatusTablet replica's current role is: LEADER
I20251209 08:58:02.876228 22961 tablet_replica.cc:442] T b805b1032118471f8277b20cefe39dd8 P f2190f825aa04a0cb6b89cac0e770d12: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "da42139421e54188b827cd6256e162b0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da42139421e54188b827cd6256e162b0" member_type: VOTER last_known_addr { host: "127.20.49.65" port: 46509 } } peers { permanent_uuid: "15ab8f217086450b933f077d59555f07" member_type: VOTER last_known_addr { host: "127.20.49.66" port: 45423 } } peers { permanent_uuid: "f2190f825aa04a0cb6b89cac0e770d12" member_type: VOTER last_known_addr { host: "127.20.49.67" port: 42299 } } }
I20251209 08:58:02.876365 22961 tablet_replica.cc:445] T b805b1032118471f8277b20cefe39dd8 P f2190f825aa04a0cb6b89cac0e770d12: This TxnStatusTablet replica's current role is: FOLLOWER
I20251209 08:58:02.883999 22976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:58:02.880305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33072 (local address 127.20.49.66:45423)
1209 08:58:02.880476 (+ 171us) server_negotiation.cc:207] Beginning negotiation
1209 08:58:02.880482 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1209 08:58:02.880656 (+ 174us) server_negotiation.cc:408] Connection header received
1209 08:58:02.880818 (+ 162us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1209 08:58:02.880821 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1209 08:58:02.880876 (+ 55us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1209 08:58:02.880966 (+ 90us) server_negotiation.cc:227] Negotiated authn=TOKEN
1209 08:58:02.881359 (+ 393us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:58:02.882063 (+ 704us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1209 08:58:02.882819 (+ 756us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:58:02.883038 (+ 219us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:58:02.883098 (+ 60us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1209 08:58:02.883573 (+ 475us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1209 08:58:02.883687 (+ 114us) server_negotiation.cc:1036] Waiting for connection context
1209 08:58:02.883778 (+ 91us) server_negotiation.cc:300] Negotiation successful
1209 08:58:02.883833 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20251209 08:58:02.895016 22996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:58:02.891851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.49.65:46509 (local address 127.20.49.66:54055)
1209 08:58:02.892131 (+ 280us) negotiation.cc:107] Waiting for socket to connect
1209 08:58:02.892145 (+ 14us) client_negotiation.cc:174] Beginning negotiation
1209 08:58:02.892246 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1209 08:58:02.892528 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1209 08:58:02.892530 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1209 08:58:02.892536 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
1209 08:58:02.892783 (+ 247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:58:02.892789 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:58:02.893799 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1209 08:58:02.893802 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1209 08:58:02.894224 (+ 422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:58:02.894230 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:58:02.894353 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:58:02.894403 (+ 50us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
1209 08:58:02.894779 (+ 376us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
1209 08:58:02.894783 (+ 4us) client_negotiation.cc:769] Sending connection context
1209 08:58:02.894830 (+ 47us) client_negotiation.cc:240] Negotiation successful
1209 08:58:02.894886 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":96,"threads_started":1}
I20251209 08:58:02.895051 22948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:58:02.891938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.49.66:54055 (local address 127.20.49.65:46509)
1209 08:58:02.892078 (+ 140us) server_negotiation.cc:207] Beginning negotiation
1209 08:58:02.892083 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1209 08:58:02.892221 (+ 138us) server_negotiation.cc:408] Connection header received
1209 08:58:02.892365 (+ 144us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1209 08:58:02.892369 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1209 08:58:02.892430 (+ 61us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1209 08:58:02.892523 (+ 93us) server_negotiation.cc:227] Negotiated authn=TOKEN
1209 08:58:02.892876 (+ 353us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:58:02.893648 (+ 772us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1209 08:58:02.894361 (+ 713us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:58:02.894561 (+ 200us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:58:02.894612 (+ 51us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1209 08:58:02.894698 (+ 86us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1209 08:58:02.894778 (+ 80us) server_negotiation.cc:1036] Waiting for connection context
1209 08:58:02.894867 (+ 89us) server_negotiation.cc:300] Negotiation successful
1209 08:58:02.894918 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20251209 08:58:02.926811 22976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:58:02.923485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.49.65:57539 (local address 127.20.49.66:45423)
1209 08:58:02.923628 (+ 143us) server_negotiation.cc:207] Beginning negotiation
1209 08:58:02.923633 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1209 08:58:02.923840 (+ 207us) server_negotiation.cc:408] Connection header received
1209 08:58:02.923984 (+ 144us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1209 08:58:02.923989 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1209 08:58:02.924035 (+ 46us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1209 08:58:02.924101 (+ 66us) server_negotiation.cc:227] Negotiated authn=TOKEN
1209 08:58:02.924539 (+ 438us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:58:02.925095 (+ 556us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1209 08:58:02.925938 (+ 843us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:58:02.926115 (+ 177us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:58:02.926359 (+ 244us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1209 08:58:02.926436 (+ 77us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1209 08:58:02.926503 (+ 67us) server_negotiation.cc:1036] Waiting for connection context
1209 08:58:02.926617 (+ 114us) server_negotiation.cc:300] Negotiation successful
1209 08:58:02.926667 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20251209 08:58:02.926831 23000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:58:02.923412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.49.66:45423 (local address 127.20.49.65:57539)
1209 08:58:02.923757 (+ 345us) negotiation.cc:107] Waiting for socket to connect
1209 08:58:02.923771 (+ 14us) client_negotiation.cc:174] Beginning negotiation
1209 08:58:02.923882 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1209 08:58:02.924120 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1209 08:58:02.924123 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1209 08:58:02.924131 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
1209 08:58:02.924424 (+ 293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:58:02.924431 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:58:02.925231 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1209 08:58:02.925234 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1209 08:58:02.925801 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:58:02.925810 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:58:02.926183 (+ 373us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:58:02.926225 (+ 42us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
1209 08:58:02.926528 (+ 303us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
1209 08:58:02.926534 (+ 6us) client_negotiation.cc:769] Sending connection context
1209 08:58:02.926596 (+ 62us) client_negotiation.cc:240] Negotiation successful
1209 08:58:02.926674 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":124,"threads_started":1}
I20251209 08:58:02.952850 22441 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42090:
table { table_name: "test-table" } modify_external_catalogs: true
I20251209 08:58:02.953063 22441 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42090:
table { table_name: "test-table" } modify_external_catalogs: true
I20251209 08:58:02.956117 22441 catalog_manager.cc:5941] T 00000000000000000000000000000000 P 39341759984c48989ceba01b8acb2da9: Sending DeleteTablet for 3 replicas of tablet e98c48f6c144444c97e5450ffe99615d
I20251209 08:58:02.957338 22722 tablet_service.cc:1552] Processing DeleteTablet for tablet e98c48f6c144444c97e5450ffe99615d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-09 08:58:02 UTC) from {username='kudu', principal='kudu/127.20.49.126@KRBTEST.COM'} at 127.0.0.1:33058
I20251209 08:58:02.957393 22580 tablet_service.cc:1552] Processing DeleteTablet for tablet e98c48f6c144444c97e5450ffe99615d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-09 08:58:02 UTC) from {username='kudu', principal='kudu/127.20.49.126@KRBTEST.COM'} at 127.0.0.1:44186
I20251209 08:58:02.957836 22864 tablet_service.cc:1552] Processing DeleteTablet for tablet e98c48f6c144444c97e5450ffe99615d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-09 08:58:02 UTC) from {username='kudu', principal='kudu/127.20.49.126@KRBTEST.COM'} at 127.0.0.1:57140
I20251209 08:58:02.958609 23005 tablet_replica.cc:333] T e98c48f6c144444c97e5450ffe99615d P f2190f825aa04a0cb6b89cac0e770d12: stopping tablet replica
I20251209 08:58:02.958700 22441 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42090:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251209 08:58:02.959116 23005 raft_consensus.cc:2243] T e98c48f6c144444c97e5450ffe99615d P f2190f825aa04a0cb6b89cac0e770d12 [term 1 FOLLOWER]: Raft consensus shutting down.
W20251209 08:58:02.959189 22441 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251209 08:58:02.959527 23005 raft_consensus.cc:2272] T e98c48f6c144444c97e5450ffe99615d P f2190f825aa04a0cb6b89cac0e770d12 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251209 08:58:02.959990 23006 tablet_replica.cc:333] T e98c48f6c144444c97e5450ffe99615d P da42139421e54188b827cd6256e162b0: stopping tablet replica
I20251209 08:58:02.961321 23005 ts_tablet_manager.cc:1916] T e98c48f6c144444c97e5450ffe99615d P f2190f825aa04a0cb6b89cac0e770d12: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251209 08:58:02.961668 23006 raft_consensus.cc:2243] T e98c48f6c144444c97e5450ffe99615d P da42139421e54188b827cd6256e162b0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251209 08:58:02.961956 23006 raft_consensus.cc:2272] T e98c48f6c144444c97e5450ffe99615d P da42139421e54188b827cd6256e162b0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251209 08:58:02.963464 23006 ts_tablet_manager.cc:1916] T e98c48f6c144444c97e5450ffe99615d P da42139421e54188b827cd6256e162b0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251209 08:58:02.965355 22722 tablet_service.cc:1505] Processing CreateTablet for tablet 0aacf01c8a534c9da4886c3130df112f (DEFAULT_TABLE table=test-table [id=179d208d15f04a3c8d21dcffbf5bf06e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251209 08:58:02.965709 22722 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0aacf01c8a534c9da4886c3130df112f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251209 08:58:02.965759 22864 tablet_service.cc:1505] Processing CreateTablet for tablet 0aacf01c8a534c9da4886c3130df112f (DEFAULT_TABLE table=test-table [id=179d208d15f04a3c8d21dcffbf5bf06e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251209 08:58:02.966007 22864 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0aacf01c8a534c9da4886c3130df112f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251209 08:58:02.968463 22954 tablet_bootstrap.cc:492] T 0aacf01c8a534c9da4886c3130df112f P 15ab8f217086450b933f077d59555f07: Bootstrap starting.
I20251209 08:58:02.969638 22954 tablet_bootstrap.cc:654] T 0aacf01c8a534c9da4886c3130df112f P 15ab8f217086450b933f077d59555f07: Neither blocks nor log segments found. Creating new log.
I20251209 08:58:02.969827 22956 tablet_bootstrap.cc:492] T 0aacf01c8a534c9da4886c3130df112f P f2190f825aa04a0cb6b89cac0e770d12: Bootstrap starting.
I20251209 08:58:02.970834 22956 tablet_bootstrap.cc:654] T 0aacf01c8a534c9da4886c3130df112f P f2190f825aa04a0cb6b89cac0e770d12: Neither blocks nor log segments found. Creating new log.
I20251209 08:58:02.972638 23004 tablet_replica.cc:333] T e98c48f6c144444c97e5450ffe99615d P 15ab8f217086450b933f077d59555f07: stopping tablet replica
I20251209 08:58:02.973037 23004 raft_consensus.cc:2243] T e98c48f6c144444c97e5450ffe99615d P 15ab8f217086450b933f077d59555f07 [term 1 LEADER]: Raft consensus shutting down.
I20251209 08:58:02.965394 22580 tablet_service.cc:1505] Processing CreateTablet for tablet 0aacf01c8a534c9da4886c3130df112f (DEFAULT_TABLE table=test-table [id=179d208d15f04a3c8d21dcffbf5bf06e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251209 08:58:02.973405 22580 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0aacf01c8a534c9da4886c3130df112f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251209 08:58:02.976095 22953 tablet_bootstrap.cc:492] T 0aacf01c8a534c9da4886c3130df112f P da42139421e54188b827cd6256e162b0: Bootstrap starting.
I20251209 08:58:02.977218 22953 tablet_bootstrap.cc:654] T 0aacf01c8a534c9da4886c3130df112f P da42139421e54188b827cd6256e162b0: Neither blocks nor log segments found. Creating new log.
I20251209 08:58:02.973379 22954 tablet_bootstrap.cc:492] T 0aacf01c8a534c9da4886c3130df112f P 15ab8f217086450b933f077d59555f07: No bootstrap required, opened a new log
I20251209 08:58:02.977452 22954 ts_tablet_manager.cc:1403] T 0aacf01c8a534c9da4886c3130df112f P 15ab8f217086450b933f077d59555f07: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20251209 08:58:02.978034 22954 raft_consensus.cc:359] T 0aacf01c8a534c9da4886c3130df112f P 15ab8f217086450b933f077d59555f07 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da42139421e54188b827cd6256e162b0" member_type: VOTER last_known_addr { host: "127.20.49.65" port: 46509 } } peers { permanent_uuid: "15ab8f217086450b933f077d59555f07" member_type: VOTER last_known_addr { host: "127.20.49.66" port: 45423 } } peers { permanent_uuid: "f2190f825aa04a0cb6b89cac0e770d12" member_type: VOTER last_known_addr { host: "127.20.49.67" port: 42299 } }
I20251209 08:58:02.978181 22954 raft_consensus.cc:385] T 0aacf01c8a534c9da4886c3130df112f P 15ab8f217086450b933f077d59555f07 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251209 08:58:02.978214 22954 raft_consensus.cc:740] T 0aacf01c8a534c9da4886c3130df112f P 15ab8f217086450b933f077d59555f07 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 15ab8f217086450b933f077d59555f07, State: Initialized, Role: FOLLOWER
I20251209 08:58:02.978605 22954 consensus_queue.cc:260] T 0aacf01c8a534c9da4886c3130df112f P 15ab8f217086450b933f077d59555f07 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da42139421e54188b827cd6256e162b0" member_type: VOTER last_known_addr { host: "127.20.49.65" port: 46509 } } peers { permanent_uuid: "15ab8f217086450b933f077d59555f07" member_type: VOTER last_known_addr { host: "127.20.49.66" port: 45423 } } peers { permanent_uuid: "f2190f825aa04a0cb6b89cac0e770d12" member_type: VOTER last_known_addr { host: "127.20.49.67" port: 42299 } }
I20251209 08:58:02.979421 22953 tablet_bootstrap.cc:492] T 0aacf01c8a534c9da4886c3130df112f P da42139421e54188b827cd6256e162b0: No bootstrap required, opened a new log
I20251209 08:58:02.979524 22953 ts_tablet_manager.cc:1403] T 0aacf01c8a534c9da4886c3130df112f P da42139421e54188b827cd6256e162b0: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20251209 08:58:02.980017 22953 raft_consensus.cc:359] T 0aacf01c8a534c9da4886c3130df112f P da42139421e54188b827cd6256e162b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da42139421e54188b827cd6256e162b0" member_type: VOTER last_known_addr { host: "127.20.49.65" port: 46509 } } peers { permanent_uuid: "15ab8f217086450b933f077d59555f07" member_type: VOTER last_known_addr { host: "127.20.49.66" port: 45423 } } peers { permanent_uuid: "f2190f825aa04a0cb6b89cac0e770d12" member_type: VOTER last_known_addr { host: "127.20.49.67" port: 42299 } }
I20251209 08:58:02.980118 22953 raft_consensus.cc:385] T 0aacf01c8a534c9da4886c3130df112f P da42139421e54188b827cd6256e162b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251209 08:58:02.980149 22953 raft_consensus.cc:740] T 0aacf01c8a534c9da4886c3130df112f P da42139421e54188b827cd6256e162b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: da42139421e54188b827cd6256e162b0, State: Initialized, Role: FOLLOWER
I20251209 08:58:02.980268 22953 consensus_queue.cc:260] T 0aacf01c8a534c9da4886c3130df112f P da42139421e54188b827cd6256e162b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da42139421e54188b827cd6256e162b0" member_type: VOTER last_known_addr { host: "127.20.49.65" port: 46509 } } peers { permanent_uuid: "15ab8f217086450b933f077d59555f07" member_type: VOTER last_known_addr { host: "127.20.49.66" port: 45423 } } peers { permanent_uuid: "f2190f825aa04a0cb6b89cac0e770d12" member_type: VOTER last_known_addr { host: "127.20.49.67" port: 42299 } }
I20251209 08:58:02.980516 22953 ts_tablet_manager.cc:1434] T 0aacf01c8a534c9da4886c3130df112f P da42139421e54188b827cd6256e162b0: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20251209 08:58:02.981051 22954 ts_tablet_manager.cc:1434] T 0aacf01c8a534c9da4886c3130df112f P 15ab8f217086450b933f077d59555f07: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20251209 08:58:02.983394 23004 raft_consensus.cc:2272] T e98c48f6c144444c97e5450ffe99615d P 15ab8f217086450b933f077d59555f07 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251209 08:58:02.987058 23004 ts_tablet_manager.cc:1916] T e98c48f6c144444c97e5450ffe99615d P 15ab8f217086450b933f077d59555f07: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251209 08:58:02.988073 23005 ts_tablet_manager.cc:1929] T e98c48f6c144444c97e5450ffe99615d P f2190f825aa04a0cb6b89cac0e770d12: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20251209 08:58:02.989884 23006 ts_tablet_manager.cc:1929] T e98c48f6c144444c97e5450ffe99615d P da42139421e54188b827cd6256e162b0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20251209 08:58:02.988628 22956 tablet_bootstrap.cc:492] T 0aacf01c8a534c9da4886c3130df112f P f2190f825aa04a0cb6b89cac0e770d12: No bootstrap required, opened a new log
I20251209 08:58:02.990108 22956 ts_tablet_manager.cc:1403] T 0aacf01c8a534c9da4886c3130df112f P f2190f825aa04a0cb6b89cac0e770d12: Time spent bootstrapping tablet: real 0.020s user 0.002s sys 0.000s
I20251209 08:58:02.990684 23004 ts_tablet_manager.cc:1929] T e98c48f6c144444c97e5450ffe99615d P 15ab8f217086450b933f077d59555f07: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20251209 08:58:02.990697 22956 raft_consensus.cc:359] T 0aacf01c8a534c9da4886c3130df112f P f2190f825aa04a0cb6b89cac0e770d12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da42139421e54188b827cd6256e162b0" member_type: VOTER last_known_addr { host: "127.20.49.65" port: 46509 } } peers { permanent_uuid: "15ab8f217086450b933f077d59555f07" member_type: VOTER last_known_addr { host: "127.20.49.66" port: 45423 } } peers { permanent_uuid: "f2190f825aa04a0cb6b89cac0e770d12" member_type: VOTER last_known_addr { host: "127.20.49.67" port: 42299 } }
I20251209 08:58:02.990813 23004 log.cc:1199] T e98c48f6c144444c97e5450ffe99615d P 15ab8f217086450b933f077d59555f07: Deleting WAL directory at /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765270668420570-20677-0/minicluster-data/ts-1/wal/wals/e98c48f6c144444c97e5450ffe99615d
I20251209 08:58:02.990841 22956 raft_consensus.cc:385] T 0aacf01c8a534c9da4886c3130df112f P f2190f825aa04a0cb6b89cac0e770d12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251209 08:58:02.990880 22956 raft_consensus.cc:740] T 0aacf01c8a534c9da4886c3130df112f P f2190f825aa04a0cb6b89cac0e770d12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f2190f825aa04a0cb6b89cac0e770d12, State: Initialized, Role: FOLLOWER
I20251209 08:58:02.991015 22956 consensus_queue.cc:260] T 0aacf01c8a534c9da4886c3130df112f P f2190f825aa04a0cb6b89cac0e770d12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da42139421e54188b827cd6256e162b0" member_type: VOTER last_known_addr { host: "127.20.49.65" port: 46509 } } peers { permanent_uuid: "15ab8f217086450b933f077d59555f07" member_type: VOTER last_known_addr { host: "127.20.49.66" port: 45423 } } peers { permanent_uuid: "f2190f825aa04a0cb6b89cac0e770d12" member_type: VOTER last_known_addr { host: "127.20.49.67" port: 42299 } }
I20251209 08:58:02.991120 23004 ts_tablet_manager.cc:1950] T e98c48f6c144444c97e5450ffe99615d P 15ab8f217086450b933f077d59555f07: Deleting consensus metadata
I20251209 08:58:02.991312 22956 ts_tablet_manager.cc:1434] T 0aacf01c8a534c9da4886c3130df112f P f2190f825aa04a0cb6b89cac0e770d12: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20251209 08:58:02.992868 22427 catalog_manager.cc:4985] TS 15ab8f217086450b933f077d59555f07 (127.20.49.66:45423): tablet e98c48f6c144444c97e5450ffe99615d (table test-table [id=fd588d506ef541619d07881a13274efb]) successfully deleted
I20251209 08:58:02.993150 23005 log.cc:1199] T e98c48f6c144444c97e5450ffe99615d P f2190f825aa04a0cb6b89cac0e770d12: Deleting WAL directory at /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765270668420570-20677-0/minicluster-data/ts-2/wal/wals/e98c48f6c144444c97e5450ffe99615d
I20251209 08:58:02.993639 23005 ts_tablet_manager.cc:1950] T e98c48f6c144444c97e5450ffe99615d P f2190f825aa04a0cb6b89cac0e770d12: Deleting consensus metadata
I20251209 08:58:02.994904 23006 log.cc:1199] T e98c48f6c144444c97e5450ffe99615d P da42139421e54188b827cd6256e162b0: Deleting WAL directory at /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765270668420570-20677-0/minicluster-data/ts-0/wal/wals/e98c48f6c144444c97e5450ffe99615d
I20251209 08:58:02.994952 22426 catalog_manager.cc:4985] TS f2190f825aa04a0cb6b89cac0e770d12 (127.20.49.67:42299): tablet e98c48f6c144444c97e5450ffe99615d (table test-table [id=fd588d506ef541619d07881a13274efb]) successfully deleted
I20251209 08:58:02.995391 23006 ts_tablet_manager.cc:1950] T e98c48f6c144444c97e5450ffe99615d P da42139421e54188b827cd6256e162b0: Deleting consensus metadata
I20251209 08:58:02.996619 22426 catalog_manager.cc:4985] TS da42139421e54188b827cd6256e162b0 (127.20.49.65:46509): tablet e98c48f6c144444c97e5450ffe99615d (table test-table [id=fd588d506ef541619d07881a13274efb]) successfully deleted
I20251209 08:58:03.037612 22959 raft_consensus.cc:493] T 0aacf01c8a534c9da4886c3130df112f P 15ab8f217086450b933f077d59555f07 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251209 08:58:03.037797 22959 raft_consensus.cc:515] T 0aacf01c8a534c9da4886c3130df112f P 15ab8f217086450b933f077d59555f07 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da42139421e54188b827cd6256e162b0" member_type: VOTER last_known_addr { host: "127.20.49.65" port: 46509 } } peers { permanent_uuid: "15ab8f217086450b933f077d59555f07" member_type: VOTER last_known_addr { host: "127.20.49.66" port: 45423 } } peers { permanent_uuid: "f2190f825aa04a0cb6b89cac0e770d12" member_type: VOTER last_known_addr { host: "127.20.49.67" port: 42299 } }
I20251209 08:58:03.038250 22959 leader_election.cc:290] T 0aacf01c8a534c9da4886c3130df112f P 15ab8f217086450b933f077d59555f07 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers da42139421e54188b827cd6256e162b0 (127.20.49.65:46509), f2190f825aa04a0cb6b89cac0e770d12 (127.20.49.67:42299)
I20251209 08:58:03.038731 22600 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0aacf01c8a534c9da4886c3130df112f" candidate_uuid: "15ab8f217086450b933f077d59555f07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "da42139421e54188b827cd6256e162b0" is_pre_election: true
I20251209 08:58:03.038776 22884 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0aacf01c8a534c9da4886c3130df112f" candidate_uuid: "15ab8f217086450b933f077d59555f07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f2190f825aa04a0cb6b89cac0e770d12" is_pre_election: true
I20251209 08:58:03.038899 22884 raft_consensus.cc:2468] T 0aacf01c8a534c9da4886c3130df112f P f2190f825aa04a0cb6b89cac0e770d12 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 15ab8f217086450b933f077d59555f07 in term 0.
I20251209 08:58:03.038900 22600 raft_consensus.cc:2468] T 0aacf01c8a534c9da4886c3130df112f P da42139421e54188b827cd6256e162b0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 15ab8f217086450b933f077d59555f07 in term 0.
I20251209 08:58:03.039216 22674 leader_election.cc:304] T 0aacf01c8a534c9da4886c3130df112f P 15ab8f217086450b933f077d59555f07 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 15ab8f217086450b933f077d59555f07, da42139421e54188b827cd6256e162b0; no voters:
I20251209 08:58:03.039443 22959 raft_consensus.cc:2804] T 0aacf01c8a534c9da4886c3130df112f P 15ab8f217086450b933f077d59555f07 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251209 08:58:03.039521 22959 raft_consensus.cc:493] T 0aacf01c8a534c9da4886c3130df112f P 15ab8f217086450b933f077d59555f07 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251209 08:58:03.039561 22959 raft_consensus.cc:3060] T 0aacf01c8a534c9da4886c3130df112f P 15ab8f217086450b933f077d59555f07 [term 0 FOLLOWER]: Advancing to term 1
I20251209 08:58:03.040712 22959 raft_consensus.cc:515] T 0aacf01c8a534c9da4886c3130df112f P 15ab8f217086450b933f077d59555f07 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da42139421e54188b827cd6256e162b0" member_type: VOTER last_known_addr { host: "127.20.49.65" port: 46509 } } peers { permanent_uuid: "15ab8f217086450b933f077d59555f07" member_type: VOTER last_known_addr { host: "127.20.49.66" port: 45423 } } peers { permanent_uuid: "f2190f825aa04a0cb6b89cac0e770d12" member_type: VOTER last_known_addr { host: "127.20.49.67" port: 42299 } }
I20251209 08:58:03.041069 22959 leader_election.cc:290] T 0aacf01c8a534c9da4886c3130df112f P 15ab8f217086450b933f077d59555f07 [CANDIDATE]: Term 1 election: Requested vote from peers da42139421e54188b827cd6256e162b0 (127.20.49.65:46509), f2190f825aa04a0cb6b89cac0e770d12 (127.20.49.67:42299)
I20251209 08:58:03.041601 22884 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0aacf01c8a534c9da4886c3130df112f" candidate_uuid: "15ab8f217086450b933f077d59555f07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f2190f825aa04a0cb6b89cac0e770d12"
I20251209 08:58:03.041749 22884 raft_consensus.cc:3060] T 0aacf01c8a534c9da4886c3130df112f P f2190f825aa04a0cb6b89cac0e770d12 [term 0 FOLLOWER]: Advancing to term 1
I20251209 08:58:03.041839 22600 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0aacf01c8a534c9da4886c3130df112f" candidate_uuid: "15ab8f217086450b933f077d59555f07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "da42139421e54188b827cd6256e162b0"
I20251209 08:58:03.041960 22600 raft_consensus.cc:3060] T 0aacf01c8a534c9da4886c3130df112f P da42139421e54188b827cd6256e162b0 [term 0 FOLLOWER]: Advancing to term 1
I20251209 08:58:03.042901 22884 raft_consensus.cc:2468] T 0aacf01c8a534c9da4886c3130df112f P f2190f825aa04a0cb6b89cac0e770d12 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 15ab8f217086450b933f077d59555f07 in term 1.
I20251209 08:58:03.043061 22600 raft_consensus.cc:2468] T 0aacf01c8a534c9da4886c3130df112f P da42139421e54188b827cd6256e162b0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 15ab8f217086450b933f077d59555f07 in term 1.
I20251209 08:58:03.043293 22674 leader_election.cc:304] T 0aacf01c8a534c9da4886c3130df112f P 15ab8f217086450b933f077d59555f07 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 15ab8f217086450b933f077d59555f07, f2190f825aa04a0cb6b89cac0e770d12; no voters:
I20251209 08:58:03.043483 22959 raft_consensus.cc:2804] T 0aacf01c8a534c9da4886c3130df112f P 15ab8f217086450b933f077d59555f07 [term 1 FOLLOWER]: Leader election won for term 1
I20251209 08:58:03.043581 22959 raft_consensus.cc:697] T 0aacf01c8a534c9da4886c3130df112f P 15ab8f217086450b933f077d59555f07 [term 1 LEADER]: Becoming Leader. State: Replica: 15ab8f217086450b933f077d59555f07, State: Running, Role: LEADER
I20251209 08:58:03.043735 22959 consensus_queue.cc:237] T 0aacf01c8a534c9da4886c3130df112f P 15ab8f217086450b933f077d59555f07 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da42139421e54188b827cd6256e162b0" member_type: VOTER last_known_addr { host: "127.20.49.65" port: 46509 } } peers { permanent_uuid: "15ab8f217086450b933f077d59555f07" member_type: VOTER last_known_addr { host: "127.20.49.66" port: 45423 } } peers { permanent_uuid: "f2190f825aa04a0cb6b89cac0e770d12" member_type: VOTER last_known_addr { host: "127.20.49.67" port: 42299 } }
I20251209 08:58:03.045936 22441 catalog_manager.cc:5654] T 0aacf01c8a534c9da4886c3130df112f P 15ab8f217086450b933f077d59555f07 reported cstate change: term changed from 0 to 1, leader changed from <none> to 15ab8f217086450b933f077d59555f07 (127.20.49.66). New cstate: current_term: 1 leader_uuid: "15ab8f217086450b933f077d59555f07" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da42139421e54188b827cd6256e162b0" member_type: VOTER last_known_addr { host: "127.20.49.65" port: 46509 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "15ab8f217086450b933f077d59555f07" member_type: VOTER last_known_addr { host: "127.20.49.66" port: 45423 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f2190f825aa04a0cb6b89cac0e770d12" member_type: VOTER last_known_addr { host: "127.20.49.67" port: 42299 } health_report { overall_health: UNKNOWN } } }
I20251209 08:58:03.066854 22884 raft_consensus.cc:1275] T 0aacf01c8a534c9da4886c3130df112f P f2190f825aa04a0cb6b89cac0e770d12 [term 1 FOLLOWER]: Refusing update from remote peer 15ab8f217086450b933f077d59555f07: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251209 08:58:03.066849 22600 raft_consensus.cc:1275] T 0aacf01c8a534c9da4886c3130df112f P da42139421e54188b827cd6256e162b0 [term 1 FOLLOWER]: Refusing update from remote peer 15ab8f217086450b933f077d59555f07: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251209 08:58:03.067229 22959 consensus_queue.cc:1048] T 0aacf01c8a534c9da4886c3130df112f P 15ab8f217086450b933f077d59555f07 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f2190f825aa04a0cb6b89cac0e770d12" member_type: VOTER last_known_addr { host: "127.20.49.67" port: 42299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251209 08:58:03.067390 22998 consensus_queue.cc:1048] T 0aacf01c8a534c9da4886c3130df112f P 15ab8f217086450b933f077d59555f07 [LEADER]: Connected to new peer: Peer: permanent_uuid: "da42139421e54188b827cd6256e162b0" member_type: VOTER last_known_addr { host: "127.20.49.65" port: 46509 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251209 08:58:03.070297 22983 mvcc.cc:204] Tried to move back new op lower bound from 7230548717838540800 to 7230548717749616640. Current Snapshot: MvccSnapshot[applied={T|T < 7230548717838540800}]
W20251209 08:58:03.084328 22982 tablet_replica.cc:1306] Aborted: operation has been aborted: cancelling pending write operations
I20251209 08:58:04.639539 22441 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42090:
table { table_name: "test-table" } modify_external_catalogs: true
I20251209 08:58:04.639693 22441 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42090:
table { table_name: "test-table" } modify_external_catalogs: true
I20251209 08:58:04.642040 22441 catalog_manager.cc:5941] T 00000000000000000000000000000000 P 39341759984c48989ceba01b8acb2da9: Sending DeleteTablet for 3 replicas of tablet 0aacf01c8a534c9da4886c3130df112f
I20251209 08:58:04.642633 22580 tablet_service.cc:1552] Processing DeleteTablet for tablet 0aacf01c8a534c9da4886c3130df112f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-09 08:58:04 UTC) from {username='kudu', principal='kudu/127.20.49.126@KRBTEST.COM'} at 127.0.0.1:44186
I20251209 08:58:04.642688 22722 tablet_service.cc:1552] Processing DeleteTablet for tablet 0aacf01c8a534c9da4886c3130df112f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-09 08:58:04 UTC) from {username='kudu', principal='kudu/127.20.49.126@KRBTEST.COM'} at 127.0.0.1:33058
I20251209 08:58:04.642686 22864 tablet_service.cc:1552] Processing DeleteTablet for tablet 0aacf01c8a534c9da4886c3130df112f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-09 08:58:04 UTC) from {username='kudu', principal='kudu/127.20.49.126@KRBTEST.COM'} at 127.0.0.1:57140
I20251209 08:58:04.643023 23036 tablet_replica.cc:333] T 0aacf01c8a534c9da4886c3130df112f P f2190f825aa04a0cb6b89cac0e770d12: stopping tablet replica
I20251209 08:58:04.643047 23035 tablet_replica.cc:333] T 0aacf01c8a534c9da4886c3130df112f P da42139421e54188b827cd6256e162b0: stopping tablet replica
I20251209 08:58:04.643157 23035 raft_consensus.cc:2243] T 0aacf01c8a534c9da4886c3130df112f P da42139421e54188b827cd6256e162b0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251209 08:58:04.643157 23036 raft_consensus.cc:2243] T 0aacf01c8a534c9da4886c3130df112f P f2190f825aa04a0cb6b89cac0e770d12 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251209 08:58:04.643278 23035 raft_consensus.cc:2272] T 0aacf01c8a534c9da4886c3130df112f P da42139421e54188b827cd6256e162b0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251209 08:58:04.643280 23036 raft_consensus.cc:2272] T 0aacf01c8a534c9da4886c3130df112f P f2190f825aa04a0cb6b89cac0e770d12 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251209 08:58:04.643565 23037 tablet_replica.cc:333] T 0aacf01c8a534c9da4886c3130df112f P 15ab8f217086450b933f077d59555f07: stopping tablet replica
I20251209 08:58:04.643705 23037 raft_consensus.cc:2243] T 0aacf01c8a534c9da4886c3130df112f P 15ab8f217086450b933f077d59555f07 [term 1 LEADER]: Raft consensus shutting down.
I20251209 08:58:04.643913 23037 raft_consensus.cc:2272] T 0aacf01c8a534c9da4886c3130df112f P 15ab8f217086450b933f077d59555f07 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251209 08:58:04.643971 23036 ts_tablet_manager.cc:1916] T 0aacf01c8a534c9da4886c3130df112f P f2190f825aa04a0cb6b89cac0e770d12: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251209 08:58:04.644716 23037 ts_tablet_manager.cc:1916] T 0aacf01c8a534c9da4886c3130df112f P 15ab8f217086450b933f077d59555f07: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251209 08:58:04.646438 23036 ts_tablet_manager.cc:1929] T 0aacf01c8a534c9da4886c3130df112f P f2190f825aa04a0cb6b89cac0e770d12: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20251209 08:58:04.646513 23036 log.cc:1199] T 0aacf01c8a534c9da4886c3130df112f P f2190f825aa04a0cb6b89cac0e770d12: Deleting WAL directory at /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765270668420570-20677-0/minicluster-data/ts-2/wal/wals/0aacf01c8a534c9da4886c3130df112f
I20251209 08:58:04.646773 23036 ts_tablet_manager.cc:1950] T 0aacf01c8a534c9da4886c3130df112f P f2190f825aa04a0cb6b89cac0e770d12: Deleting consensus metadata
I20251209 08:58:04.647290 23037 ts_tablet_manager.cc:1929] T 0aacf01c8a534c9da4886c3130df112f P 15ab8f217086450b933f077d59555f07: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20251209 08:58:04.647398 23037 log.cc:1199] T 0aacf01c8a534c9da4886c3130df112f P 15ab8f217086450b933f077d59555f07: Deleting WAL directory at /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765270668420570-20677-0/minicluster-data/ts-1/wal/wals/0aacf01c8a534c9da4886c3130df112f
I20251209 08:58:04.647509 22426 catalog_manager.cc:4985] TS f2190f825aa04a0cb6b89cac0e770d12 (127.20.49.67:42299): tablet 0aacf01c8a534c9da4886c3130df112f (table test-table [id=179d208d15f04a3c8d21dcffbf5bf06e]) successfully deleted
I20251209 08:58:04.647749 23037 ts_tablet_manager.cc:1950] T 0aacf01c8a534c9da4886c3130df112f P 15ab8f217086450b933f077d59555f07: Deleting consensus metadata
I20251209 08:58:04.648381 22427 catalog_manager.cc:4985] TS 15ab8f217086450b933f077d59555f07 (127.20.49.66:45423): tablet 0aacf01c8a534c9da4886c3130df112f (table test-table [id=179d208d15f04a3c8d21dcffbf5bf06e]) successfully deleted
I20251209 08:58:04.650286 23035 ts_tablet_manager.cc:1916] T 0aacf01c8a534c9da4886c3130df112f P da42139421e54188b827cd6256e162b0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251209 08:58:04.652945 23035 ts_tablet_manager.cc:1929] T 0aacf01c8a534c9da4886c3130df112f P da42139421e54188b827cd6256e162b0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20251209 08:58:04.653019 23035 log.cc:1199] T 0aacf01c8a534c9da4886c3130df112f P da42139421e54188b827cd6256e162b0: Deleting WAL directory at /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765270668420570-20677-0/minicluster-data/ts-0/wal/wals/0aacf01c8a534c9da4886c3130df112f
I20251209 08:58:04.653262 23035 ts_tablet_manager.cc:1950] T 0aacf01c8a534c9da4886c3130df112f P da42139421e54188b827cd6256e162b0: Deleting consensus metadata
I20251209 08:58:04.654021 22426 catalog_manager.cc:4985] TS da42139421e54188b827cd6256e162b0 (127.20.49.65:46509): tablet 0aacf01c8a534c9da4886c3130df112f (table test-table [id=179d208d15f04a3c8d21dcffbf5bf06e]) successfully deleted
Dec 09 08:58:04 dist-test-slave-8w6c krb5kdc[22393](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765270684, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Dec 09 08:58:04 dist-test-slave-8w6c krb5kdc[22393](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765270684, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.20.49.126@KRBTEST.COM
I20251209 08:58:04.674844 23045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:58:04.666969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42098 (local address 127.20.49.126:39233)
1209 08:58:04.667271 (+ 302us) server_negotiation.cc:207] Beginning negotiation
1209 08:58:04.667274 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1209 08:58:04.667291 (+ 17us) server_negotiation.cc:408] Connection header received
1209 08:58:04.667416 (+ 125us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1209 08:58:04.667418 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1209 08:58:04.667465 (+ 47us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1209 08:58:04.667550 (+ 85us) server_negotiation.cc:227] Negotiated authn=SASL
1209 08:58:04.668315 (+ 765us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:58:04.668841 (+ 526us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1209 08:58:04.669564 (+ 723us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:58:04.669712 (+ 148us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:58:04.671849 (+ 2137us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1209 08:58:04.671868 (+ 19us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1209 08:58:04.671870 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1209 08:58:04.671895 (+ 25us) server_negotiation.cc:923] calling sasl_server_start()
1209 08:58:04.673395 (+ 1500us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1209 08:58:04.673806 (+ 411us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1209 08:58:04.673809 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1209 08:58:04.673811 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1209 08:58:04.673855 (+ 44us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1209 08:58:04.674054 (+ 199us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1209 08:58:04.674056 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1209 08:58:04.674057 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1209 08:58:04.674210 (+ 153us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1209 08:58:04.674312 (+ 102us) server_negotiation.cc:1036] Waiting for connection context
1209 08:58:04.674516 (+ 204us) server_negotiation.cc:300] Negotiation successful
1209 08:58:04.674630 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":212,"thread_start_us":106,"threads_started":1}
I20251209 08:58:04.677342 22441 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42098:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251209 08:58:04.677819 22441 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251209 08:58:04.682622 22580 tablet_service.cc:1505] Processing CreateTablet for tablet e1edac22722d44fea96fb4b9bb05b4a1 (DEFAULT_TABLE table=test-table [id=f1ceb51a02254e39b0557363ee1f3671]), partition=RANGE (key) PARTITION UNBOUNDED
I20251209 08:58:04.682695 22722 tablet_service.cc:1505] Processing CreateTablet for tablet e1edac22722d44fea96fb4b9bb05b4a1 (DEFAULT_TABLE table=test-table [id=f1ceb51a02254e39b0557363ee1f3671]), partition=RANGE (key) PARTITION UNBOUNDED
I20251209 08:58:04.682960 22722 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e1edac22722d44fea96fb4b9bb05b4a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251209 08:58:04.682960 22580 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e1edac22722d44fea96fb4b9bb05b4a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251209 08:58:04.682994 22864 tablet_service.cc:1505] Processing CreateTablet for tablet e1edac22722d44fea96fb4b9bb05b4a1 (DEFAULT_TABLE table=test-table [id=f1ceb51a02254e39b0557363ee1f3671]), partition=RANGE (key) PARTITION UNBOUNDED
I20251209 08:58:04.683259 22864 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e1edac22722d44fea96fb4b9bb05b4a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251209 08:58:04.685523 23047 tablet_bootstrap.cc:492] T e1edac22722d44fea96fb4b9bb05b4a1 P da42139421e54188b827cd6256e162b0: Bootstrap starting.
I20251209 08:58:04.685889 23048 tablet_bootstrap.cc:492] T e1edac22722d44fea96fb4b9bb05b4a1 P 15ab8f217086450b933f077d59555f07: Bootstrap starting.
I20251209 08:58:04.686514 23047 tablet_bootstrap.cc:654] T e1edac22722d44fea96fb4b9bb05b4a1 P da42139421e54188b827cd6256e162b0: Neither blocks nor log segments found. Creating new log.
I20251209 08:58:04.686843 23048 tablet_bootstrap.cc:654] T e1edac22722d44fea96fb4b9bb05b4a1 P 15ab8f217086450b933f077d59555f07: Neither blocks nor log segments found. Creating new log.
I20251209 08:58:04.687244 23046 tablet_bootstrap.cc:492] T e1edac22722d44fea96fb4b9bb05b4a1 P f2190f825aa04a0cb6b89cac0e770d12: Bootstrap starting.
I20251209 08:58:04.688222 23048 tablet_bootstrap.cc:492] T e1edac22722d44fea96fb4b9bb05b4a1 P 15ab8f217086450b933f077d59555f07: No bootstrap required, opened a new log
I20251209 08:58:04.688326 23048 ts_tablet_manager.cc:1403] T e1edac22722d44fea96fb4b9bb05b4a1 P 15ab8f217086450b933f077d59555f07: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20251209 08:58:04.688355 23047 tablet_bootstrap.cc:492] T e1edac22722d44fea96fb4b9bb05b4a1 P da42139421e54188b827cd6256e162b0: No bootstrap required, opened a new log
I20251209 08:58:04.688423 23047 ts_tablet_manager.cc:1403] T e1edac22722d44fea96fb4b9bb05b4a1 P da42139421e54188b827cd6256e162b0: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20251209 08:58:04.688779 23048 raft_consensus.cc:359] T e1edac22722d44fea96fb4b9bb05b4a1 P 15ab8f217086450b933f077d59555f07 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da42139421e54188b827cd6256e162b0" member_type: VOTER last_known_addr { host: "127.20.49.65" port: 46509 } } peers { permanent_uuid: "15ab8f217086450b933f077d59555f07" member_type: VOTER last_known_addr { host: "127.20.49.66" port: 45423 } } peers { permanent_uuid: "f2190f825aa04a0cb6b89cac0e770d12" member_type: VOTER last_known_addr { host: "127.20.49.67" port: 42299 } }
I20251209 08:58:04.688803 23047 raft_consensus.cc:359] T e1edac22722d44fea96fb4b9bb05b4a1 P da42139421e54188b827cd6256e162b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da42139421e54188b827cd6256e162b0" member_type: VOTER last_known_addr { host: "127.20.49.65" port: 46509 } } peers { permanent_uuid: "15ab8f217086450b933f077d59555f07" member_type: VOTER last_known_addr { host: "127.20.49.66" port: 45423 } } peers { permanent_uuid: "f2190f825aa04a0cb6b89cac0e770d12" member_type: VOTER last_known_addr { host: "127.20.49.67" port: 42299 } }
I20251209 08:58:04.688915 23048 raft_consensus.cc:385] T e1edac22722d44fea96fb4b9bb05b4a1 P 15ab8f217086450b933f077d59555f07 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251209 08:58:04.688915 23047 raft_consensus.cc:385] T e1edac22722d44fea96fb4b9bb05b4a1 P da42139421e54188b827cd6256e162b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251209 08:58:04.688973 23047 raft_consensus.cc:740] T e1edac22722d44fea96fb4b9bb05b4a1 P da42139421e54188b827cd6256e162b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: da42139421e54188b827cd6256e162b0, State: Initialized, Role: FOLLOWER
I20251209 08:58:04.688973 23048 raft_consensus.cc:740] T e1edac22722d44fea96fb4b9bb05b4a1 P 15ab8f217086450b933f077d59555f07 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 15ab8f217086450b933f077d59555f07, State: Initialized, Role: FOLLOWER
I20251209 08:58:04.689119 23048 consensus_queue.cc:260] T e1edac22722d44fea96fb4b9bb05b4a1 P 15ab8f217086450b933f077d59555f07 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da42139421e54188b827cd6256e162b0" member_type: VOTER last_known_addr { host: "127.20.49.65" port: 46509 } } peers { permanent_uuid: "15ab8f217086450b933f077d59555f07" member_type: VOTER last_known_addr { host: "127.20.49.66" port: 45423 } } peers { permanent_uuid: "f2190f825aa04a0cb6b89cac0e770d12" member_type: VOTER last_known_addr { host: "127.20.49.67" port: 42299 } }
I20251209 08:58:04.689421 23046 tablet_bootstrap.cc:654] T e1edac22722d44fea96fb4b9bb05b4a1 P f2190f825aa04a0cb6b89cac0e770d12: Neither blocks nor log segments found. Creating new log.
I20251209 08:58:04.689436 23048 ts_tablet_manager.cc:1434] T e1edac22722d44fea96fb4b9bb05b4a1 P 15ab8f217086450b933f077d59555f07: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20251209 08:58:04.690378 23047 consensus_queue.cc:260] T e1edac22722d44fea96fb4b9bb05b4a1 P da42139421e54188b827cd6256e162b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da42139421e54188b827cd6256e162b0" member_type: VOTER last_known_addr { host: "127.20.49.65" port: 46509 } } peers { permanent_uuid: "15ab8f217086450b933f077d59555f07" member_type: VOTER last_known_addr { host: "127.20.49.66" port: 45423 } } peers { permanent_uuid: "f2190f825aa04a0cb6b89cac0e770d12" member_type: VOTER last_known_addr { host: "127.20.49.67" port: 42299 } }
I20251209 08:58:04.690793 23047 ts_tablet_manager.cc:1434] T e1edac22722d44fea96fb4b9bb05b4a1 P da42139421e54188b827cd6256e162b0: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251209 08:58:04.691309 23046 tablet_bootstrap.cc:492] T e1edac22722d44fea96fb4b9bb05b4a1 P f2190f825aa04a0cb6b89cac0e770d12: No bootstrap required, opened a new log
I20251209 08:58:04.691394 23046 ts_tablet_manager.cc:1403] T e1edac22722d44fea96fb4b9bb05b4a1 P f2190f825aa04a0cb6b89cac0e770d12: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20251209 08:58:04.691903 23046 raft_consensus.cc:359] T e1edac22722d44fea96fb4b9bb05b4a1 P f2190f825aa04a0cb6b89cac0e770d12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da42139421e54188b827cd6256e162b0" member_type: VOTER last_known_addr { host: "127.20.49.65" port: 46509 } } peers { permanent_uuid: "15ab8f217086450b933f077d59555f07" member_type: VOTER last_known_addr { host: "127.20.49.66" port: 45423 } } peers { permanent_uuid: "f2190f825aa04a0cb6b89cac0e770d12" member_type: VOTER last_known_addr { host: "127.20.49.67" port: 42299 } }
I20251209 08:58:04.692023 23046 raft_consensus.cc:385] T e1edac22722d44fea96fb4b9bb05b4a1 P f2190f825aa04a0cb6b89cac0e770d12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251209 08:58:04.692063 23046 raft_consensus.cc:740] T e1edac22722d44fea96fb4b9bb05b4a1 P f2190f825aa04a0cb6b89cac0e770d12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f2190f825aa04a0cb6b89cac0e770d12, State: Initialized, Role: FOLLOWER
I20251209 08:58:04.692206 23046 consensus_queue.cc:260] T e1edac22722d44fea96fb4b9bb05b4a1 P f2190f825aa04a0cb6b89cac0e770d12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da42139421e54188b827cd6256e162b0" member_type: VOTER last_known_addr { host: "127.20.49.65" port: 46509 } } peers { permanent_uuid: "15ab8f217086450b933f077d59555f07" member_type: VOTER last_known_addr { host: "127.20.49.66" port: 45423 } } peers { permanent_uuid: "f2190f825aa04a0cb6b89cac0e770d12" member_type: VOTER last_known_addr { host: "127.20.49.67" port: 42299 } }
I20251209 08:58:04.692600 23046 ts_tablet_manager.cc:1434] T e1edac22722d44fea96fb4b9bb05b4a1 P f2190f825aa04a0cb6b89cac0e770d12: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20251209 08:58:04.761883 22959 raft_consensus.cc:493] T e1edac22722d44fea96fb4b9bb05b4a1 P 15ab8f217086450b933f077d59555f07 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251209 08:58:04.762017 22959 raft_consensus.cc:515] T e1edac22722d44fea96fb4b9bb05b4a1 P 15ab8f217086450b933f077d59555f07 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da42139421e54188b827cd6256e162b0" member_type: VOTER last_known_addr { host: "127.20.49.65" port: 46509 } } peers { permanent_uuid: "15ab8f217086450b933f077d59555f07" member_type: VOTER last_known_addr { host: "127.20.49.66" port: 45423 } } peers { permanent_uuid: "f2190f825aa04a0cb6b89cac0e770d12" member_type: VOTER last_known_addr { host: "127.20.49.67" port: 42299 } }
I20251209 08:58:04.762478 22959 leader_election.cc:290] T e1edac22722d44fea96fb4b9bb05b4a1 P 15ab8f217086450b933f077d59555f07 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers da42139421e54188b827cd6256e162b0 (127.20.49.65:46509), f2190f825aa04a0cb6b89cac0e770d12 (127.20.49.67:42299)
I20251209 08:58:04.762969 22600 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e1edac22722d44fea96fb4b9bb05b4a1" candidate_uuid: "15ab8f217086450b933f077d59555f07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "da42139421e54188b827cd6256e162b0" is_pre_election: true
I20251209 08:58:04.762998 22884 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e1edac22722d44fea96fb4b9bb05b4a1" candidate_uuid: "15ab8f217086450b933f077d59555f07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f2190f825aa04a0cb6b89cac0e770d12" is_pre_election: true
I20251209 08:58:04.763113 22600 raft_consensus.cc:2468] T e1edac22722d44fea96fb4b9bb05b4a1 P da42139421e54188b827cd6256e162b0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 15ab8f217086450b933f077d59555f07 in term 0.
I20251209 08:58:04.763139 22884 raft_consensus.cc:2468] T e1edac22722d44fea96fb4b9bb05b4a1 P f2190f825aa04a0cb6b89cac0e770d12 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 15ab8f217086450b933f077d59555f07 in term 0.
I20251209 08:58:04.763546 22674 leader_election.cc:304] T e1edac22722d44fea96fb4b9bb05b4a1 P 15ab8f217086450b933f077d59555f07 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 15ab8f217086450b933f077d59555f07, f2190f825aa04a0cb6b89cac0e770d12; no voters:
I20251209 08:58:04.763758 22959 raft_consensus.cc:2804] T e1edac22722d44fea96fb4b9bb05b4a1 P 15ab8f217086450b933f077d59555f07 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251209 08:58:04.763821 22959 raft_consensus.cc:493] T e1edac22722d44fea96fb4b9bb05b4a1 P 15ab8f217086450b933f077d59555f07 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251209 08:58:04.763876 22959 raft_consensus.cc:3060] T e1edac22722d44fea96fb4b9bb05b4a1 P 15ab8f217086450b933f077d59555f07 [term 0 FOLLOWER]: Advancing to term 1
I20251209 08:58:04.764792 22959 raft_consensus.cc:515] T e1edac22722d44fea96fb4b9bb05b4a1 P 15ab8f217086450b933f077d59555f07 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da42139421e54188b827cd6256e162b0" member_type: VOTER last_known_addr { host: "127.20.49.65" port: 46509 } } peers { permanent_uuid: "15ab8f217086450b933f077d59555f07" member_type: VOTER last_known_addr { host: "127.20.49.66" port: 45423 } } peers { permanent_uuid: "f2190f825aa04a0cb6b89cac0e770d12" member_type: VOTER last_known_addr { host: "127.20.49.67" port: 42299 } }
I20251209 08:58:04.765106 22959 leader_election.cc:290] T e1edac22722d44fea96fb4b9bb05b4a1 P 15ab8f217086450b933f077d59555f07 [CANDIDATE]: Term 1 election: Requested vote from peers da42139421e54188b827cd6256e162b0 (127.20.49.65:46509), f2190f825aa04a0cb6b89cac0e770d12 (127.20.49.67:42299)
I20251209 08:58:04.765503 22600 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e1edac22722d44fea96fb4b9bb05b4a1" candidate_uuid: "15ab8f217086450b933f077d59555f07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "da42139421e54188b827cd6256e162b0"
I20251209 08:58:04.765533 22884 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e1edac22722d44fea96fb4b9bb05b4a1" candidate_uuid: "15ab8f217086450b933f077d59555f07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f2190f825aa04a0cb6b89cac0e770d12"
I20251209 08:58:04.765615 22600 raft_consensus.cc:3060] T e1edac22722d44fea96fb4b9bb05b4a1 P da42139421e54188b827cd6256e162b0 [term 0 FOLLOWER]: Advancing to term 1
I20251209 08:58:04.765648 22884 raft_consensus.cc:3060] T e1edac22722d44fea96fb4b9bb05b4a1 P f2190f825aa04a0cb6b89cac0e770d12 [term 0 FOLLOWER]: Advancing to term 1
I20251209 08:58:04.766713 22884 raft_consensus.cc:2468] T e1edac22722d44fea96fb4b9bb05b4a1 P f2190f825aa04a0cb6b89cac0e770d12 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 15ab8f217086450b933f077d59555f07 in term 1.
I20251209 08:58:04.766744 22600 raft_consensus.cc:2468] T e1edac22722d44fea96fb4b9bb05b4a1 P da42139421e54188b827cd6256e162b0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 15ab8f217086450b933f077d59555f07 in term 1.
I20251209 08:58:04.767081 22674 leader_election.cc:304] T e1edac22722d44fea96fb4b9bb05b4a1 P 15ab8f217086450b933f077d59555f07 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 15ab8f217086450b933f077d59555f07, da42139421e54188b827cd6256e162b0; no voters:
I20251209 08:58:04.767311 22959 raft_consensus.cc:2804] T e1edac22722d44fea96fb4b9bb05b4a1 P 15ab8f217086450b933f077d59555f07 [term 1 FOLLOWER]: Leader election won for term 1
I20251209 08:58:04.767417 22959 raft_consensus.cc:697] T e1edac22722d44fea96fb4b9bb05b4a1 P 15ab8f217086450b933f077d59555f07 [term 1 LEADER]: Becoming Leader. State: Replica: 15ab8f217086450b933f077d59555f07, State: Running, Role: LEADER
I20251209 08:58:04.767566 22959 consensus_queue.cc:237] T e1edac22722d44fea96fb4b9bb05b4a1 P 15ab8f217086450b933f077d59555f07 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da42139421e54188b827cd6256e162b0" member_type: VOTER last_known_addr { host: "127.20.49.65" port: 46509 } } peers { permanent_uuid: "15ab8f217086450b933f077d59555f07" member_type: VOTER last_known_addr { host: "127.20.49.66" port: 45423 } } peers { permanent_uuid: "f2190f825aa04a0cb6b89cac0e770d12" member_type: VOTER last_known_addr { host: "127.20.49.67" port: 42299 } }
I20251209 08:58:04.769285 22440 catalog_manager.cc:5654] T e1edac22722d44fea96fb4b9bb05b4a1 P 15ab8f217086450b933f077d59555f07 reported cstate change: term changed from 0 to 1, leader changed from <none> to 15ab8f217086450b933f077d59555f07 (127.20.49.66). New cstate: current_term: 1 leader_uuid: "15ab8f217086450b933f077d59555f07" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da42139421e54188b827cd6256e162b0" member_type: VOTER last_known_addr { host: "127.20.49.65" port: 46509 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "15ab8f217086450b933f077d59555f07" member_type: VOTER last_known_addr { host: "127.20.49.66" port: 45423 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f2190f825aa04a0cb6b89cac0e770d12" member_type: VOTER last_known_addr { host: "127.20.49.67" port: 42299 } health_report { overall_health: UNKNOWN } } }
I20251209 08:58:04.791592 23055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:58:04.789077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33074 (local address 127.20.49.66:45423)
1209 08:58:04.789315 (+ 238us) server_negotiation.cc:207] Beginning negotiation
1209 08:58:04.789319 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1209 08:58:04.789332 (+ 13us) server_negotiation.cc:408] Connection header received
1209 08:58:04.789383 (+ 51us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1209 08:58:04.789385 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1209 08:58:04.789428 (+ 43us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1209 08:58:04.789501 (+ 73us) server_negotiation.cc:227] Negotiated authn=TOKEN
1209 08:58:04.789862 (+ 361us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:58:04.790351 (+ 489us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1209 08:58:04.791000 (+ 649us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:58:04.791141 (+ 141us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:58:04.791179 (+ 38us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1209 08:58:04.791241 (+ 62us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1209 08:58:04.791310 (+ 69us) server_negotiation.cc:1036] Waiting for connection context
1209 08:58:04.791428 (+ 118us) server_negotiation.cc:300] Negotiation successful
1209 08:58:04.791478 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":170,"thread_start_us":98,"threads_started":1}
I20251209 08:58:04.797575 22884 raft_consensus.cc:1275] T e1edac22722d44fea96fb4b9bb05b4a1 P f2190f825aa04a0cb6b89cac0e770d12 [term 1 FOLLOWER]: Refusing update from remote peer 15ab8f217086450b933f077d59555f07: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251209 08:58:04.797591 22600 raft_consensus.cc:1275] T e1edac22722d44fea96fb4b9bb05b4a1 P da42139421e54188b827cd6256e162b0 [term 1 FOLLOWER]: Refusing update from remote peer 15ab8f217086450b933f077d59555f07: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251209 08:58:04.797991 23030 consensus_queue.cc:1048] T e1edac22722d44fea96fb4b9bb05b4a1 P 15ab8f217086450b933f077d59555f07 [LEADER]: Connected to new peer: Peer: permanent_uuid: "da42139421e54188b827cd6256e162b0" member_type: VOTER last_known_addr { host: "127.20.49.65" port: 46509 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251209 08:58:04.798125 22959 consensus_queue.cc:1048] T e1edac22722d44fea96fb4b9bb05b4a1 P 15ab8f217086450b933f077d59555f07 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f2190f825aa04a0cb6b89cac0e770d12" member_type: VOTER last_known_addr { host: "127.20.49.67" port: 42299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251209 08:58:04.828130 22439 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42098:
table { table_name: "test-table" } modify_external_catalogs: true
I20251209 08:58:04.828272 22439 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42098:
table { table_name: "test-table" } modify_external_catalogs: true
I20251209 08:58:04.830168 22439 catalog_manager.cc:5941] T 00000000000000000000000000000000 P 39341759984c48989ceba01b8acb2da9: Sending DeleteTablet for 3 replicas of tablet e1edac22722d44fea96fb4b9bb05b4a1
I20251209 08:58:04.830713 22580 tablet_service.cc:1552] Processing DeleteTablet for tablet e1edac22722d44fea96fb4b9bb05b4a1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-09 08:58:04 UTC) from {username='kudu', principal='kudu/127.20.49.126@KRBTEST.COM'} at 127.0.0.1:44186
I20251209 08:58:04.830746 22864 tablet_service.cc:1552] Processing DeleteTablet for tablet e1edac22722d44fea96fb4b9bb05b4a1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-09 08:58:04 UTC) from {username='kudu', principal='kudu/127.20.49.126@KRBTEST.COM'} at 127.0.0.1:57140
I20251209 08:58:04.830788 22722 tablet_service.cc:1552] Processing DeleteTablet for tablet e1edac22722d44fea96fb4b9bb05b4a1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-09 08:58:04 UTC) from {username='kudu', principal='kudu/127.20.49.126@KRBTEST.COM'} at 127.0.0.1:33058
I20251209 08:58:04.830904 23035 tablet_replica.cc:333] T e1edac22722d44fea96fb4b9bb05b4a1 P da42139421e54188b827cd6256e162b0: stopping tablet replica
I20251209 08:58:04.830917 23036 tablet_replica.cc:333] T e1edac22722d44fea96fb4b9bb05b4a1 P f2190f825aa04a0cb6b89cac0e770d12: stopping tablet replica
I20251209 08:58:04.830956 23037 tablet_replica.cc:333] T e1edac22722d44fea96fb4b9bb05b4a1 P 15ab8f217086450b933f077d59555f07: stopping tablet replica
I20251209 08:58:04.831009 23036 raft_consensus.cc:2243] T e1edac22722d44fea96fb4b9bb05b4a1 P f2190f825aa04a0cb6b89cac0e770d12 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251209 08:58:04.831009 23035 raft_consensus.cc:2243] T e1edac22722d44fea96fb4b9bb05b4a1 P da42139421e54188b827cd6256e162b0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251209 08:58:04.831076 23037 raft_consensus.cc:2243] T e1edac22722d44fea96fb4b9bb05b4a1 P 15ab8f217086450b933f077d59555f07 [term 1 LEADER]: Raft consensus shutting down.
I20251209 08:58:04.831110 23036 raft_consensus.cc:2272] T e1edac22722d44fea96fb4b9bb05b4a1 P f2190f825aa04a0cb6b89cac0e770d12 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251209 08:58:04.831110 23035 raft_consensus.cc:2272] T e1edac22722d44fea96fb4b9bb05b4a1 P da42139421e54188b827cd6256e162b0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251209 08:58:04.831256 23037 raft_consensus.cc:2272] T e1edac22722d44fea96fb4b9bb05b4a1 P 15ab8f217086450b933f077d59555f07 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251209 08:58:04.831277 22439 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42098:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251209 08:58:04.831705 22439 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251209 08:58:04.831986 23037 ts_tablet_manager.cc:1916] T e1edac22722d44fea96fb4b9bb05b4a1 P 15ab8f217086450b933f077d59555f07: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251209 08:58:04.831995 23036 ts_tablet_manager.cc:1916] T e1edac22722d44fea96fb4b9bb05b4a1 P f2190f825aa04a0cb6b89cac0e770d12: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251209 08:58:04.832250 23035 ts_tablet_manager.cc:1916] T e1edac22722d44fea96fb4b9bb05b4a1 P da42139421e54188b827cd6256e162b0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251209 08:58:04.833950 23037 ts_tablet_manager.cc:1929] T e1edac22722d44fea96fb4b9bb05b4a1 P 15ab8f217086450b933f077d59555f07: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20251209 08:58:04.834002 23037 log.cc:1199] T e1edac22722d44fea96fb4b9bb05b4a1 P 15ab8f217086450b933f077d59555f07: Deleting WAL directory at /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765270668420570-20677-0/minicluster-data/ts-1/wal/wals/e1edac22722d44fea96fb4b9bb05b4a1
I20251209 08:58:04.834193 23037 ts_tablet_manager.cc:1950] T e1edac22722d44fea96fb4b9bb05b4a1 P 15ab8f217086450b933f077d59555f07: Deleting consensus metadata
I20251209 08:58:04.834416 23036 ts_tablet_manager.cc:1929] T e1edac22722d44fea96fb4b9bb05b4a1 P f2190f825aa04a0cb6b89cac0e770d12: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20251209 08:58:04.834489 23036 log.cc:1199] T e1edac22722d44fea96fb4b9bb05b4a1 P f2190f825aa04a0cb6b89cac0e770d12: Deleting WAL directory at /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765270668420570-20677-0/minicluster-data/ts-2/wal/wals/e1edac22722d44fea96fb4b9bb05b4a1
I20251209 08:58:04.834643 23035 ts_tablet_manager.cc:1929] T e1edac22722d44fea96fb4b9bb05b4a1 P da42139421e54188b827cd6256e162b0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20251209 08:58:04.834712 23035 log.cc:1199] T e1edac22722d44fea96fb4b9bb05b4a1 P da42139421e54188b827cd6256e162b0: Deleting WAL directory at /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765270668420570-20677-0/minicluster-data/ts-0/wal/wals/e1edac22722d44fea96fb4b9bb05b4a1
I20251209 08:58:04.834753 23036 ts_tablet_manager.cc:1950] T e1edac22722d44fea96fb4b9bb05b4a1 P f2190f825aa04a0cb6b89cac0e770d12: Deleting consensus metadata
I20251209 08:58:04.834770 22427 catalog_manager.cc:4985] TS 15ab8f217086450b933f077d59555f07 (127.20.49.66:45423): tablet e1edac22722d44fea96fb4b9bb05b4a1 (table test-table [id=f1ceb51a02254e39b0557363ee1f3671]) successfully deleted
I20251209 08:58:04.834950 23035 ts_tablet_manager.cc:1950] T e1edac22722d44fea96fb4b9bb05b4a1 P da42139421e54188b827cd6256e162b0: Deleting consensus metadata
I20251209 08:58:04.835439 22426 catalog_manager.cc:4985] TS f2190f825aa04a0cb6b89cac0e770d12 (127.20.49.67:42299): tablet e1edac22722d44fea96fb4b9bb05b4a1 (table test-table [id=f1ceb51a02254e39b0557363ee1f3671]) successfully deleted
I20251209 08:58:04.835624 22426 catalog_manager.cc:4985] TS da42139421e54188b827cd6256e162b0 (127.20.49.65:46509): tablet e1edac22722d44fea96fb4b9bb05b4a1 (table test-table [id=f1ceb51a02254e39b0557363ee1f3671]) successfully deleted
I20251209 08:58:04.838979 22580 tablet_service.cc:1505] Processing CreateTablet for tablet f27a0a3fdbdb4956916fad982d660dd4 (DEFAULT_TABLE table=test-table [id=06eaf1df7ab843039b2da1e3352a68dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20251209 08:58:04.839180 22722 tablet_service.cc:1505] Processing CreateTablet for tablet f27a0a3fdbdb4956916fad982d660dd4 (DEFAULT_TABLE table=test-table [id=06eaf1df7ab843039b2da1e3352a68dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20251209 08:58:04.839249 22580 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f27a0a3fdbdb4956916fad982d660dd4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251209 08:58:04.839416 22722 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f27a0a3fdbdb4956916fad982d660dd4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251209 08:58:04.839507 22864 tablet_service.cc:1505] Processing CreateTablet for tablet f27a0a3fdbdb4956916fad982d660dd4 (DEFAULT_TABLE table=test-table [id=06eaf1df7ab843039b2da1e3352a68dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20251209 08:58:04.839829 22864 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f27a0a3fdbdb4956916fad982d660dd4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251209 08:58:04.841490 23048 tablet_bootstrap.cc:492] T f27a0a3fdbdb4956916fad982d660dd4 P 15ab8f217086450b933f077d59555f07: Bootstrap starting.
I20251209 08:58:04.841645 23046 tablet_bootstrap.cc:492] T f27a0a3fdbdb4956916fad982d660dd4 P f2190f825aa04a0cb6b89cac0e770d12: Bootstrap starting.
I20251209 08:58:04.842519 23048 tablet_bootstrap.cc:654] T f27a0a3fdbdb4956916fad982d660dd4 P 15ab8f217086450b933f077d59555f07: Neither blocks nor log segments found. Creating new log.
I20251209 08:58:04.842623 23046 tablet_bootstrap.cc:654] T f27a0a3fdbdb4956916fad982d660dd4 P f2190f825aa04a0cb6b89cac0e770d12: Neither blocks nor log segments found. Creating new log.
I20251209 08:58:04.843109 23047 tablet_bootstrap.cc:492] T f27a0a3fdbdb4956916fad982d660dd4 P da42139421e54188b827cd6256e162b0: Bootstrap starting.
I20251209 08:58:04.843612 23048 tablet_bootstrap.cc:492] T f27a0a3fdbdb4956916fad982d660dd4 P 15ab8f217086450b933f077d59555f07: No bootstrap required, opened a new log
I20251209 08:58:04.843701 23048 ts_tablet_manager.cc:1403] T f27a0a3fdbdb4956916fad982d660dd4 P 15ab8f217086450b933f077d59555f07: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20251209 08:58:04.844013 23047 tablet_bootstrap.cc:654] T f27a0a3fdbdb4956916fad982d660dd4 P da42139421e54188b827cd6256e162b0: Neither blocks nor log segments found. Creating new log.
I20251209 08:58:04.844115 23046 tablet_bootstrap.cc:492] T f27a0a3fdbdb4956916fad982d660dd4 P f2190f825aa04a0cb6b89cac0e770d12: No bootstrap required, opened a new log
I20251209 08:58:04.844194 23046 ts_tablet_manager.cc:1403] T f27a0a3fdbdb4956916fad982d660dd4 P f2190f825aa04a0cb6b89cac0e770d12: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20251209 08:58:04.844164 23048 raft_consensus.cc:359] T f27a0a3fdbdb4956916fad982d660dd4 P 15ab8f217086450b933f077d59555f07 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da42139421e54188b827cd6256e162b0" member_type: VOTER last_known_addr { host: "127.20.49.65" port: 46509 } } peers { permanent_uuid: "15ab8f217086450b933f077d59555f07" member_type: VOTER last_known_addr { host: "127.20.49.66" port: 45423 } } peers { permanent_uuid: "f2190f825aa04a0cb6b89cac0e770d12" member_type: VOTER last_known_addr { host: "127.20.49.67" port: 42299 } }
I20251209 08:58:04.844266 23048 raft_consensus.cc:385] T f27a0a3fdbdb4956916fad982d660dd4 P 15ab8f217086450b933f077d59555f07 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251209 08:58:04.844305 23048 raft_consensus.cc:740] T f27a0a3fdbdb4956916fad982d660dd4 P 15ab8f217086450b933f077d59555f07 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 15ab8f217086450b933f077d59555f07, State: Initialized, Role: FOLLOWER
I20251209 08:58:04.844478 23048 consensus_queue.cc:260] T f27a0a3fdbdb4956916fad982d660dd4 P 15ab8f217086450b933f077d59555f07 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da42139421e54188b827cd6256e162b0" member_type: VOTER last_known_addr { host: "127.20.49.65" port: 46509 } } peers { permanent_uuid: "15ab8f217086450b933f077d59555f07" member_type: VOTER last_known_addr { host: "127.20.49.66" port: 45423 } } peers { permanent_uuid: "f2190f825aa04a0cb6b89cac0e770d12" member_type: VOTER last_known_addr { host: "127.20.49.67" port: 42299 } }
I20251209 08:58:04.844676 23046 raft_consensus.cc:359] T f27a0a3fdbdb4956916fad982d660dd4 P f2190f825aa04a0cb6b89cac0e770d12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da42139421e54188b827cd6256e162b0" member_type: VOTER last_known_addr { host: "127.20.49.65" port: 46509 } } peers { permanent_uuid: "15ab8f217086450b933f077d59555f07" member_type: VOTER last_known_addr { host: "127.20.49.66" port: 45423 } } peers { permanent_uuid: "f2190f825aa04a0cb6b89cac0e770d12" member_type: VOTER last_known_addr { host: "127.20.49.67" port: 42299 } }
I20251209 08:58:04.844745 23048 ts_tablet_manager.cc:1434] T f27a0a3fdbdb4956916fad982d660dd4 P 15ab8f217086450b933f077d59555f07: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20251209 08:58:04.844774 23046 raft_consensus.cc:385] T f27a0a3fdbdb4956916fad982d660dd4 P f2190f825aa04a0cb6b89cac0e770d12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251209 08:58:04.844805 23046 raft_consensus.cc:740] T f27a0a3fdbdb4956916fad982d660dd4 P f2190f825aa04a0cb6b89cac0e770d12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f2190f825aa04a0cb6b89cac0e770d12, State: Initialized, Role: FOLLOWER
I20251209 08:58:04.844937 23046 consensus_queue.cc:260] T f27a0a3fdbdb4956916fad982d660dd4 P f2190f825aa04a0cb6b89cac0e770d12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da42139421e54188b827cd6256e162b0" member_type: VOTER last_known_addr { host: "127.20.49.65" port: 46509 } } peers { permanent_uuid: "15ab8f217086450b933f077d59555f07" member_type: VOTER last_known_addr { host: "127.20.49.66" port: 45423 } } peers { permanent_uuid: "f2190f825aa04a0cb6b89cac0e770d12" member_type: VOTER last_known_addr { host: "127.20.49.67" port: 42299 } }
I20251209 08:58:04.845175 23046 ts_tablet_manager.cc:1434] T f27a0a3fdbdb4956916fad982d660dd4 P f2190f825aa04a0cb6b89cac0e770d12: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20251209 08:58:04.845322 23047 tablet_bootstrap.cc:492] T f27a0a3fdbdb4956916fad982d660dd4 P da42139421e54188b827cd6256e162b0: No bootstrap required, opened a new log
I20251209 08:58:04.845422 23047 ts_tablet_manager.cc:1403] T f27a0a3fdbdb4956916fad982d660dd4 P da42139421e54188b827cd6256e162b0: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20251209 08:58:04.845924 23047 raft_consensus.cc:359] T f27a0a3fdbdb4956916fad982d660dd4 P da42139421e54188b827cd6256e162b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da42139421e54188b827cd6256e162b0" member_type: VOTER last_known_addr { host: "127.20.49.65" port: 46509 } } peers { permanent_uuid: "15ab8f217086450b933f077d59555f07" member_type: VOTER last_known_addr { host: "127.20.49.66" port: 45423 } } peers { permanent_uuid: "f2190f825aa04a0cb6b89cac0e770d12" member_type: VOTER last_known_addr { host: "127.20.49.67" port: 42299 } }
I20251209 08:58:04.846040 23047 raft_consensus.cc:385] T f27a0a3fdbdb4956916fad982d660dd4 P da42139421e54188b827cd6256e162b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251209 08:58:04.846076 23047 raft_consensus.cc:740] T f27a0a3fdbdb4956916fad982d660dd4 P da42139421e54188b827cd6256e162b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: da42139421e54188b827cd6256e162b0, State: Initialized, Role: FOLLOWER
I20251209 08:58:04.846197 23047 consensus_queue.cc:260] T f27a0a3fdbdb4956916fad982d660dd4 P da42139421e54188b827cd6256e162b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da42139421e54188b827cd6256e162b0" member_type: VOTER last_known_addr { host: "127.20.49.65" port: 46509 } } peers { permanent_uuid: "15ab8f217086450b933f077d59555f07" member_type: VOTER last_known_addr { host: "127.20.49.66" port: 45423 } } peers { permanent_uuid: "f2190f825aa04a0cb6b89cac0e770d12" member_type: VOTER last_known_addr { host: "127.20.49.67" port: 42299 } }
I20251209 08:58:04.846462 23047 ts_tablet_manager.cc:1434] T f27a0a3fdbdb4956916fad982d660dd4 P da42139421e54188b827cd6256e162b0: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20251209 08:58:04.860781 23026 raft_consensus.cc:493] T f27a0a3fdbdb4956916fad982d660dd4 P 15ab8f217086450b933f077d59555f07 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251209 08:58:04.860913 23026 raft_consensus.cc:515] T f27a0a3fdbdb4956916fad982d660dd4 P 15ab8f217086450b933f077d59555f07 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da42139421e54188b827cd6256e162b0" member_type: VOTER last_known_addr { host: "127.20.49.65" port: 46509 } } peers { permanent_uuid: "15ab8f217086450b933f077d59555f07" member_type: VOTER last_known_addr { host: "127.20.49.66" port: 45423 } } peers { permanent_uuid: "f2190f825aa04a0cb6b89cac0e770d12" member_type: VOTER last_known_addr { host: "127.20.49.67" port: 42299 } }
I20251209 08:58:04.861395 23026 leader_election.cc:290] T f27a0a3fdbdb4956916fad982d660dd4 P 15ab8f217086450b933f077d59555f07 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers da42139421e54188b827cd6256e162b0 (127.20.49.65:46509), f2190f825aa04a0cb6b89cac0e770d12 (127.20.49.67:42299)
I20251209 08:58:04.861826 22600 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f27a0a3fdbdb4956916fad982d660dd4" candidate_uuid: "15ab8f217086450b933f077d59555f07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "da42139421e54188b827cd6256e162b0" is_pre_election: true
I20251209 08:58:04.861858 22884 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f27a0a3fdbdb4956916fad982d660dd4" candidate_uuid: "15ab8f217086450b933f077d59555f07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f2190f825aa04a0cb6b89cac0e770d12" is_pre_election: true
I20251209 08:58:04.861967 22600 raft_consensus.cc:2468] T f27a0a3fdbdb4956916fad982d660dd4 P da42139421e54188b827cd6256e162b0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 15ab8f217086450b933f077d59555f07 in term 0.
I20251209 08:58:04.862006 22884 raft_consensus.cc:2468] T f27a0a3fdbdb4956916fad982d660dd4 P f2190f825aa04a0cb6b89cac0e770d12 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 15ab8f217086450b933f077d59555f07 in term 0.
I20251209 08:58:04.862371 22674 leader_election.cc:304] T f27a0a3fdbdb4956916fad982d660dd4 P 15ab8f217086450b933f077d59555f07 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 15ab8f217086450b933f077d59555f07, da42139421e54188b827cd6256e162b0; no voters:
I20251209 08:58:04.862596 23026 raft_consensus.cc:2804] T f27a0a3fdbdb4956916fad982d660dd4 P 15ab8f217086450b933f077d59555f07 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251209 08:58:04.862659 23026 raft_consensus.cc:493] T f27a0a3fdbdb4956916fad982d660dd4 P 15ab8f217086450b933f077d59555f07 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251209 08:58:04.862708 23026 raft_consensus.cc:3060] T f27a0a3fdbdb4956916fad982d660dd4 P 15ab8f217086450b933f077d59555f07 [term 0 FOLLOWER]: Advancing to term 1
I20251209 08:58:04.863636 23026 raft_consensus.cc:515] T f27a0a3fdbdb4956916fad982d660dd4 P 15ab8f217086450b933f077d59555f07 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da42139421e54188b827cd6256e162b0" member_type: VOTER last_known_addr { host: "127.20.49.65" port: 46509 } } peers { permanent_uuid: "15ab8f217086450b933f077d59555f07" member_type: VOTER last_known_addr { host: "127.20.49.66" port: 45423 } } peers { permanent_uuid: "f2190f825aa04a0cb6b89cac0e770d12" member_type: VOTER last_known_addr { host: "127.20.49.67" port: 42299 } }
I20251209 08:58:04.863933 23026 leader_election.cc:290] T f27a0a3fdbdb4956916fad982d660dd4 P 15ab8f217086450b933f077d59555f07 [CANDIDATE]: Term 1 election: Requested vote from peers da42139421e54188b827cd6256e162b0 (127.20.49.65:46509), f2190f825aa04a0cb6b89cac0e770d12 (127.20.49.67:42299)
I20251209 08:58:04.864404 22600 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f27a0a3fdbdb4956916fad982d660dd4" candidate_uuid: "15ab8f217086450b933f077d59555f07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "da42139421e54188b827cd6256e162b0"
I20251209 08:58:04.864444 22884 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f27a0a3fdbdb4956916fad982d660dd4" candidate_uuid: "15ab8f217086450b933f077d59555f07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f2190f825aa04a0cb6b89cac0e770d12"
I20251209 08:58:04.864534 22600 raft_consensus.cc:3060] T f27a0a3fdbdb4956916fad982d660dd4 P da42139421e54188b827cd6256e162b0 [term 0 FOLLOWER]: Advancing to term 1
I20251209 08:58:04.864557 22884 raft_consensus.cc:3060] T f27a0a3fdbdb4956916fad982d660dd4 P f2190f825aa04a0cb6b89cac0e770d12 [term 0 FOLLOWER]: Advancing to term 1
I20251209 08:58:04.865408 22600 raft_consensus.cc:2468] T f27a0a3fdbdb4956916fad982d660dd4 P da42139421e54188b827cd6256e162b0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 15ab8f217086450b933f077d59555f07 in term 1.
I20251209 08:58:04.865420 22884 raft_consensus.cc:2468] T f27a0a3fdbdb4956916fad982d660dd4 P f2190f825aa04a0cb6b89cac0e770d12 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 15ab8f217086450b933f077d59555f07 in term 1.
I20251209 08:58:04.865753 22674 leader_election.cc:304] T f27a0a3fdbdb4956916fad982d660dd4 P 15ab8f217086450b933f077d59555f07 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 15ab8f217086450b933f077d59555f07, da42139421e54188b827cd6256e162b0; no voters:
I20251209 08:58:04.865921 23026 raft_consensus.cc:2804] T f27a0a3fdbdb4956916fad982d660dd4 P 15ab8f217086450b933f077d59555f07 [term 1 FOLLOWER]: Leader election won for term 1
I20251209 08:58:04.866021 23026 raft_consensus.cc:697] T f27a0a3fdbdb4956916fad982d660dd4 P 15ab8f217086450b933f077d59555f07 [term 1 LEADER]: Becoming Leader. State: Replica: 15ab8f217086450b933f077d59555f07, State: Running, Role: LEADER
I20251209 08:58:04.866590 23026 consensus_queue.cc:237] T f27a0a3fdbdb4956916fad982d660dd4 P 15ab8f217086450b933f077d59555f07 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da42139421e54188b827cd6256e162b0" member_type: VOTER last_known_addr { host: "127.20.49.65" port: 46509 } } peers { permanent_uuid: "15ab8f217086450b933f077d59555f07" member_type: VOTER last_known_addr { host: "127.20.49.66" port: 45423 } } peers { permanent_uuid: "f2190f825aa04a0cb6b89cac0e770d12" member_type: VOTER last_known_addr { host: "127.20.49.67" port: 42299 } }
I20251209 08:58:04.868477 22439 catalog_manager.cc:5654] T f27a0a3fdbdb4956916fad982d660dd4 P 15ab8f217086450b933f077d59555f07 reported cstate change: term changed from 0 to 1, leader changed from <none> to 15ab8f217086450b933f077d59555f07 (127.20.49.66). New cstate: current_term: 1 leader_uuid: "15ab8f217086450b933f077d59555f07" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da42139421e54188b827cd6256e162b0" member_type: VOTER last_known_addr { host: "127.20.49.65" port: 46509 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "15ab8f217086450b933f077d59555f07" member_type: VOTER last_known_addr { host: "127.20.49.66" port: 45423 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f2190f825aa04a0cb6b89cac0e770d12" member_type: VOTER last_known_addr { host: "127.20.49.67" port: 42299 } health_report { overall_health: UNKNOWN } } }
I20251209 08:58:04.885893 22884 raft_consensus.cc:1275] T f27a0a3fdbdb4956916fad982d660dd4 P f2190f825aa04a0cb6b89cac0e770d12 [term 1 FOLLOWER]: Refusing update from remote peer 15ab8f217086450b933f077d59555f07: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251209 08:58:04.886011 22600 raft_consensus.cc:1275] T f27a0a3fdbdb4956916fad982d660dd4 P da42139421e54188b827cd6256e162b0 [term 1 FOLLOWER]: Refusing update from remote peer 15ab8f217086450b933f077d59555f07: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251209 08:58:04.886411 23026 consensus_queue.cc:1048] T f27a0a3fdbdb4956916fad982d660dd4 P 15ab8f217086450b933f077d59555f07 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f2190f825aa04a0cb6b89cac0e770d12" member_type: VOTER last_known_addr { host: "127.20.49.67" port: 42299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251209 08:58:04.886562 23030 consensus_queue.cc:1048] T f27a0a3fdbdb4956916fad982d660dd4 P 15ab8f217086450b933f077d59555f07 [LEADER]: Connected to new peer: Peer: permanent_uuid: "da42139421e54188b827cd6256e162b0" member_type: VOTER last_known_addr { host: "127.20.49.65" port: 46509 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20251209 08:58:04.899185 23018 tablet_replica.cc:1306] Aborted: operation has been aborted: cancelling pending write operations
I20251209 08:58:06.471048 22440 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42098:
table { table_name: "test-table" } modify_external_catalogs: true
I20251209 08:58:06.471199 22440 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42098:
table { table_name: "test-table" } modify_external_catalogs: true
I20251209 08:58:06.474159 22440 catalog_manager.cc:5941] T 00000000000000000000000000000000 P 39341759984c48989ceba01b8acb2da9: Sending DeleteTablet for 3 replicas of tablet f27a0a3fdbdb4956916fad982d660dd4
I20251209 08:58:06.474830 22580 tablet_service.cc:1552] Processing DeleteTablet for tablet f27a0a3fdbdb4956916fad982d660dd4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-09 08:58:06 UTC) from {username='kudu', principal='kudu/127.20.49.126@KRBTEST.COM'} at 127.0.0.1:44186
I20251209 08:58:06.475190 22864 tablet_service.cc:1552] Processing DeleteTablet for tablet f27a0a3fdbdb4956916fad982d660dd4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-09 08:58:06 UTC) from {username='kudu', principal='kudu/127.20.49.126@KRBTEST.COM'} at 127.0.0.1:57140
I20251209 08:58:06.475188 22722 tablet_service.cc:1552] Processing DeleteTablet for tablet f27a0a3fdbdb4956916fad982d660dd4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-09 08:58:06 UTC) from {username='kudu', principal='kudu/127.20.49.126@KRBTEST.COM'} at 127.0.0.1:33058
I20251209 08:58:06.475232 23085 tablet_replica.cc:333] T f27a0a3fdbdb4956916fad982d660dd4 P da42139421e54188b827cd6256e162b0: stopping tablet replica
I20251209 08:58:06.475364 23085 raft_consensus.cc:2243] T f27a0a3fdbdb4956916fad982d660dd4 P da42139421e54188b827cd6256e162b0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251209 08:58:06.475464 20677 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task8XuDZV/build/debug/bin/kudu with pid 22515
I20251209 08:58:06.476963 23087 tablet_replica.cc:333] T f27a0a3fdbdb4956916fad982d660dd4 P f2190f825aa04a0cb6b89cac0e770d12: stopping tablet replica
I20251209 08:58:06.477190 23087 raft_consensus.cc:2243] T f27a0a3fdbdb4956916fad982d660dd4 P f2190f825aa04a0cb6b89cac0e770d12 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251209 08:58:06.477348 23087 raft_consensus.cc:2272] T f27a0a3fdbdb4956916fad982d660dd4 P f2190f825aa04a0cb6b89cac0e770d12 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251209 08:58:06.478148 23087 ts_tablet_manager.cc:1916] T f27a0a3fdbdb4956916fad982d660dd4 P f2190f825aa04a0cb6b89cac0e770d12: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251209 08:58:06.481550 23087 ts_tablet_manager.cc:1929] T f27a0a3fdbdb4956916fad982d660dd4 P f2190f825aa04a0cb6b89cac0e770d12: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20251209 08:58:06.481647 23087 log.cc:1199] T f27a0a3fdbdb4956916fad982d660dd4 P f2190f825aa04a0cb6b89cac0e770d12: Deleting WAL directory at /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1765270668420570-20677-0/minicluster-data/ts-2/wal/wals/f27a0a3fdbdb4956916fad982d660dd4
I20251209 08:58:06.481989 23087 ts_tablet_manager.cc:1950] T f27a0a3fdbdb4956916fad982d660dd4 P f2190f825aa04a0cb6b89cac0e770d12: Deleting consensus metadata
I20251209 08:58:06.482865 22426 catalog_manager.cc:4985] TS f2190f825aa04a0cb6b89cac0e770d12 (127.20.49.67:42299): tablet f27a0a3fdbdb4956916fad982d660dd4 (table test-table [id=06eaf1df7ab843039b2da1e3352a68dc]) successfully deleted
I20251209 08:58:06.483109 23086 tablet_replica.cc:333] T f27a0a3fdbdb4956916fad982d660dd4 P 15ab8f217086450b933f077d59555f07: stopping tablet replica
I20251209 08:58:06.483278 23086 raft_consensus.cc:2243] T f27a0a3fdbdb4956916fad982d660dd4 P 15ab8f217086450b933f077d59555f07 [term 1 LEADER]: Raft consensus shutting down.
I20251209 08:58:06.483528 23086 raft_consensus.cc:2272] T f27a0a3fdbdb4956916fad982d660dd4 P 15ab8f217086450b933f077d59555f07 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251209 08:58:06.484274 23086 ts_tablet_manager.cc:1916] T f27a0a3fdbdb4956916fad982d660dd4 P 15ab8f217086450b933f077d59555f07: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251209 08:58:06.486522 22426 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.20.49.65:46509 (error 108)
I20251209 08:58:06.486860 20677 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task8XuDZV/build/debug/bin/kudu with pid 22657
I20251209 08:58:06.488423 23089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:58:06.487908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.49.65:46509 (local address 127.0.0.1:44216)
1209 08:58:06.488195 (+ 287us) negotiation.cc:107] Waiting for socket to connect
1209 08:58:06.488275 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.49.65:46509: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":90,"threads_started":1}
W20251209 08:58:06.488605 22426 catalog_manager.cc:4712] TS da42139421e54188b827cd6256e162b0 (127.20.49.65:46509): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f27a0a3fdbdb4956916fad982d660dd4: Network error: Client connection negotiation failed: client connection to 127.20.49.65:46509: connect: Connection refused (error 111)
I20251209 08:58:06.495494 20677 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task8XuDZV/build/debug/bin/kudu with pid 22799
I20251209 08:58:06.496608 23089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:58:06.495729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.49.66:45423 (local address 127.0.0.1:33078)
1209 08:58:06.496433 (+ 704us) negotiation.cc:107] Waiting for socket to connect
1209 08:58:06.496497 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.49.66:45423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":573}
I20251209 08:58:06.502480 20677 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task8XuDZV/build/debug/bin/kudu with pid 22409
2025-12-09T08:58:06Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (7546 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765270668420570-20677-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Dec 09 08:58:06 dist-test-slave-8w6c krb5kdc[23092](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Dec 09 08:58:06 dist-test-slave-8w6c krb5kdc[23092](info): set up 2 sockets
Dec 09 08:58:06 dist-test-slave-8w6c krb5kdc[23092](info): commencing operation
krb5kdc: starting...
W20251209 08:58:08.522436 20677 mini_kdc.cc:121] Time spent starting KDC: real 1.995s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765270668420570-20677-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765270668420570-20677-0/krb5kdc/test-user.keytab.
Dec 09 08:58:08 dist-test-slave-8w6c krb5kdc[23092](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765270688, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-12-09T08:58:08Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-12-09T08:58:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.49.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.49.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.49.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765270668420570-20677-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.20.49.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765270668420570-20677-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.20.49.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.20.49.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.20.49.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765270668420570-20677-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.20.49.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765270668420570-20677-0/krb5kdc/kudu.keytab.
I20251209 08:58:08.664299 20677 external_mini_cluster.cc:1366] Running /tmp/dist-test-task8XuDZV/build/debug/bin/kudu
/tmp/dist-test-task8XuDZV/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765270668420570-20677-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765270668420570-20677-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765270668420570-20677-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765270668420570-20677-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.49.126:43381
--webserver_interface=127.20.49.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765270668420570-20677-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.49.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.20.49.84:43811
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.49.126:43381
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765270668420570-20677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765270668420570-20677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765270668420570-20677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251209 08:58:08.774338 23108 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251209 08:58:08.774583 23108 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251209 08:58:08.774639 23108 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251209 08:58:08.777985 23108 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251209 08:58:08.778054 23108 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251209 08:58:08.778079 23108 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251209 08:58:08.778098 23108 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251209 08:58:08.778115 23108 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251209 08:58:08.782395 23108 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.49.84:43811
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765270668420570-20677-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765270668420570-20677-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.49.126:43381
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765270668420570-20677-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.49.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.49.126:43381
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765270668420570-20677-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.49.126
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23108
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765270668420570-20677-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 468897755969d686fb08386f42f7835685e7953a
build type DEBUG
built by None at 09 Dec 2025 08:43:15 UTC on 24a791456cd2
build id 9426
I20251209 08:58:08.783455 23108 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251209 08:58:08.784266 23108 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251209 08:58:08.789762 23116 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251209 08:58:08.789764 23113 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251209 08:58:08.789767 23114 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251209 08:58:08.789991 23108 server_base.cc:1047] running on GCE node
I20251209 08:58:08.790474 23108 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251209 08:58:08.791326 23108 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251209 08:58:08.792472 23108 hybrid_clock.cc:648] HybridClock initialized: now 1765270688792453 us; error 38 us; skew 500 ppm
Dec 09 08:58:08 dist-test-slave-8w6c krb5kdc[23092](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765270688, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.49.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251209 08:58:08.795058 23108 init.cc:377] Logged in from keytab as kudu/127.20.49.126@KRBTEST.COM (short username kudu)
I20251209 08:58:08.796177 23108 webserver.cc:492] Webserver started at http://127.20.49.126:36537/ using document root <none> and password file <none>
I20251209 08:58:08.796717 23108 fs_manager.cc:362] Metadata directory not provided
I20251209 08:58:08.796779 23108 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251209 08:58:08.796965 23108 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251209 08:58:08.798801 23108 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765270668420570-20677-0/minicluster-data/master-0/data/instance:
uuid: "5ca98d771a8447f6b94cbe503e8d4024"
format_stamp: "Formatted at 2025-12-09 08:58:08 on dist-test-slave-8w6c"
server_key: "3ef244ad431b4570a4dff4b87ed604fc"
server_key_iv: "89a40be989f85568e2ef89803694aba7"
server_key_version: "encryptionkey@0"
I20251209 08:58:08.799302 23108 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765270668420570-20677-0/minicluster-data/master-0/wal/instance:
uuid: "5ca98d771a8447f6b94cbe503e8d4024"
format_stamp: "Formatted at 2025-12-09 08:58:08 on dist-test-slave-8w6c"
server_key: "3ef244ad431b4570a4dff4b87ed604fc"
server_key_iv: "89a40be989f85568e2ef89803694aba7"
server_key_version: "encryptionkey@0"
I20251209 08:58:08.802884 23108 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20251209 08:58:08.805186 23123 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251209 08:58:08.806285 23108 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20251209 08:58:08.806407 23108 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765270668420570-20677-0/minicluster-data/master-0/data,/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765270668420570-20677-0/minicluster-data/master-0/wal
uuid: "5ca98d771a8447f6b94cbe503e8d4024"
format_stamp: "Formatted at 2025-12-09 08:58:08 on dist-test-slave-8w6c"
server_key: "3ef244ad431b4570a4dff4b87ed604fc"
server_key_iv: "89a40be989f85568e2ef89803694aba7"
server_key_version: "encryptionkey@0"
I20251209 08:58:08.806505 23108 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765270668420570-20677-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765270668420570-20677-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765270668420570-20677-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251209 08:58:08.828205 23108 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251209 08:58:08.834141 23108 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251209 08:58:08.834326 23108 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251209 08:58:08.841955 23108 rpc_server.cc:307] RPC server started. Bound to: 127.20.49.126:43381
I20251209 08:58:08.841967 23175 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.49.126:43381 every 8 connection(s)
I20251209 08:58:08.842985 23108 server_base.cc:1179] Dumped server information to /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765270668420570-20677-0/minicluster-data/master-0/data/info.pb
I20251209 08:58:08.845762 23176 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251209 08:58:08.850847 20677 external_mini_cluster.cc:1428] Started /tmp/dist-test-task8XuDZV/build/debug/bin/kudu as pid 23108
I20251209 08:58:08.851006 20677 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765270668420570-20677-0/minicluster-data/master-0/wal/instance
I20251209 08:58:08.851251 20677 external_mini_cluster.cc:1468] Setting key 14d86e8769316f5a8ef5de9254fc2ed6
I20251209 08:58:08.851691 23176 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5ca98d771a8447f6b94cbe503e8d4024: Bootstrap starting.
I20251209 08:58:08.854174 23176 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5ca98d771a8447f6b94cbe503e8d4024: Neither blocks nor log segments found. Creating new log.
I20251209 08:58:08.854938 23176 log.cc:826] T 00000000000000000000000000000000 P 5ca98d771a8447f6b94cbe503e8d4024: Log is configured to *not* fsync() on all Append() calls
I20251209 08:58:08.856670 23176 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5ca98d771a8447f6b94cbe503e8d4024: No bootstrap required, opened a new log
Dec 09 08:58:08 dist-test-slave-8w6c krb5kdc[23092](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765270688, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.49.126@KRBTEST.COM
I20251209 08:58:08.859721 23176 raft_consensus.cc:359] T 00000000000000000000000000000000 P 5ca98d771a8447f6b94cbe503e8d4024 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ca98d771a8447f6b94cbe503e8d4024" member_type: VOTER last_known_addr { host: "127.20.49.126" port: 43381 } }
I20251209 08:58:08.859978 23176 raft_consensus.cc:385] T 00000000000000000000000000000000 P 5ca98d771a8447f6b94cbe503e8d4024 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251209 08:58:08.860054 23176 raft_consensus.cc:740] T 00000000000000000000000000000000 P 5ca98d771a8447f6b94cbe503e8d4024 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ca98d771a8447f6b94cbe503e8d4024, State: Initialized, Role: FOLLOWER
I20251209 08:58:08.860553 23176 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5ca98d771a8447f6b94cbe503e8d4024 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ca98d771a8447f6b94cbe503e8d4024" member_type: VOTER last_known_addr { host: "127.20.49.126" port: 43381 } }
I20251209 08:58:08.860689 23176 raft_consensus.cc:399] T 00000000000000000000000000000000 P 5ca98d771a8447f6b94cbe503e8d4024 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251209 08:58:08.860759 23176 raft_consensus.cc:493] T 00000000000000000000000000000000 P 5ca98d771a8447f6b94cbe503e8d4024 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251209 08:58:08.860869 23176 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 5ca98d771a8447f6b94cbe503e8d4024 [term 0 FOLLOWER]: Advancing to term 1
I20251209 08:58:08.862039 23176 raft_consensus.cc:515] T 00000000000000000000000000000000 P 5ca98d771a8447f6b94cbe503e8d4024 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ca98d771a8447f6b94cbe503e8d4024" member_type: VOTER last_known_addr { host: "127.20.49.126" port: 43381 } }
I20251209 08:58:08.862412 23176 leader_election.cc:304] T 00000000000000000000000000000000 P 5ca98d771a8447f6b94cbe503e8d4024 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5ca98d771a8447f6b94cbe503e8d4024; no voters:
I20251209 08:58:08.862756 23176 leader_election.cc:290] T 00000000000000000000000000000000 P 5ca98d771a8447f6b94cbe503e8d4024 [CANDIDATE]: Term 1 election: Requested vote from peers
I20251209 08:58:08.863129 23181 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 5ca98d771a8447f6b94cbe503e8d4024 [term 1 FOLLOWER]: Leader election won for term 1
I20251209 08:58:08.863507 23181 raft_consensus.cc:697] T 00000000000000000000000000000000 P 5ca98d771a8447f6b94cbe503e8d4024 [term 1 LEADER]: Becoming Leader. State: Replica: 5ca98d771a8447f6b94cbe503e8d4024, State: Running, Role: LEADER
I20251209 08:58:08.863853 23181 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5ca98d771a8447f6b94cbe503e8d4024 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ca98d771a8447f6b94cbe503e8d4024" member_type: VOTER last_known_addr { host: "127.20.49.126" port: 43381 } }
I20251209 08:58:08.864017 23176 sys_catalog.cc:565] T 00000000000000000000000000000000 P 5ca98d771a8447f6b94cbe503e8d4024 [sys.catalog]: configured and running, proceeding with master startup.
I20251209 08:58:08.865105 23181 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5ca98d771a8447f6b94cbe503e8d4024 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5ca98d771a8447f6b94cbe503e8d4024. Latest consensus state: current_term: 1 leader_uuid: "5ca98d771a8447f6b94cbe503e8d4024" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ca98d771a8447f6b94cbe503e8d4024" member_type: VOTER last_known_addr { host: "127.20.49.126" port: 43381 } } }
I20251209 08:58:08.865211 23181 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5ca98d771a8447f6b94cbe503e8d4024 [sys.catalog]: This master's current role is: LEADER
I20251209 08:58:08.865377 23179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:58:08.852609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38566 (local address 127.20.49.126:43381)
1209 08:58:08.853091 (+ 482us) server_negotiation.cc:207] Beginning negotiation
1209 08:58:08.853101 (+ 10us) server_negotiation.cc:400] Waiting for connection header
1209 08:58:08.853138 (+ 37us) server_negotiation.cc:408] Connection header received
1209 08:58:08.853763 (+ 625us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1209 08:58:08.853786 (+ 23us) server_negotiation.cc:462] Received NEGOTIATE request from client
1209 08:58:08.854096 (+ 310us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1209 08:58:08.854432 (+ 336us) server_negotiation.cc:227] Negotiated authn=SASL
1209 08:58:08.855399 (+ 967us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:58:08.856126 (+ 727us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1209 08:58:08.856763 (+ 637us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:58:08.857053 (+ 290us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:58:08.859476 (+ 2423us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1209 08:58:08.859497 (+ 21us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1209 08:58:08.859509 (+ 12us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1209 08:58:08.859543 (+ 34us) server_negotiation.cc:923] calling sasl_server_start()
1209 08:58:08.861881 (+ 2338us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1209 08:58:08.862387 (+ 506us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1209 08:58:08.862394 (+ 7us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1209 08:58:08.862399 (+ 5us) server_negotiation.cc:968] calling sasl_server_step()
1209 08:58:08.862499 (+ 100us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1209 08:58:08.862792 (+ 293us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1209 08:58:08.862795 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1209 08:58:08.862799 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
1209 08:58:08.863249 (+ 450us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1209 08:58:08.863470 (+ 221us) server_negotiation.cc:1036] Waiting for connection context
1209 08:58:08.864254 (+ 784us) server_negotiation.cc:300] Negotiation successful
1209 08:58:08.864521 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":343,"thread_start_us":179,"threads_started":1}
I20251209 08:58:08.866094 23182 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5ca98d771a8447f6b94cbe503e8d4024 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5ca98d771a8447f6b94cbe503e8d4024" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ca98d771a8447f6b94cbe503e8d4024" member_type: VOTER last_known_addr { host: "127.20.49.126" port: 43381 } } }
I20251209 08:58:08.866475 23182 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5ca98d771a8447f6b94cbe503e8d4024 [sys.catalog]: This master's current role is: LEADER
W20251209 08:58:08.868856 23196 catalog_manager.cc:1568] T 00000000000000000000000000000000 P 5ca98d771a8447f6b94cbe503e8d4024: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20251209 08:58:08.868930 23196 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20251209 08:58:08.869020 23190 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251209 08:58:08.869863 23190 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251209 08:58:08.875430 23190 catalog_manager.cc:1357] Generated new cluster ID: bec771a8c1334884af27316425d8a5ba
I20251209 08:58:08.875512 23190 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251209 08:58:08.885593 23190 catalog_manager.cc:1380] Generated new certificate authority record
I20251209 08:58:08.886868 23190 catalog_manager.cc:1514] Loading token signing keys...
I20251209 08:58:08.909847 23190 catalog_manager.cc:6027] T 00000000000000000000000000000000 P 5ca98d771a8447f6b94cbe503e8d4024: Generated new TSK 0
I20251209 08:58:08.910916 23190 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.20.49.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.49.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.49.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765270668420570-20677-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.20.49.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765270668420570-20677-0/krb5kdc/kudu.keytab.
W20251209 08:58:09.852463 20677 mini_kdc.cc:268] Time spent creating service keytab for kudu/127.20.49.65: real 0.930s user 0.000s sys 0.004s
WARNING: no policy specified for HTTP/127.20.49.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.20.49.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.20.49.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765270668420570-20677-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.20.49.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765270668420570-20677-0/krb5kdc/kudu.keytab.
W20251209 08:58:10.821920 20677 mini_kdc.cc:268] Time spent creating service keytab for HTTP/127.20.49.65: real 0.969s user 0.000s sys 0.004s
I20251209 08:58:10.822476 20677 external_mini_cluster.cc:1366] Running /tmp/dist-test-task8XuDZV/build/debug/bin/kudu
/tmp/dist-test-task8XuDZV/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765270668420570-20677-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765270668420570-20677-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765270668420570-20677-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765270668420570-20677-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.49.65:0
--local_ip_for_outbound_sockets=127.20.49.65
--webserver_interface=127.20.49.65
--webserver_port=0
--tserver_master_addrs=127.20.49.126:43381
--keytab_file=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765270668420570-20677-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.49.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.20.49.84:43811
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765270668420570-20677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765270668420570-20677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765270668420570-20677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251209 08:58:10.937739 23204 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251209 08:58:10.937994 23204 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251209 08:58:10.938055 23204 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251209 08:58:10.941447 23204 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251209 08:58:10.941535 23204 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251209 08:58:10.941637 23204 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.49.65
I20251209 08:58:10.946111 23204 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.49.84:43811
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765270668420570-20677-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765270668420570-20677-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765270668420570-20677-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.49.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.49.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765270668420570-20677-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.49.65
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.20.49.126:43381
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23204
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.49.65
--log_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765270668420570-20677-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 468897755969d686fb08386f42f7835685e7953a
build type DEBUG
built by None at 09 Dec 2025 08:43:15 UTC on 24a791456cd2
build id 9426
I20251209 08:58:10.947227 23204 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251209 08:58:10.948087 23204 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251209 08:58:10.954993 23209 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251209 08:58:10.954988 23210 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251209 08:58:10.955271 23204 server_base.cc:1047] running on GCE node
W20251209 08:58:10.954988 23212 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251209 08:58:10.955767 23204 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251209 08:58:10.956382 23204 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251209 08:58:10.957546 23204 hybrid_clock.cc:648] HybridClock initialized: now 1765270690957529 us; error 44 us; skew 500 ppm
Dec 09 08:58:10 dist-test-slave-8w6c krb5kdc[23092](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765270690, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.49.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251209 08:58:10.960624 23204 init.cc:377] Logged in from keytab as kudu/127.20.49.65@KRBTEST.COM (short username kudu)
I20251209 08:58:10.961830 23204 webserver.cc:492] Webserver started at http://127.20.49.65:38221/ using document root <none> and password file <none>
I20251209 08:58:10.962415 23204 fs_manager.cc:362] Metadata directory not provided
I20251209 08:58:10.962471 23204 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251209 08:58:10.962672 23204 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251209 08:58:10.964553 23204 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765270668420570-20677-0/minicluster-data/ts-0/data/instance:
uuid: "2b0c9e1ceee54ade898b7cdee3eec515"
format_stamp: "Formatted at 2025-12-09 08:58:10 on dist-test-slave-8w6c"
server_key: "59551795df2c097cfb0ce1721cbe0d5f"
server_key_iv: "f3d540cd0bcfc72a2281b5f1ba2f7ca9"
server_key_version: "encryptionkey@0"
I20251209 08:58:10.965088 23204 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765270668420570-20677-0/minicluster-data/ts-0/wal/instance:
uuid: "2b0c9e1ceee54ade898b7cdee3eec515"
format_stamp: "Formatted at 2025-12-09 08:58:10 on dist-test-slave-8w6c"
server_key: "59551795df2c097cfb0ce1721cbe0d5f"
server_key_iv: "f3d540cd0bcfc72a2281b5f1ba2f7ca9"
server_key_version: "encryptionkey@0"
I20251209 08:58:10.968825 23204 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.003s
I20251209 08:58:10.971177 23219 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251209 08:58:10.972338 23204 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20251209 08:58:10.972467 23204 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765270668420570-20677-0/minicluster-data/ts-0/data,/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765270668420570-20677-0/minicluster-data/ts-0/wal
uuid: "2b0c9e1ceee54ade898b7cdee3eec515"
format_stamp: "Formatted at 2025-12-09 08:58:10 on dist-test-slave-8w6c"
server_key: "59551795df2c097cfb0ce1721cbe0d5f"
server_key_iv: "f3d540cd0bcfc72a2281b5f1ba2f7ca9"
server_key_version: "encryptionkey@0"
I20251209 08:58:10.972615 23204 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765270668420570-20677-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765270668420570-20677-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765270668420570-20677-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251209 08:58:11.000277 23204 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251209 08:58:11.003654 23204 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251209 08:58:11.003847 23204 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251209 08:58:11.004436 23204 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251209 08:58:11.005371 23204 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251209 08:58:11.005424 23204 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251209 08:58:11.005540 23204 ts_tablet_manager.cc:616] Registered 0 tablets
I20251209 08:58:11.005568 23204 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251209 08:58:11.015599 23204 rpc_server.cc:307] RPC server started. Bound to: 127.20.49.65:40319
I20251209 08:58:11.015650 23332 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.49.65:40319 every 8 connection(s)
I20251209 08:58:11.016595 23204 server_base.cc:1179] Dumped server information to /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765270668420570-20677-0/minicluster-data/ts-0/data/info.pb
I20251209 08:58:11.019644 20677 external_mini_cluster.cc:1428] Started /tmp/dist-test-task8XuDZV/build/debug/bin/kudu as pid 23204
I20251209 08:58:11.019779 20677 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765270668420570-20677-0/minicluster-data/ts-0/wal/instance
I20251209 08:58:11.020025 20677 external_mini_cluster.cc:1468] Setting key 737f3dbff5062356d126cb5836942775
Dec 09 08:58:11 dist-test-slave-8w6c krb5kdc[23092](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765270690, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.49.65@KRBTEST.COM for kudu/127.20.49.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.20.49.66@KRBTEST.COM; defaulting to no policy
I20251209 08:58:11.031901 23335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:58:11.018544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.49.65:55509 (local address 127.20.49.126:43381)
1209 08:58:11.018820 (+ 276us) server_negotiation.cc:207] Beginning negotiation
1209 08:58:11.018824 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1209 08:58:11.019489 (+ 665us) server_negotiation.cc:408] Connection header received
1209 08:58:11.020442 (+ 953us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1209 08:58:11.020446 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1209 08:58:11.020499 (+ 53us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1209 08:58:11.020617 (+ 118us) server_negotiation.cc:227] Negotiated authn=SASL
1209 08:58:11.022676 (+ 2059us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:58:11.023571 (+ 895us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1209 08:58:11.024564 (+ 993us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:58:11.024724 (+ 160us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:58:11.027803 (+ 3079us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1209 08:58:11.027826 (+ 23us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1209 08:58:11.027829 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1209 08:58:11.027857 (+ 28us) server_negotiation.cc:923] calling sasl_server_start()
1209 08:58:11.029433 (+ 1576us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1209 08:58:11.030199 (+ 766us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1209 08:58:11.030206 (+ 7us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1209 08:58:11.030208 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1209 08:58:11.030280 (+ 72us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1209 08:58:11.030639 (+ 359us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1209 08:58:11.030645 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1209 08:58:11.030648 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
1209 08:58:11.030863 (+ 215us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1209 08:58:11.030952 (+ 89us) server_negotiation.cc:1036] Waiting for connection context
1209 08:58:11.031611 (+ 659us) server_negotiation.cc:300] Negotiation successful
1209 08:58:11.031756 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":187,"thread_start_us":86,"threads_started":1}
I20251209 08:58:11.032618 23336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:58:11.018856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.49.126:43381 (local address 127.20.49.65:55509)
1209 08:58:11.019332 (+ 476us) negotiation.cc:107] Waiting for socket to connect
1209 08:58:11.019368 (+ 36us) client_negotiation.cc:174] Beginning negotiation
1209 08:58:11.020230 (+ 862us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1209 08:58:11.020784 (+ 554us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1209 08:58:11.020795 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
1209 08:58:11.021357 (+ 562us) client_negotiation.cc:189] Negotiated authn=SASL
1209 08:58:11.022502 (+ 1145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:58:11.022526 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:58:11.023717 (+ 1191us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1209 08:58:11.023723 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1209 08:58:11.024427 (+ 704us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:58:11.024442 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:58:11.024730 (+ 288us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:58:11.025429 (+ 699us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1209 08:58:11.025476 (+ 47us) client_negotiation.cc:656] calling sasl_client_start()
1209 08:58:11.027565 (+ 2089us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1209 08:58:11.029583 (+ 2018us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1209 08:58:11.029592 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1209 08:58:11.029611 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
1209 08:58:11.030031 (+ 420us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1209 08:58:11.030401 (+ 370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1209 08:58:11.030404 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1209 08:58:11.030405 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1209 08:58:11.030498 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1209 08:58:11.030995 (+ 497us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1209 08:58:11.031002 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1209 08:58:11.031299 (+ 297us) client_negotiation.cc:769] Sending connection context
1209 08:58:11.031542 (+ 243us) client_negotiation.cc:240] Negotiation successful
1209 08:58:11.031810 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":293,"thread_start_us":152,"threads_started":1}
I20251209 08:58:11.034118 23333 heartbeater.cc:344] Connected to a master server at 127.20.49.126:43381
I20251209 08:58:11.034391 23333 heartbeater.cc:461] Registering TS with master...
I20251209 08:58:11.034965 23333 heartbeater.cc:507] Master 127.20.49.126:43381 requested a full tablet report, sending...
I20251209 08:58:11.036490 23140 ts_manager.cc:194] Registered new tserver with Master: 2b0c9e1ceee54ade898b7cdee3eec515 (127.20.49.65:40319)
I20251209 08:58:11.037701 23140 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.49.65@KRBTEST.COM'} at 127.20.49.65:55509
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.49.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.49.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765270668420570-20677-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.20.49.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765270668420570-20677-0/krb5kdc/kudu.keytab.
W20251209 08:58:11.849140 20677 mini_kdc.cc:268] Time spent creating service keytab for kudu/127.20.49.66: real 0.827s user 0.000s sys 0.004s
WARNING: no policy specified for HTTP/127.20.49.66@KRBTEST.COM; defaulting to no policy
I20251209 08:58:12.040365 23333 heartbeater.cc:499] Master 127.20.49.126:43381 was elected leader, sending a full tablet report...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.20.49.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.20.49.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765270668420570-20677-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.20.49.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765270668420570-20677-0/krb5kdc/kudu.keytab.
W20251209 08:58:12.819586 20677 mini_kdc.cc:268] Time spent creating service keytab for HTTP/127.20.49.66: real 0.970s user 0.000s sys 0.004s
I20251209 08:58:12.820140 20677 external_mini_cluster.cc:1366] Running /tmp/dist-test-task8XuDZV/build/debug/bin/kudu
/tmp/dist-test-task8XuDZV/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765270668420570-20677-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765270668420570-20677-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765270668420570-20677-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765270668420570-20677-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.49.66:0
--local_ip_for_outbound_sockets=127.20.49.66
--webserver_interface=127.20.49.66
--webserver_port=0
--tserver_master_addrs=127.20.49.126:43381
--keytab_file=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765270668420570-20677-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.49.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.20.49.84:43811
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765270668420570-20677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765270668420570-20677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765270668420570-20677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251209 08:58:12.929945 23341 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251209 08:58:12.930189 23341 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251209 08:58:12.930253 23341 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251209 08:58:12.933588 23341 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251209 08:58:12.933666 23341 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251209 08:58:12.933800 23341 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.49.66
I20251209 08:58:12.938253 23341 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.49.84:43811
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765270668420570-20677-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765270668420570-20677-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765270668420570-20677-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.49.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.49.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765270668420570-20677-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.49.66
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.20.49.126:43381
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23341
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.49.66
--log_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765270668420570-20677-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 468897755969d686fb08386f42f7835685e7953a
build type DEBUG
built by None at 09 Dec 2025 08:43:15 UTC on 24a791456cd2
build id 9426
I20251209 08:58:12.939397 23341 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251209 08:58:12.940270 23341 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251209 08:58:12.947058 23347 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251209 08:58:12.947059 23346 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251209 08:58:12.947352 23341 server_base.cc:1047] running on GCE node
W20251209 08:58:12.947543 23349 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251209 08:58:12.947971 23341 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251209 08:58:12.948568 23341 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251209 08:58:12.949774 23341 hybrid_clock.cc:648] HybridClock initialized: now 1765270692949748 us; error 41 us; skew 500 ppm
Dec 09 08:58:12 dist-test-slave-8w6c krb5kdc[23092](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765270692, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.49.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251209 08:58:12.952462 23341 init.cc:377] Logged in from keytab as kudu/127.20.49.66@KRBTEST.COM (short username kudu)
I20251209 08:58:12.953516 23341 webserver.cc:492] Webserver started at http://127.20.49.66:38439/ using document root <none> and password file <none>
I20251209 08:58:12.954097 23341 fs_manager.cc:362] Metadata directory not provided
I20251209 08:58:12.954149 23341 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251209 08:58:12.954301 23341 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251209 08:58:12.956028 23341 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765270668420570-20677-0/minicluster-data/ts-1/data/instance:
uuid: "8956850199b34483a71e10afa2487cf2"
format_stamp: "Formatted at 2025-12-09 08:58:12 on dist-test-slave-8w6c"
server_key: "c96b89378032c6b2215379b2755de061"
server_key_iv: "92067490a1034e4972df30e25079560a"
server_key_version: "encryptionkey@0"
I20251209 08:58:12.956455 23341 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765270668420570-20677-0/minicluster-data/ts-1/wal/instance:
uuid: "8956850199b34483a71e10afa2487cf2"
format_stamp: "Formatted at 2025-12-09 08:58:12 on dist-test-slave-8w6c"
server_key: "c96b89378032c6b2215379b2755de061"
server_key_iv: "92067490a1034e4972df30e25079560a"
server_key_version: "encryptionkey@0"
I20251209 08:58:12.959803 23341 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20251209 08:58:12.961973 23356 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251209 08:58:12.962971 23341 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20251209 08:58:12.963065 23341 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765270668420570-20677-0/minicluster-data/ts-1/data,/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765270668420570-20677-0/minicluster-data/ts-1/wal
uuid: "8956850199b34483a71e10afa2487cf2"
format_stamp: "Formatted at 2025-12-09 08:58:12 on dist-test-slave-8w6c"
server_key: "c96b89378032c6b2215379b2755de061"
server_key_iv: "92067490a1034e4972df30e25079560a"
server_key_version: "encryptionkey@0"
I20251209 08:58:12.963162 23341 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765270668420570-20677-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765270668420570-20677-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765270668420570-20677-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251209 08:58:12.983441 23341 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251209 08:58:12.986425 23341 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251209 08:58:12.986606 23341 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251209 08:58:12.987152 23341 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251209 08:58:12.987999 23341 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251209 08:58:12.988046 23341 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251209 08:58:12.988091 23341 ts_tablet_manager.cc:616] Registered 0 tablets
I20251209 08:58:12.988125 23341 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251209 08:58:12.997092 23341 rpc_server.cc:307] RPC server started. Bound to: 127.20.49.66:46225
I20251209 08:58:12.997128 23469 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.49.66:46225 every 8 connection(s)
I20251209 08:58:12.998046 23341 server_base.cc:1179] Dumped server information to /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765270668420570-20677-0/minicluster-data/ts-1/data/info.pb
Dec 09 08:58:13 dist-test-slave-8w6c krb5kdc[23092](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765270692, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.49.66@KRBTEST.COM for kudu/127.20.49.126@KRBTEST.COM
I20251209 08:58:13.006783 20677 external_mini_cluster.cc:1428] Started /tmp/dist-test-task8XuDZV/build/debug/bin/kudu as pid 23341
I20251209 08:58:13.006912 20677 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765270668420570-20677-0/minicluster-data/ts-1/wal/instance
I20251209 08:58:13.007177 20677 external_mini_cluster.cc:1468] Setting key e341a31daa18ec980b7953985f77ca4b
I20251209 08:58:13.011329 23472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:58:12.999890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.49.66:43589 (local address 127.20.49.126:43381)
1209 08:58:13.000132 (+ 242us) server_negotiation.cc:207] Beginning negotiation
1209 08:58:13.000136 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1209 08:58:13.000705 (+ 569us) server_negotiation.cc:408] Connection header received
1209 08:58:13.001560 (+ 855us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1209 08:58:13.001564 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1209 08:58:13.001626 (+ 62us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1209 08:58:13.001749 (+ 123us) server_negotiation.cc:227] Negotiated authn=SASL
1209 08:58:13.003452 (+ 1703us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:58:13.003994 (+ 542us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1209 08:58:13.004653 (+ 659us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:58:13.004850 (+ 197us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:58:13.007191 (+ 2341us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1209 08:58:13.007219 (+ 28us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1209 08:58:13.007225 (+ 6us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1209 08:58:13.007259 (+ 34us) server_negotiation.cc:923] calling sasl_server_start()
1209 08:58:13.009029 (+ 1770us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1209 08:58:13.009637 (+ 608us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1209 08:58:13.009643 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1209 08:58:13.009648 (+ 5us) server_negotiation.cc:968] calling sasl_server_step()
1209 08:58:13.009734 (+ 86us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1209 08:58:13.010041 (+ 307us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1209 08:58:13.010047 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1209 08:58:13.010052 (+ 5us) server_negotiation.cc:968] calling sasl_server_step()
1209 08:58:13.010259 (+ 207us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1209 08:58:13.010364 (+ 105us) server_negotiation.cc:1036] Waiting for connection context
1209 08:58:13.010991 (+ 627us) server_negotiation.cc:300] Negotiation successful
1209 08:58:13.011153 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":167,"thread_start_us":74,"threads_started":1}
I20251209 08:58:13.011922 23473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:58:13.000070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.49.126:43381 (local address 127.20.49.66:43589)
1209 08:58:13.000516 (+ 446us) negotiation.cc:107] Waiting for socket to connect
1209 08:58:13.000587 (+ 71us) client_negotiation.cc:174] Beginning negotiation
1209 08:58:13.001356 (+ 769us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1209 08:58:13.001921 (+ 565us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1209 08:58:13.001929 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
1209 08:58:13.002326 (+ 397us) client_negotiation.cc:189] Negotiated authn=SASL
1209 08:58:13.003299 (+ 973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:58:13.003313 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:58:13.004116 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1209 08:58:13.004119 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1209 08:58:13.004534 (+ 415us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:58:13.004541 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:58:13.004761 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:58:13.005399 (+ 638us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1209 08:58:13.005422 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
1209 08:58:13.007037 (+ 1615us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1209 08:58:13.009171 (+ 2134us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1209 08:58:13.009179 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1209 08:58:13.009188 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
1209 08:58:13.009501 (+ 313us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1209 08:58:13.009850 (+ 349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1209 08:58:13.009853 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1209 08:58:13.009855 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1209 08:58:13.009950 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1209 08:58:13.010396 (+ 446us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1209 08:58:13.010401 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1209 08:58:13.010718 (+ 317us) client_negotiation.cc:769] Sending connection context
1209 08:58:13.010943 (+ 225us) client_negotiation.cc:240] Negotiation successful
1209 08:58:13.011229 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":108,"threads_started":1}
I20251209 08:58:13.013104 23470 heartbeater.cc:344] Connected to a master server at 127.20.49.126:43381
I20251209 08:58:13.013347 23470 heartbeater.cc:461] Registering TS with master...
I20251209 08:58:13.013890 23470 heartbeater.cc:507] Master 127.20.49.126:43381 requested a full tablet report, sending...
I20251209 08:58:13.014871 23140 ts_manager.cc:194] Registered new tserver with Master: 8956850199b34483a71e10afa2487cf2 (127.20.49.66:46225)
I20251209 08:58:13.015404 23140 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.49.66@KRBTEST.COM'} at 127.20.49.66:43589
WARNING: no policy specified for kudu/127.20.49.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.49.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.49.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765270668420570-20677-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.20.49.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765270668420570-20677-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.20.49.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.20.49.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.20.49.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765270668420570-20677-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.20.49.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765270668420570-20677-0/krb5kdc/kudu.keytab.
I20251209 08:58:13.060256 20677 external_mini_cluster.cc:1366] Running /tmp/dist-test-task8XuDZV/build/debug/bin/kudu
/tmp/dist-test-task8XuDZV/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765270668420570-20677-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765270668420570-20677-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765270668420570-20677-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765270668420570-20677-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.49.67:0
--local_ip_for_outbound_sockets=127.20.49.67
--webserver_interface=127.20.49.67
--webserver_port=0
--tserver_master_addrs=127.20.49.126:43381
--keytab_file=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765270668420570-20677-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.49.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.20.49.84:43811
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765270668420570-20677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765270668420570-20677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765270668420570-20677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251209 08:58:13.167562 23478 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251209 08:58:13.167788 23478 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251209 08:58:13.167878 23478 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251209 08:58:13.171162 23478 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251209 08:58:13.171226 23478 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251209 08:58:13.171304 23478 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.49.67
I20251209 08:58:13.175415 23478 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.49.84:43811
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765270668420570-20677-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765270668420570-20677-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765270668420570-20677-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.49.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.49.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765270668420570-20677-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.49.67
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.20.49.126:43381
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23478
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.49.67
--log_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765270668420570-20677-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 468897755969d686fb08386f42f7835685e7953a
build type DEBUG
built by None at 09 Dec 2025 08:43:15 UTC on 24a791456cd2
build id 9426
I20251209 08:58:13.176486 23478 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251209 08:58:13.177378 23478 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251209 08:58:13.183917 23483 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251209 08:58:13.183917 23486 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251209 08:58:13.184187 23478 server_base.cc:1047] running on GCE node
W20251209 08:58:13.184501 23484 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251209 08:58:13.184887 23478 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251209 08:58:13.185401 23478 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251209 08:58:13.186571 23478 hybrid_clock.cc:648] HybridClock initialized: now 1765270693186545 us; error 46 us; skew 500 ppm
Dec 09 08:58:13 dist-test-slave-8w6c krb5kdc[23092](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765270693, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.49.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251209 08:58:13.189429 23478 init.cc:377] Logged in from keytab as kudu/127.20.49.67@KRBTEST.COM (short username kudu)
I20251209 08:58:13.190778 23478 webserver.cc:492] Webserver started at http://127.20.49.67:41185/ using document root <none> and password file <none>
I20251209 08:58:13.191437 23478 fs_manager.cc:362] Metadata directory not provided
I20251209 08:58:13.191494 23478 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251209 08:58:13.191681 23478 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251209 08:58:13.193413 23478 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765270668420570-20677-0/minicluster-data/ts-2/data/instance:
uuid: "1d27be26e6c5426ba582372f29a8f229"
format_stamp: "Formatted at 2025-12-09 08:58:13 on dist-test-slave-8w6c"
server_key: "5634d92d3654d203a1200d608e1a137f"
server_key_iv: "2946d364845db04f6f8546e3f8b8c61c"
server_key_version: "encryptionkey@0"
I20251209 08:58:13.193905 23478 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765270668420570-20677-0/minicluster-data/ts-2/wal/instance:
uuid: "1d27be26e6c5426ba582372f29a8f229"
format_stamp: "Formatted at 2025-12-09 08:58:13 on dist-test-slave-8w6c"
server_key: "5634d92d3654d203a1200d608e1a137f"
server_key_iv: "2946d364845db04f6f8546e3f8b8c61c"
server_key_version: "encryptionkey@0"
I20251209 08:58:13.197525 23478 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20251209 08:58:13.199894 23493 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251209 08:58:13.200906 23478 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20251209 08:58:13.201014 23478 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765270668420570-20677-0/minicluster-data/ts-2/data,/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765270668420570-20677-0/minicluster-data/ts-2/wal
uuid: "1d27be26e6c5426ba582372f29a8f229"
format_stamp: "Formatted at 2025-12-09 08:58:13 on dist-test-slave-8w6c"
server_key: "5634d92d3654d203a1200d608e1a137f"
server_key_iv: "2946d364845db04f6f8546e3f8b8c61c"
server_key_version: "encryptionkey@0"
I20251209 08:58:13.201102 23478 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765270668420570-20677-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765270668420570-20677-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765270668420570-20677-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251209 08:58:13.212267 23478 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251209 08:58:13.215248 23478 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251209 08:58:13.215430 23478 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251209 08:58:13.215953 23478 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251209 08:58:13.216820 23478 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251209 08:58:13.216871 23478 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251209 08:58:13.216933 23478 ts_tablet_manager.cc:616] Registered 0 tablets
I20251209 08:58:13.216960 23478 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251209 08:58:13.226356 23478 rpc_server.cc:307] RPC server started. Bound to: 127.20.49.67:46145
I20251209 08:58:13.226378 23606 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.49.67:46145 every 8 connection(s)
I20251209 08:58:13.227301 23478 server_base.cc:1179] Dumped server information to /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765270668420570-20677-0/minicluster-data/ts-2/data/info.pb
Dec 09 08:58:13 dist-test-slave-8w6c krb5kdc[23092](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765270693, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.49.67@KRBTEST.COM for kudu/127.20.49.126@KRBTEST.COM
I20251209 08:58:13.236696 20677 external_mini_cluster.cc:1428] Started /tmp/dist-test-task8XuDZV/build/debug/bin/kudu as pid 23478
I20251209 08:58:13.236868 20677 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1765270668420570-20677-0/minicluster-data/ts-2/wal/instance
I20251209 08:58:13.237116 20677 external_mini_cluster.cc:1468] Setting key 7c1ef3071c7ef8298b0a274aa4303955
I20251209 08:58:13.240463 23472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:58:13.228999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.49.67:42887 (local address 127.20.49.126:43381)
1209 08:58:13.229157 (+ 158us) server_negotiation.cc:207] Beginning negotiation
1209 08:58:13.229160 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1209 08:58:13.229849 (+ 689us) server_negotiation.cc:408] Connection header received
1209 08:58:13.231098 (+ 1249us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1209 08:58:13.231102 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1209 08:58:13.231166 (+ 64us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1209 08:58:13.231260 (+ 94us) server_negotiation.cc:227] Negotiated authn=SASL
1209 08:58:13.232948 (+ 1688us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:58:13.233455 (+ 507us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1209 08:58:13.234191 (+ 736us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:58:13.234339 (+ 148us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:58:13.236502 (+ 2163us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1209 08:58:13.236530 (+ 28us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1209 08:58:13.236536 (+ 6us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1209 08:58:13.236608 (+ 72us) server_negotiation.cc:923] calling sasl_server_start()
1209 08:58:13.238421 (+ 1813us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1209 08:58:13.239028 (+ 607us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1209 08:58:13.239031 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1209 08:58:13.239032 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1209 08:58:13.239079 (+ 47us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1209 08:58:13.239392 (+ 313us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1209 08:58:13.239395 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1209 08:58:13.239397 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1209 08:58:13.239562 (+ 165us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1209 08:58:13.239652 (+ 90us) server_negotiation.cc:1036] Waiting for connection context
1209 08:58:13.240159 (+ 507us) server_negotiation.cc:300] Negotiation successful
1209 08:58:13.240298 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20251209 08:58:13.241055 23609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:58:13.229224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.49.126:43381 (local address 127.20.49.67:42887)
1209 08:58:13.229697 (+ 473us) negotiation.cc:107] Waiting for socket to connect
1209 08:58:13.229751 (+ 54us) client_negotiation.cc:174] Beginning negotiation
1209 08:58:13.230860 (+ 1109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1209 08:58:13.231396 (+ 536us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1209 08:58:13.231404 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
1209 08:58:13.231833 (+ 429us) client_negotiation.cc:189] Negotiated authn=SASL
1209 08:58:13.232782 (+ 949us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:58:13.232797 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:58:13.233591 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1209 08:58:13.233593 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1209 08:58:13.234081 (+ 488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:58:13.234090 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:58:13.234261 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:58:13.234859 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1209 08:58:13.234881 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
1209 08:58:13.236329 (+ 1448us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1209 08:58:13.238559 (+ 2230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1209 08:58:13.238566 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1209 08:58:13.238578 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
1209 08:58:13.238915 (+ 337us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1209 08:58:13.239161 (+ 246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1209 08:58:13.239164 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1209 08:58:13.239166 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1209 08:58:13.239291 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1209 08:58:13.239668 (+ 377us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1209 08:58:13.239676 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1209 08:58:13.239919 (+ 243us) client_negotiation.cc:769] Sending connection context
1209 08:58:13.240109 (+ 190us) client_negotiation.cc:240] Negotiation successful
1209 08:58:13.240376 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":110,"threads_started":1}
I20251209 08:58:13.242128 23607 heartbeater.cc:344] Connected to a master server at 127.20.49.126:43381
I20251209 08:58:13.242372 23607 heartbeater.cc:461] Registering TS with master...
I20251209 08:58:13.242882 23607 heartbeater.cc:507] Master 127.20.49.126:43381 requested a full tablet report, sending...
I20251209 08:58:13.243834 23140 ts_manager.cc:194] Registered new tserver with Master: 1d27be26e6c5426ba582372f29a8f229 (127.20.49.67:46145)
I20251209 08:58:13.244375 23140 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.49.67@KRBTEST.COM'} at 127.20.49.67:42887
I20251209 08:58:13.251644 20677 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
W20251209 08:58:13.264534 23472 negotiation.cc:337] Failed RPC negotiation. Trace:
1209 08:58:13.263223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38582 (local address 127.20.49.126:43381)
1209 08:58:13.263405 (+ 182us) server_negotiation.cc:207] Beginning negotiation
1209 08:58:13.263409 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1209 08:58:13.263481 (+ 72us) server_negotiation.cc:408] Connection header received
1209 08:58:13.263602 (+ 121us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1209 08:58:13.263605 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1209 08:58:13.263655 (+ 50us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1209 08:58:13.263732 (+ 77us) server_negotiation.cc:227] Negotiated authn=SASL
1209 08:58:13.264402 (+ 670us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38582: BlockingRecv error: recv got EOF from 127.0.0.1:38582 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
I20251209 08:58:13.266057 20677 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task8XuDZV/build/debug/bin/kudu with pid 23204
I20251209 08:58:13.272224 20677 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task8XuDZV/build/debug/bin/kudu with pid 23341
I20251209 08:58:13.278000 20677 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task8XuDZV/build/debug/bin/kudu with pid 23478
I20251209 08:58:13.283980 20677 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task8XuDZV/build/debug/bin/kudu with pid 23108
2025-12-09T08:58:13Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (6778 ms)
[ RUN ] SecurityITest.TestRebalanceReportUnauthorized
Loading random data
Initializing database '/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765270668420570-20677-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Dec 09 08:58:13 dist-test-slave-8w6c krb5kdc[23619](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Dec 09 08:58:13 dist-test-slave-8w6c krb5kdc[23619](info): set up 2 sockets
Dec 09 08:58:13 dist-test-slave-8w6c krb5kdc[23619](info): commencing operation
krb5kdc: starting...
W20251209 08:58:15.278246 20677 mini_kdc.cc:121] Time spent starting KDC: real 1.972s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765270668420570-20677-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765270668420570-20677-0/krb5kdc/test-user.keytab.
Dec 09 08:58:15 dist-test-slave-8w6c krb5kdc[23619](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765270695, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-12-09T08:58:15Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-12-09T08:58:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.49.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.49.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.49.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765270668420570-20677-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.20.49.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765270668420570-20677-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.20.49.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.20.49.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.20.49.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765270668420570-20677-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.20.49.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765270668420570-20677-0/krb5kdc/kudu.keytab.
I20251209 08:58:15.418831 20677 external_mini_cluster.cc:1366] Running /tmp/dist-test-task8XuDZV/build/debug/bin/kudu
/tmp/dist-test-task8XuDZV/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765270668420570-20677-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765270668420570-20677-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765270668420570-20677-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765270668420570-20677-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.49.126:38079
--webserver_interface=127.20.49.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765270668420570-20677-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.49.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.20.49.84:44993
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.49.126:38079
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765270668420570-20677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765270668420570-20677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765270668420570-20677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251209 08:58:15.526662 23635 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251209 08:58:15.526908 23635 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251209 08:58:15.526964 23635 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251209 08:58:15.530169 23635 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251209 08:58:15.530230 23635 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251209 08:58:15.530253 23635 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251209 08:58:15.530272 23635 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251209 08:58:15.530290 23635 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251209 08:58:15.534528 23635 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.49.84:44993
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765270668420570-20677-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765270668420570-20677-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.49.126:38079
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765270668420570-20677-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.49.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.49.126:38079
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765270668420570-20677-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.49.126
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23635
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765270668420570-20677-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 468897755969d686fb08386f42f7835685e7953a
build type DEBUG
built by None at 09 Dec 2025 08:43:15 UTC on 24a791456cd2
build id 9426
I20251209 08:58:15.535544 23635 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251209 08:58:15.536355 23635 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251209 08:58:15.541965 23641 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251209 08:58:15.541952 23643 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251209 08:58:15.541952 23640 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251209 08:58:15.542219 23635 server_base.cc:1047] running on GCE node
I20251209 08:58:15.542778 23635 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251209 08:58:15.543646 23635 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251209 08:58:15.544806 23635 hybrid_clock.cc:648] HybridClock initialized: now 1765270695544791 us; error 38 us; skew 500 ppm
Dec 09 08:58:15 dist-test-slave-8w6c krb5kdc[23619](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765270695, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.49.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251209 08:58:15.547348 23635 init.cc:377] Logged in from keytab as kudu/127.20.49.126@KRBTEST.COM (short username kudu)
I20251209 08:58:15.548384 23635 webserver.cc:492] Webserver started at http://127.20.49.126:40985/ using document root <none> and password file <none>
I20251209 08:58:15.548923 23635 fs_manager.cc:362] Metadata directory not provided
I20251209 08:58:15.548977 23635 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251209 08:58:15.549153 23635 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251209 08:58:15.550863 23635 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765270668420570-20677-0/minicluster-data/master-0/data/instance:
uuid: "5fb810ead6cc430ea4ebc34348867881"
format_stamp: "Formatted at 2025-12-09 08:58:15 on dist-test-slave-8w6c"
server_key: "95c7f82dcc0dedebd2b6b34750b82b9a"
server_key_iv: "0d3a7cb18275556b0458663a8799a7bd"
server_key_version: "encryptionkey@0"
I20251209 08:58:15.551343 23635 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765270668420570-20677-0/minicluster-data/master-0/wal/instance:
uuid: "5fb810ead6cc430ea4ebc34348867881"
format_stamp: "Formatted at 2025-12-09 08:58:15 on dist-test-slave-8w6c"
server_key: "95c7f82dcc0dedebd2b6b34750b82b9a"
server_key_iv: "0d3a7cb18275556b0458663a8799a7bd"
server_key_version: "encryptionkey@0"
I20251209 08:58:15.554668 23635 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20251209 08:58:15.556759 23650 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251209 08:58:15.557770 23635 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20251209 08:58:15.557889 23635 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765270668420570-20677-0/minicluster-data/master-0/data,/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765270668420570-20677-0/minicluster-data/master-0/wal
uuid: "5fb810ead6cc430ea4ebc34348867881"
format_stamp: "Formatted at 2025-12-09 08:58:15 on dist-test-slave-8w6c"
server_key: "95c7f82dcc0dedebd2b6b34750b82b9a"
server_key_iv: "0d3a7cb18275556b0458663a8799a7bd"
server_key_version: "encryptionkey@0"
I20251209 08:58:15.557974 23635 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765270668420570-20677-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765270668420570-20677-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765270668420570-20677-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251209 08:58:15.594808 23635 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251209 08:58:15.605011 23635 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251209 08:58:15.605211 23635 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251209 08:58:15.612689 23702 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.49.126:38079 every 8 connection(s)
I20251209 08:58:15.612695 23635 rpc_server.cc:307] RPC server started. Bound to: 127.20.49.126:38079
I20251209 08:58:15.613811 23635 server_base.cc:1179] Dumped server information to /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765270668420570-20677-0/minicluster-data/master-0/data/info.pb
I20251209 08:58:15.615260 20677 external_mini_cluster.cc:1428] Started /tmp/dist-test-task8XuDZV/build/debug/bin/kudu as pid 23635
I20251209 08:58:15.615398 20677 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765270668420570-20677-0/minicluster-data/master-0/wal/instance
I20251209 08:58:15.615690 20677 external_mini_cluster.cc:1468] Setting key bfedd207e627c7c1f89c996d7a9201b0
I20251209 08:58:15.617201 23703 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251209 08:58:15.624409 23703 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5fb810ead6cc430ea4ebc34348867881: Bootstrap starting.
Dec 09 08:58:15 dist-test-slave-8w6c krb5kdc[23619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765270695, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.49.126@KRBTEST.COM
I20251209 08:58:15.627207 23703 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5fb810ead6cc430ea4ebc34348867881: Neither blocks nor log segments found. Creating new log.
I20251209 08:58:15.627915 23703 log.cc:826] T 00000000000000000000000000000000 P 5fb810ead6cc430ea4ebc34348867881: Log is configured to *not* fsync() on all Append() calls
I20251209 08:58:15.630080 23703 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5fb810ead6cc430ea4ebc34348867881: No bootstrap required, opened a new log
I20251209 08:58:15.630340 23706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:58:15.617348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60502 (local address 127.20.49.126:38079)
1209 08:58:15.617843 (+ 495us) server_negotiation.cc:207] Beginning negotiation
1209 08:58:15.617857 (+ 14us) server_negotiation.cc:400] Waiting for connection header
1209 08:58:15.617913 (+ 56us) server_negotiation.cc:408] Connection header received
1209 08:58:15.619362 (+ 1449us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1209 08:58:15.619393 (+ 31us) server_negotiation.cc:462] Received NEGOTIATE request from client
1209 08:58:15.619777 (+ 384us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1209 08:58:15.620152 (+ 375us) server_negotiation.cc:227] Negotiated authn=SASL
1209 08:58:15.621234 (+ 1082us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:58:15.622140 (+ 906us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1209 08:58:15.623032 (+ 892us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:58:15.623241 (+ 209us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:58:15.625864 (+ 2623us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1209 08:58:15.625916 (+ 52us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1209 08:58:15.625926 (+ 10us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1209 08:58:15.625960 (+ 34us) server_negotiation.cc:923] calling sasl_server_start()
1209 08:58:15.627776 (+ 1816us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1209 08:58:15.628229 (+ 453us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1209 08:58:15.628234 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1209 08:58:15.628238 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
1209 08:58:15.628311 (+ 73us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1209 08:58:15.628537 (+ 226us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1209 08:58:15.628544 (+ 7us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1209 08:58:15.628548 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
1209 08:58:15.628962 (+ 414us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1209 08:58:15.629130 (+ 168us) server_negotiation.cc:1036] Waiting for connection context
1209 08:58:15.629409 (+ 279us) server_negotiation.cc:300] Negotiation successful
1209 08:58:15.629668 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":304,"thread_start_us":141,"threads_started":1}
I20251209 08:58:15.632884 23703 raft_consensus.cc:359] T 00000000000000000000000000000000 P 5fb810ead6cc430ea4ebc34348867881 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fb810ead6cc430ea4ebc34348867881" member_type: VOTER last_known_addr { host: "127.20.49.126" port: 38079 } }
I20251209 08:58:15.633071 23703 raft_consensus.cc:385] T 00000000000000000000000000000000 P 5fb810ead6cc430ea4ebc34348867881 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251209 08:58:15.633138 23703 raft_consensus.cc:740] T 00000000000000000000000000000000 P 5fb810ead6cc430ea4ebc34348867881 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fb810ead6cc430ea4ebc34348867881, State: Initialized, Role: FOLLOWER
I20251209 08:58:15.633520 23703 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5fb810ead6cc430ea4ebc34348867881 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fb810ead6cc430ea4ebc34348867881" member_type: VOTER last_known_addr { host: "127.20.49.126" port: 38079 } }
I20251209 08:58:15.633639 23703 raft_consensus.cc:399] T 00000000000000000000000000000000 P 5fb810ead6cc430ea4ebc34348867881 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251209 08:58:15.633733 23703 raft_consensus.cc:493] T 00000000000000000000000000000000 P 5fb810ead6cc430ea4ebc34348867881 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251209 08:58:15.633826 23703 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 5fb810ead6cc430ea4ebc34348867881 [term 0 FOLLOWER]: Advancing to term 1
I20251209 08:58:15.634629 23703 raft_consensus.cc:515] T 00000000000000000000000000000000 P 5fb810ead6cc430ea4ebc34348867881 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fb810ead6cc430ea4ebc34348867881" member_type: VOTER last_known_addr { host: "127.20.49.126" port: 38079 } }
I20251209 08:58:15.634948 23703 leader_election.cc:304] T 00000000000000000000000000000000 P 5fb810ead6cc430ea4ebc34348867881 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5fb810ead6cc430ea4ebc34348867881; no voters:
I20251209 08:58:15.635226 23703 leader_election.cc:290] T 00000000000000000000000000000000 P 5fb810ead6cc430ea4ebc34348867881 [CANDIDATE]: Term 1 election: Requested vote from peers
I20251209 08:58:15.635542 23708 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 5fb810ead6cc430ea4ebc34348867881 [term 1 FOLLOWER]: Leader election won for term 1
I20251209 08:58:15.635739 23708 raft_consensus.cc:697] T 00000000000000000000000000000000 P 5fb810ead6cc430ea4ebc34348867881 [term 1 LEADER]: Becoming Leader. State: Replica: 5fb810ead6cc430ea4ebc34348867881, State: Running, Role: LEADER
I20251209 08:58:15.636001 23708 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5fb810ead6cc430ea4ebc34348867881 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fb810ead6cc430ea4ebc34348867881" member_type: VOTER last_known_addr { host: "127.20.49.126" port: 38079 } }
I20251209 08:58:15.636327 23703 sys_catalog.cc:565] T 00000000000000000000000000000000 P 5fb810ead6cc430ea4ebc34348867881 [sys.catalog]: configured and running, proceeding with master startup.
I20251209 08:58:15.637409 23709 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5fb810ead6cc430ea4ebc34348867881 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5fb810ead6cc430ea4ebc34348867881" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fb810ead6cc430ea4ebc34348867881" member_type: VOTER last_known_addr { host: "127.20.49.126" port: 38079 } } }
I20251209 08:58:15.637530 23709 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5fb810ead6cc430ea4ebc34348867881 [sys.catalog]: This master's current role is: LEADER
I20251209 08:58:15.637898 23710 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5fb810ead6cc430ea4ebc34348867881 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5fb810ead6cc430ea4ebc34348867881. Latest consensus state: current_term: 1 leader_uuid: "5fb810ead6cc430ea4ebc34348867881" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fb810ead6cc430ea4ebc34348867881" member_type: VOTER last_known_addr { host: "127.20.49.126" port: 38079 } } }
I20251209 08:58:15.637992 23710 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5fb810ead6cc430ea4ebc34348867881 [sys.catalog]: This master's current role is: LEADER
W20251209 08:58:15.641072 23723 catalog_manager.cc:1568] T 00000000000000000000000000000000 P 5fb810ead6cc430ea4ebc34348867881: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20251209 08:58:15.641172 23723 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20251209 08:58:15.641307 23717 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251209 08:58:15.642062 23717 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251209 08:58:15.646864 23717 catalog_manager.cc:1357] Generated new cluster ID: 861934a16d6f404bbffa5d868c01c6db
I20251209 08:58:15.646939 23717 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251209 08:58:15.656031 23717 catalog_manager.cc:1380] Generated new certificate authority record
I20251209 08:58:15.656883 23717 catalog_manager.cc:1514] Loading token signing keys...
I20251209 08:58:15.667286 23717 catalog_manager.cc:6027] T 00000000000000000000000000000000 P 5fb810ead6cc430ea4ebc34348867881: Generated new TSK 0
I20251209 08:58:15.668020 23717 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.20.49.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.49.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.49.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765270668420570-20677-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.20.49.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765270668420570-20677-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.20.49.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.20.49.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.20.49.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765270668420570-20677-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.20.49.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765270668420570-20677-0/krb5kdc/kudu.keytab.
I20251209 08:58:15.726758 20677 external_mini_cluster.cc:1366] Running /tmp/dist-test-task8XuDZV/build/debug/bin/kudu
/tmp/dist-test-task8XuDZV/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765270668420570-20677-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765270668420570-20677-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765270668420570-20677-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765270668420570-20677-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.49.65:0
--local_ip_for_outbound_sockets=127.20.49.65
--webserver_interface=127.20.49.65
--webserver_port=0
--tserver_master_addrs=127.20.49.126:38079
--keytab_file=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765270668420570-20677-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.49.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.20.49.84:44993
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765270668420570-20677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765270668420570-20677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765270668420570-20677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251209 08:58:15.837074 23731 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251209 08:58:15.837321 23731 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251209 08:58:15.837384 23731 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251209 08:58:15.840858 23731 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251209 08:58:15.840940 23731 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251209 08:58:15.841045 23731 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.49.65
I20251209 08:58:15.845412 23731 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.49.84:44993
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765270668420570-20677-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765270668420570-20677-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765270668420570-20677-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.49.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.49.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765270668420570-20677-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.49.65
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.20.49.126:38079
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23731
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.49.65
--log_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765270668420570-20677-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 468897755969d686fb08386f42f7835685e7953a
build type DEBUG
built by None at 09 Dec 2025 08:43:15 UTC on 24a791456cd2
build id 9426
I20251209 08:58:15.846576 23731 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251209 08:58:15.847451 23731 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251209 08:58:15.854422 23736 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251209 08:58:15.854387 23739 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251209 08:58:15.854383 23737 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251209 08:58:15.854645 23731 server_base.cc:1047] running on GCE node
I20251209 08:58:15.855042 23731 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251209 08:58:15.855621 23731 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251209 08:58:15.856838 23731 hybrid_clock.cc:648] HybridClock initialized: now 1765270695856812 us; error 46 us; skew 500 ppm
Dec 09 08:58:15 dist-test-slave-8w6c krb5kdc[23619](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765270695, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.49.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251209 08:58:15.859894 23731 init.cc:377] Logged in from keytab as kudu/127.20.49.65@KRBTEST.COM (short username kudu)
I20251209 08:58:15.861047 23731 webserver.cc:492] Webserver started at http://127.20.49.65:39971/ using document root <none> and password file <none>
I20251209 08:58:15.861609 23731 fs_manager.cc:362] Metadata directory not provided
I20251209 08:58:15.861665 23731 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251209 08:58:15.861901 23731 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251209 08:58:15.863701 23731 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765270668420570-20677-0/minicluster-data/ts-0/data/instance:
uuid: "a5e5b6c5683645db8fa5ad2ec958de2d"
format_stamp: "Formatted at 2025-12-09 08:58:15 on dist-test-slave-8w6c"
server_key: "116ee1f316bc788c0a80e62692b347dc"
server_key_iv: "17e77855ffd9f9d79bdcbfe9d9a5a525"
server_key_version: "encryptionkey@0"
I20251209 08:58:15.864197 23731 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765270668420570-20677-0/minicluster-data/ts-0/wal/instance:
uuid: "a5e5b6c5683645db8fa5ad2ec958de2d"
format_stamp: "Formatted at 2025-12-09 08:58:15 on dist-test-slave-8w6c"
server_key: "116ee1f316bc788c0a80e62692b347dc"
server_key_iv: "17e77855ffd9f9d79bdcbfe9d9a5a525"
server_key_version: "encryptionkey@0"
I20251209 08:58:15.867851 23731 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20251209 08:58:15.870438 23746 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251209 08:58:15.871536 23731 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20251209 08:58:15.871657 23731 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765270668420570-20677-0/minicluster-data/ts-0/data,/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765270668420570-20677-0/minicluster-data/ts-0/wal
uuid: "a5e5b6c5683645db8fa5ad2ec958de2d"
format_stamp: "Formatted at 2025-12-09 08:58:15 on dist-test-slave-8w6c"
server_key: "116ee1f316bc788c0a80e62692b347dc"
server_key_iv: "17e77855ffd9f9d79bdcbfe9d9a5a525"
server_key_version: "encryptionkey@0"
I20251209 08:58:15.871762 23731 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765270668420570-20677-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765270668420570-20677-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765270668420570-20677-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251209 08:58:15.913529 23731 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251209 08:58:15.916488 23731 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251209 08:58:15.916675 23731 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251209 08:58:15.917240 23731 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251209 08:58:15.918148 23731 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251209 08:58:15.918200 23731 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251209 08:58:15.918259 23731 ts_tablet_manager.cc:616] Registered 0 tablets
I20251209 08:58:15.918291 23731 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251209 08:58:15.927495 23731 rpc_server.cc:307] RPC server started. Bound to: 127.20.49.65:45449
I20251209 08:58:15.927536 23859 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.49.65:45449 every 8 connection(s)
I20251209 08:58:15.928447 23731 server_base.cc:1179] Dumped server information to /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765270668420570-20677-0/minicluster-data/ts-0/data/info.pb
I20251209 08:58:15.933519 20677 external_mini_cluster.cc:1428] Started /tmp/dist-test-task8XuDZV/build/debug/bin/kudu as pid 23731
I20251209 08:58:15.933655 20677 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765270668420570-20677-0/minicluster-data/ts-0/wal/instance
I20251209 08:58:15.934060 20677 external_mini_cluster.cc:1468] Setting key 3b44cbd93c9652a620aacc0cb8996df6
Dec 09 08:58:15 dist-test-slave-8w6c krb5kdc[23619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765270695, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.49.65@KRBTEST.COM for kudu/127.20.49.126@KRBTEST.COM
I20251209 08:58:15.941847 23706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:58:15.930214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.49.65:42655 (local address 127.20.49.126:38079)
1209 08:58:15.930387 (+ 173us) server_negotiation.cc:207] Beginning negotiation
1209 08:58:15.930393 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1209 08:58:15.931042 (+ 649us) server_negotiation.cc:408] Connection header received
1209 08:58:15.931841 (+ 799us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1209 08:58:15.931845 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1209 08:58:15.931897 (+ 52us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1209 08:58:15.931995 (+ 98us) server_negotiation.cc:227] Negotiated authn=SASL
1209 08:58:15.933753 (+ 1758us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:58:15.934294 (+ 541us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1209 08:58:15.934933 (+ 639us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:58:15.935113 (+ 180us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:58:15.937600 (+ 2487us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1209 08:58:15.937634 (+ 34us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1209 08:58:15.937663 (+ 29us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1209 08:58:15.937720 (+ 57us) server_negotiation.cc:923] calling sasl_server_start()
1209 08:58:15.939454 (+ 1734us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1209 08:58:15.940162 (+ 708us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1209 08:58:15.940166 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1209 08:58:15.940167 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1209 08:58:15.940213 (+ 46us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1209 08:58:15.940586 (+ 373us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1209 08:58:15.940588 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1209 08:58:15.940589 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1209 08:58:15.940769 (+ 180us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1209 08:58:15.940842 (+ 73us) server_negotiation.cc:1036] Waiting for connection context
1209 08:58:15.941554 (+ 712us) server_negotiation.cc:300] Negotiation successful
1209 08:58:15.941716 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20251209 08:58:15.942827 23862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:58:15.930462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.49.126:38079 (local address 127.20.49.65:42655)
1209 08:58:15.930894 (+ 432us) negotiation.cc:107] Waiting for socket to connect
1209 08:58:15.930932 (+ 38us) client_negotiation.cc:174] Beginning negotiation
1209 08:58:15.931644 (+ 712us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1209 08:58:15.932139 (+ 495us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1209 08:58:15.932146 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
1209 08:58:15.932551 (+ 405us) client_negotiation.cc:189] Negotiated authn=SASL
1209 08:58:15.933546 (+ 995us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:58:15.933560 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:58:15.934418 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1209 08:58:15.934422 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1209 08:58:15.934815 (+ 393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:58:15.934822 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:58:15.935058 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:58:15.935699 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1209 08:58:15.935725 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
1209 08:58:15.937424 (+ 1699us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1209 08:58:15.939616 (+ 2192us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1209 08:58:15.939625 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1209 08:58:15.939643 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
1209 08:58:15.940030 (+ 387us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1209 08:58:15.940311 (+ 281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1209 08:58:15.940318 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1209 08:58:15.940323 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
1209 08:58:15.940489 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1209 08:58:15.940872 (+ 383us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1209 08:58:15.940881 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1209 08:58:15.941249 (+ 368us) client_negotiation.cc:769] Sending connection context
1209 08:58:15.941545 (+ 296us) client_negotiation.cc:240] Negotiation successful
1209 08:58:15.941909 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":98,"threads_started":1}
WARNING: no policy specified for kudu/127.20.49.66@KRBTEST.COM; defaulting to no policy
I20251209 08:58:15.943989 23860 heartbeater.cc:344] Connected to a master server at 127.20.49.126:38079
I20251209 08:58:15.944276 23860 heartbeater.cc:461] Registering TS with master...
I20251209 08:58:15.944833 23860 heartbeater.cc:507] Master 127.20.49.126:38079 requested a full tablet report, sending...
I20251209 08:58:15.946291 23667 ts_manager.cc:194] Registered new tserver with Master: a5e5b6c5683645db8fa5ad2ec958de2d (127.20.49.65:45449)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.49.66@KRBTEST.COM" created.
I20251209 08:58:15.947844 23667 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.49.65@KRBTEST.COM'} at 127.20.49.65:42655
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.49.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765270668420570-20677-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.20.49.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765270668420570-20677-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.20.49.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.20.49.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.20.49.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765270668420570-20677-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.20.49.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765270668420570-20677-0/krb5kdc/kudu.keytab.
I20251209 08:58:15.982952 20677 external_mini_cluster.cc:1366] Running /tmp/dist-test-task8XuDZV/build/debug/bin/kudu
/tmp/dist-test-task8XuDZV/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765270668420570-20677-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765270668420570-20677-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765270668420570-20677-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765270668420570-20677-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.49.66:0
--local_ip_for_outbound_sockets=127.20.49.66
--webserver_interface=127.20.49.66
--webserver_port=0
--tserver_master_addrs=127.20.49.126:38079
--keytab_file=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765270668420570-20677-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.49.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.20.49.84:44993
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765270668420570-20677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765270668420570-20677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765270668420570-20677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251209 08:58:16.092242 23867 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251209 08:58:16.092510 23867 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251209 08:58:16.092607 23867 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251209 08:58:16.096370 23867 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251209 08:58:16.096446 23867 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251209 08:58:16.096565 23867 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.49.66
I20251209 08:58:16.100867 23867 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.49.84:44993
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765270668420570-20677-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765270668420570-20677-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765270668420570-20677-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.49.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.49.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765270668420570-20677-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.49.66
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.20.49.126:38079
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23867
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.49.66
--log_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765270668420570-20677-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 468897755969d686fb08386f42f7835685e7953a
build type DEBUG
built by None at 09 Dec 2025 08:43:15 UTC on 24a791456cd2
build id 9426
I20251209 08:58:16.102156 23867 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251209 08:58:16.103053 23867 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251209 08:58:16.109400 23872 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251209 08:58:16.109391 23875 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251209 08:58:16.109393 23873 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251209 08:58:16.109632 23867 server_base.cc:1047] running on GCE node
I20251209 08:58:16.110004 23867 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251209 08:58:16.110508 23867 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251209 08:58:16.111706 23867 hybrid_clock.cc:648] HybridClock initialized: now 1765270696111688 us; error 29 us; skew 500 ppm
Dec 09 08:58:16 dist-test-slave-8w6c krb5kdc[23619](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765270696, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.49.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251209 08:58:16.114295 23867 init.cc:377] Logged in from keytab as kudu/127.20.49.66@KRBTEST.COM (short username kudu)
I20251209 08:58:16.115314 23867 webserver.cc:492] Webserver started at http://127.20.49.66:45393/ using document root <none> and password file <none>
I20251209 08:58:16.115851 23867 fs_manager.cc:362] Metadata directory not provided
I20251209 08:58:16.115906 23867 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251209 08:58:16.116086 23867 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251209 08:58:16.117863 23867 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765270668420570-20677-0/minicluster-data/ts-1/data/instance:
uuid: "ae65823c89b34550af1796e20434d554"
format_stamp: "Formatted at 2025-12-09 08:58:16 on dist-test-slave-8w6c"
server_key: "ed92c52fc48ccef970c66cac7b2a1d4f"
server_key_iv: "da34d938b1b63c09ad69490a18c247f2"
server_key_version: "encryptionkey@0"
I20251209 08:58:16.118322 23867 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765270668420570-20677-0/minicluster-data/ts-1/wal/instance:
uuid: "ae65823c89b34550af1796e20434d554"
format_stamp: "Formatted at 2025-12-09 08:58:16 on dist-test-slave-8w6c"
server_key: "ed92c52fc48ccef970c66cac7b2a1d4f"
server_key_iv: "da34d938b1b63c09ad69490a18c247f2"
server_key_version: "encryptionkey@0"
I20251209 08:58:16.121598 23867 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20251209 08:58:16.123847 23882 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251209 08:58:16.124826 23867 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20251209 08:58:16.124934 23867 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765270668420570-20677-0/minicluster-data/ts-1/data,/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765270668420570-20677-0/minicluster-data/ts-1/wal
uuid: "ae65823c89b34550af1796e20434d554"
format_stamp: "Formatted at 2025-12-09 08:58:16 on dist-test-slave-8w6c"
server_key: "ed92c52fc48ccef970c66cac7b2a1d4f"
server_key_iv: "da34d938b1b63c09ad69490a18c247f2"
server_key_version: "encryptionkey@0"
I20251209 08:58:16.125034 23867 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765270668420570-20677-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765270668420570-20677-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765270668420570-20677-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251209 08:58:16.143654 23867 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251209 08:58:16.146488 23867 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251209 08:58:16.146675 23867 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251209 08:58:16.147210 23867 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251209 08:58:16.148108 23867 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251209 08:58:16.148159 23867 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251209 08:58:16.148226 23867 ts_tablet_manager.cc:616] Registered 0 tablets
I20251209 08:58:16.148258 23867 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251209 08:58:16.156946 23867 rpc_server.cc:307] RPC server started. Bound to: 127.20.49.66:45279
I20251209 08:58:16.156975 23995 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.49.66:45279 every 8 connection(s)
I20251209 08:58:16.157939 23867 server_base.cc:1179] Dumped server information to /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765270668420570-20677-0/minicluster-data/ts-1/data/info.pb
I20251209 08:58:16.159356 20677 external_mini_cluster.cc:1428] Started /tmp/dist-test-task8XuDZV/build/debug/bin/kudu as pid 23867
I20251209 08:58:16.159477 20677 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765270668420570-20677-0/minicluster-data/ts-1/wal/instance
I20251209 08:58:16.159754 20677 external_mini_cluster.cc:1468] Setting key c7b8ef05eea6e4d35aec468651003765
Dec 09 08:58:16 dist-test-slave-8w6c krb5kdc[23619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765270696, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.49.66@KRBTEST.COM for kudu/127.20.49.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.20.49.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.49.67@KRBTEST.COM" created.
I20251209 08:58:16.170784 23706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:58:16.159720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.49.66:34673 (local address 127.20.49.126:38079)
1209 08:58:16.159865 (+ 145us) server_negotiation.cc:207] Beginning negotiation
1209 08:58:16.159870 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1209 08:58:16.160693 (+ 823us) server_negotiation.cc:408] Connection header received
1209 08:58:16.161672 (+ 979us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1209 08:58:16.161709 (+ 37us) server_negotiation.cc:462] Received NEGOTIATE request from client
1209 08:58:16.161765 (+ 56us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1209 08:58:16.161844 (+ 79us) server_negotiation.cc:227] Negotiated authn=SASL
1209 08:58:16.163588 (+ 1744us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:58:16.164090 (+ 502us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1209 08:58:16.164710 (+ 620us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:58:16.164887 (+ 177us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:58:16.167141 (+ 2254us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1209 08:58:16.167159 (+ 18us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1209 08:58:16.167161 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1209 08:58:16.167185 (+ 24us) server_negotiation.cc:923] calling sasl_server_start()
1209 08:58:16.168726 (+ 1541us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1209 08:58:16.169205 (+ 479us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1209 08:58:16.169239 (+ 34us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1209 08:58:16.169240 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1209 08:58:16.169284 (+ 44us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1209 08:58:16.169564 (+ 280us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1209 08:58:16.169568 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1209 08:58:16.169570 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1209 08:58:16.169783 (+ 213us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1209 08:58:16.169872 (+ 89us) server_negotiation.cc:1036] Waiting for connection context
1209 08:58:16.170433 (+ 561us) server_negotiation.cc:300] Negotiation successful
1209 08:58:16.170624 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20251209 08:58:16.171394 23998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:58:16.160076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.49.126:38079 (local address 127.20.49.66:34673)
1209 08:58:16.160543 (+ 467us) negotiation.cc:107] Waiting for socket to connect
1209 08:58:16.160589 (+ 46us) client_negotiation.cc:174] Beginning negotiation
1209 08:58:16.161471 (+ 882us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1209 08:58:16.162035 (+ 564us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1209 08:58:16.162044 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
1209 08:58:16.162495 (+ 451us) client_negotiation.cc:189] Negotiated authn=SASL
1209 08:58:16.163439 (+ 944us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:58:16.163452 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:58:16.164213 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1209 08:58:16.164216 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1209 08:58:16.164596 (+ 380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:58:16.164602 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:58:16.164818 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:58:16.165467 (+ 649us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1209 08:58:16.165491 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
1209 08:58:16.166985 (+ 1494us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1209 08:58:16.168851 (+ 1866us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1209 08:58:16.168856 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1209 08:58:16.168869 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
1209 08:58:16.169103 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1209 08:58:16.169371 (+ 268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1209 08:58:16.169374 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1209 08:58:16.169375 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1209 08:58:16.169475 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1209 08:58:16.169896 (+ 421us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1209 08:58:16.169903 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1209 08:58:16.170172 (+ 269us) client_negotiation.cc:769] Sending connection context
1209 08:58:16.170376 (+ 204us) client_negotiation.cc:240] Negotiation successful
1209 08:58:16.170665 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":115,"threads_started":1}
I20251209 08:58:16.172506 23996 heartbeater.cc:344] Connected to a master server at 127.20.49.126:38079
I20251209 08:58:16.172750 23996 heartbeater.cc:461] Registering TS with master...
I20251209 08:58:16.173247 23996 heartbeater.cc:507] Master 127.20.49.126:38079 requested a full tablet report, sending...
I20251209 08:58:16.174266 23667 ts_manager.cc:194] Registered new tserver with Master: ae65823c89b34550af1796e20434d554 (127.20.49.66:45279)
I20251209 08:58:16.174809 23667 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.49.66@KRBTEST.COM'} at 127.20.49.66:34673
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.49.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765270668420570-20677-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.20.49.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765270668420570-20677-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.20.49.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.20.49.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.20.49.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765270668420570-20677-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.20.49.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765270668420570-20677-0/krb5kdc/kudu.keytab.
I20251209 08:58:16.209563 20677 external_mini_cluster.cc:1366] Running /tmp/dist-test-task8XuDZV/build/debug/bin/kudu
/tmp/dist-test-task8XuDZV/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765270668420570-20677-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765270668420570-20677-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765270668420570-20677-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765270668420570-20677-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.49.67:0
--local_ip_for_outbound_sockets=127.20.49.67
--webserver_interface=127.20.49.67
--webserver_port=0
--tserver_master_addrs=127.20.49.126:38079
--keytab_file=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765270668420570-20677-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.49.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.20.49.84:44993
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765270668420570-20677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765270668420570-20677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765270668420570-20677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251209 08:58:16.316614 24003 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251209 08:58:16.316834 24003 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251209 08:58:16.316905 24003 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251209 08:58:16.320214 24003 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251209 08:58:16.320281 24003 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251209 08:58:16.320385 24003 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.49.67
I20251209 08:58:16.324558 24003 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.49.84:44993
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765270668420570-20677-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765270668420570-20677-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765270668420570-20677-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.49.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.49.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765270668420570-20677-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.49.67
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.20.49.126:38079
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24003
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.49.67
--log_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765270668420570-20677-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 468897755969d686fb08386f42f7835685e7953a
build type DEBUG
built by None at 09 Dec 2025 08:43:15 UTC on 24a791456cd2
build id 9426
I20251209 08:58:16.325624 24003 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251209 08:58:16.326514 24003 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251209 08:58:16.332921 24009 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251209 08:58:16.332937 24011 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251209 08:58:16.332938 24008 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251209 08:58:16.333236 24003 server_base.cc:1047] running on GCE node
I20251209 08:58:16.333556 24003 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251209 08:58:16.334123 24003 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251209 08:58:16.335302 24003 hybrid_clock.cc:648] HybridClock initialized: now 1765270696335281 us; error 33 us; skew 500 ppm
Dec 09 08:58:16 dist-test-slave-8w6c krb5kdc[23619](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765270696, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.49.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251209 08:58:16.338111 24003 init.cc:377] Logged in from keytab as kudu/127.20.49.67@KRBTEST.COM (short username kudu)
I20251209 08:58:16.339197 24003 webserver.cc:492] Webserver started at http://127.20.49.67:46471/ using document root <none> and password file <none>
I20251209 08:58:16.339744 24003 fs_manager.cc:362] Metadata directory not provided
I20251209 08:58:16.339805 24003 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251209 08:58:16.339994 24003 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251209 08:58:16.341780 24003 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765270668420570-20677-0/minicluster-data/ts-2/data/instance:
uuid: "2798a9d0fdbc40e695990b16a4acd7d8"
format_stamp: "Formatted at 2025-12-09 08:58:16 on dist-test-slave-8w6c"
server_key: "4d563fe279ef87263d9cd4ecbaf41863"
server_key_iv: "5e0321c900e83734283a57ab61ccd32f"
server_key_version: "encryptionkey@0"
I20251209 08:58:16.342299 24003 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765270668420570-20677-0/minicluster-data/ts-2/wal/instance:
uuid: "2798a9d0fdbc40e695990b16a4acd7d8"
format_stamp: "Formatted at 2025-12-09 08:58:16 on dist-test-slave-8w6c"
server_key: "4d563fe279ef87263d9cd4ecbaf41863"
server_key_iv: "5e0321c900e83734283a57ab61ccd32f"
server_key_version: "encryptionkey@0"
I20251209 08:58:16.345729 24003 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20251209 08:58:16.348032 24018 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251209 08:58:16.349076 24003 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20251209 08:58:16.349201 24003 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765270668420570-20677-0/minicluster-data/ts-2/data,/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765270668420570-20677-0/minicluster-data/ts-2/wal
uuid: "2798a9d0fdbc40e695990b16a4acd7d8"
format_stamp: "Formatted at 2025-12-09 08:58:16 on dist-test-slave-8w6c"
server_key: "4d563fe279ef87263d9cd4ecbaf41863"
server_key_iv: "5e0321c900e83734283a57ab61ccd32f"
server_key_version: "encryptionkey@0"
I20251209 08:58:16.349306 24003 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765270668420570-20677-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765270668420570-20677-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765270668420570-20677-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251209 08:58:16.378005 24003 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251209 08:58:16.381269 24003 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251209 08:58:16.381458 24003 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251209 08:58:16.382086 24003 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251209 08:58:16.382951 24003 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251209 08:58:16.382999 24003 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251209 08:58:16.383059 24003 ts_tablet_manager.cc:616] Registered 0 tablets
I20251209 08:58:16.383086 24003 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251209 08:58:16.392006 24003 rpc_server.cc:307] RPC server started. Bound to: 127.20.49.67:33995
I20251209 08:58:16.392035 24131 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.49.67:33995 every 8 connection(s)
I20251209 08:58:16.392941 24003 server_base.cc:1179] Dumped server information to /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765270668420570-20677-0/minicluster-data/ts-2/data/info.pb
I20251209 08:58:16.395802 20677 external_mini_cluster.cc:1428] Started /tmp/dist-test-task8XuDZV/build/debug/bin/kudu as pid 24003
I20251209 08:58:16.395911 20677 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1765270668420570-20677-0/minicluster-data/ts-2/wal/instance
I20251209 08:58:16.396143 20677 external_mini_cluster.cc:1468] Setting key 677c15c853c5ad0c17b6fec690de3249
Dec 09 08:58:16 dist-test-slave-8w6c krb5kdc[23619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765270696, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.49.67@KRBTEST.COM for kudu/127.20.49.126@KRBTEST.COM
I20251209 08:58:16.405087 23706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:58:16.394752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.49.67:48723 (local address 127.20.49.126:38079)
1209 08:58:16.394921 (+ 169us) server_negotiation.cc:207] Beginning negotiation
1209 08:58:16.394926 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1209 08:58:16.395550 (+ 624us) server_negotiation.cc:408] Connection header received
1209 08:58:16.396512 (+ 962us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1209 08:58:16.396515 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1209 08:58:16.396581 (+ 66us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1209 08:58:16.396684 (+ 103us) server_negotiation.cc:227] Negotiated authn=SASL
1209 08:58:16.398730 (+ 2046us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:58:16.399232 (+ 502us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1209 08:58:16.399827 (+ 595us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:58:16.399976 (+ 149us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:58:16.402151 (+ 2175us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1209 08:58:16.402166 (+ 15us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1209 08:58:16.402169 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1209 08:58:16.402190 (+ 21us) server_negotiation.cc:923] calling sasl_server_start()
1209 08:58:16.403547 (+ 1357us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1209 08:58:16.403991 (+ 444us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1209 08:58:16.403993 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1209 08:58:16.403995 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1209 08:58:16.404034 (+ 39us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1209 08:58:16.404259 (+ 225us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1209 08:58:16.404261 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1209 08:58:16.404262 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1209 08:58:16.404395 (+ 133us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1209 08:58:16.404464 (+ 69us) server_negotiation.cc:1036] Waiting for connection context
1209 08:58:16.404864 (+ 400us) server_negotiation.cc:300] Negotiation successful
1209 08:58:16.404969 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20251209 08:58:16.405771 24134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:58:16.395011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.49.126:38079 (local address 127.20.49.67:48723)
1209 08:58:16.395413 (+ 402us) negotiation.cc:107] Waiting for socket to connect
1209 08:58:16.395450 (+ 37us) client_negotiation.cc:174] Beginning negotiation
1209 08:58:16.396286 (+ 836us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1209 08:58:16.396846 (+ 560us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1209 08:58:16.396856 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
1209 08:58:16.397406 (+ 550us) client_negotiation.cc:189] Negotiated authn=SASL
1209 08:58:16.398595 (+ 1189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:58:16.398610 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:58:16.399329 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1209 08:58:16.399332 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1209 08:58:16.399730 (+ 398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:58:16.399737 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:58:16.399915 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:58:16.400529 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1209 08:58:16.400552 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
1209 08:58:16.402018 (+ 1466us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1209 08:58:16.403656 (+ 1638us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1209 08:58:16.403661 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1209 08:58:16.403672 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
1209 08:58:16.403890 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1209 08:58:16.404101 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1209 08:58:16.404103 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1209 08:58:16.404104 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1209 08:58:16.404195 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1209 08:58:16.404464 (+ 269us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1209 08:58:16.404468 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1209 08:58:16.404677 (+ 209us) client_negotiation.cc:769] Sending connection context
1209 08:58:16.404840 (+ 163us) client_negotiation.cc:240] Negotiation successful
1209 08:58:16.405060 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":94,"threads_started":1}
I20251209 08:58:16.406807 24132 heartbeater.cc:344] Connected to a master server at 127.20.49.126:38079
I20251209 08:58:16.407053 24132 heartbeater.cc:461] Registering TS with master...
I20251209 08:58:16.407541 24132 heartbeater.cc:507] Master 127.20.49.126:38079 requested a full tablet report, sending...
I20251209 08:58:16.408485 23667 ts_manager.cc:194] Registered new tserver with Master: 2798a9d0fdbc40e695990b16a4acd7d8 (127.20.49.67:33995)
I20251209 08:58:16.409108 23667 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.49.67@KRBTEST.COM'} at 127.20.49.67:48723
I20251209 08:58:16.409956 20677 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
W20251209 08:58:16.557114 23706 negotiation.cc:337] Failed RPC negotiation. Trace:
1209 08:58:16.553460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60510 (local address 127.20.49.126:38079)
1209 08:58:16.553629 (+ 169us) server_negotiation.cc:207] Beginning negotiation
1209 08:58:16.553634 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1209 08:58:16.554658 (+ 1024us) server_negotiation.cc:408] Connection header received
1209 08:58:16.555510 (+ 852us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1209 08:58:16.555512 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1209 08:58:16.555561 (+ 49us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1209 08:58:16.555640 (+ 79us) server_negotiation.cc:227] Negotiated authn=SASL
1209 08:58:16.556965 (+ 1325us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60510: BlockingRecv error: recv got EOF from 127.0.0.1:60510 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
W20251209 08:58:16.558738 23706 negotiation.cc:337] Failed RPC negotiation. Trace:
1209 08:58:16.557902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60522 (local address 127.20.49.126:38079)
1209 08:58:16.558035 (+ 133us) server_negotiation.cc:207] Beginning negotiation
1209 08:58:16.558038 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1209 08:58:16.558052 (+ 14us) server_negotiation.cc:408] Connection header received
1209 08:58:16.558144 (+ 92us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1209 08:58:16.558147 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1209 08:58:16.558184 (+ 37us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1209 08:58:16.558252 (+ 68us) server_negotiation.cc:227] Negotiated authn=SASL
1209 08:58:16.558614 (+ 362us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60522: BlockingRecv error: recv got EOF from 127.0.0.1:60522 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
W20251209 08:58:16.559738 23706 negotiation.cc:337] Failed RPC negotiation. Trace:
1209 08:58:16.559099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60530 (local address 127.20.49.126:38079)
1209 08:58:16.559211 (+ 112us) server_negotiation.cc:207] Beginning negotiation
1209 08:58:16.559214 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1209 08:58:16.559224 (+ 10us) server_negotiation.cc:408] Connection header received
1209 08:58:16.559250 (+ 26us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1209 08:58:16.559253 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1209 08:58:16.559283 (+ 30us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1209 08:58:16.559351 (+ 68us) server_negotiation.cc:227] Negotiated authn=SASL
1209 08:58:16.559675 (+ 324us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60530: BlockingRecv error: recv got EOF from 127.0.0.1:60530 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
W20251209 08:58:16.563521 23706 negotiation.cc:337] Failed RPC negotiation. Trace:
1209 08:58:16.562813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60544 (local address 127.20.49.126:38079)
1209 08:58:16.562963 (+ 150us) server_negotiation.cc:207] Beginning negotiation
1209 08:58:16.562969 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1209 08:58:16.562985 (+ 16us) server_negotiation.cc:408] Connection header received
1209 08:58:16.563012 (+ 27us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1209 08:58:16.563014 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1209 08:58:16.563044 (+ 30us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1209 08:58:16.563109 (+ 65us) server_negotiation.cc:227] Negotiated authn=SASL
1209 08:58:16.563454 (+ 345us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60544: BlockingRecv error: recv got EOF from 127.0.0.1:60544 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
I20251209 08:58:16.571964 20677 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task8XuDZV/build/debug/bin/kudu with pid 23731
I20251209 08:58:16.577852 20677 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task8XuDZV/build/debug/bin/kudu with pid 23867
I20251209 08:58:16.583249 20677 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task8XuDZV/build/debug/bin/kudu with pid 24003
I20251209 08:58:16.588869 20677 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task8XuDZV/build/debug/bin/kudu with pid 23635
2025-12-09T08:58:16Z chronyd exiting
[ OK ] SecurityITest.TestRebalanceReportUnauthorized (3305 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1765270668420570-20677-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Dec 09 08:58:16 dist-test-slave-8w6c krb5kdc[24155](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Dec 09 08:58:16 dist-test-slave-8w6c krb5kdc[24155](info): set up 2 sockets
Dec 09 08:58:16 dist-test-slave-8w6c krb5kdc[24155](info): commencing operation
krb5kdc: starting...
W20251209 08:58:18.582386 20677 mini_kdc.cc:121] Time spent starting KDC: real 1.971s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1765270668420570-20677-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1765270668420570-20677-0/krb5kdc/test-user.keytab.
Dec 09 08:58:18 dist-test-slave-8w6c krb5kdc[24155](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765270698, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-12-09T08:58:18Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-12-09T08:58:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.49.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.49.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.49.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1765270668420570-20677-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.20.49.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1765270668420570-20677-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.20.49.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.20.49.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.20.49.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1765270668420570-20677-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.20.49.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1765270668420570-20677-0/krb5kdc/kudu.keytab.
I20251209 08:58:18.724375 20677 external_mini_cluster.cc:1366] Running /tmp/dist-test-task8XuDZV/build/debug/bin/kudu
/tmp/dist-test-task8XuDZV/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1765270668420570-20677-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1765270668420570-20677-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1765270668420570-20677-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1765270668420570-20677-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.49.126:33935
--webserver_interface=127.20.49.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1765270668420570-20677-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.49.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.20.49.84:37363
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.49.126:33935
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1765270668420570-20677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1765270668420570-20677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1765270668420570-20677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251209 08:58:18.833009 24171 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251209 08:58:18.833243 24171 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251209 08:58:18.833302 24171 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251209 08:58:18.836545 24171 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251209 08:58:18.836611 24171 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251209 08:58:18.836634 24171 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251209 08:58:18.836654 24171 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251209 08:58:18.836678 24171 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251209 08:58:18.840948 24171 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.49.84:37363
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1765270668420570-20677-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1765270668420570-20677-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.49.126:33935
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1765270668420570-20677-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.49.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.49.126:33935
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1765270668420570-20677-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.49.126
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24171
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1765270668420570-20677-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 468897755969d686fb08386f42f7835685e7953a
build type DEBUG
built by None at 09 Dec 2025 08:43:15 UTC on 24a791456cd2
build id 9426
I20251209 08:58:18.842049 24171 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251209 08:58:18.842886 24171 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251209 08:58:18.848430 24176 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251209 08:58:18.848470 24177 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251209 08:58:18.848613 24171 server_base.cc:1047] running on GCE node
W20251209 08:58:18.848429 24179 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251209 08:58:18.849224 24171 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251209 08:58:18.850167 24171 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251209 08:58:18.851317 24171 hybrid_clock.cc:648] HybridClock initialized: now 1765270698851294 us; error 40 us; skew 500 ppm
Dec 09 08:58:18 dist-test-slave-8w6c krb5kdc[24155](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765270698, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.49.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251209 08:58:18.853811 24171 init.cc:377] Logged in from keytab as kudu/127.20.49.126@KRBTEST.COM (short username kudu)
I20251209 08:58:18.854816 24171 webserver.cc:492] Webserver started at http://127.20.49.126:41197/ using document root <none> and password file <none>
I20251209 08:58:18.855317 24171 fs_manager.cc:362] Metadata directory not provided
I20251209 08:58:18.855362 24171 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251209 08:58:18.855518 24171 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251209 08:58:18.857156 24171 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1765270668420570-20677-0/minicluster-data/master-0/data/instance:
uuid: "54b7c6d45b484a56ba559b144ffac9d5"
format_stamp: "Formatted at 2025-12-09 08:58:18 on dist-test-slave-8w6c"
server_key: "86581946c9810e96e827d70d7678ee35"
server_key_iv: "de3b461b84cc9868fb9de687b4db63b2"
server_key_version: "encryptionkey@0"
I20251209 08:58:18.857592 24171 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1765270668420570-20677-0/minicluster-data/master-0/wal/instance:
uuid: "54b7c6d45b484a56ba559b144ffac9d5"
format_stamp: "Formatted at 2025-12-09 08:58:18 on dist-test-slave-8w6c"
server_key: "86581946c9810e96e827d70d7678ee35"
server_key_iv: "de3b461b84cc9868fb9de687b4db63b2"
server_key_version: "encryptionkey@0"
I20251209 08:58:18.861042 24171 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20251209 08:58:18.863317 24186 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251209 08:58:18.864352 24171 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20251209 08:58:18.864450 24171 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1765270668420570-20677-0/minicluster-data/master-0/data,/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1765270668420570-20677-0/minicluster-data/master-0/wal
uuid: "54b7c6d45b484a56ba559b144ffac9d5"
format_stamp: "Formatted at 2025-12-09 08:58:18 on dist-test-slave-8w6c"
server_key: "86581946c9810e96e827d70d7678ee35"
server_key_iv: "de3b461b84cc9868fb9de687b4db63b2"
server_key_version: "encryptionkey@0"
I20251209 08:58:18.864534 24171 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1765270668420570-20677-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1765270668420570-20677-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1765270668420570-20677-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251209 08:58:18.879632 24171 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251209 08:58:18.882601 24171 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251209 08:58:18.882768 24171 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251209 08:58:18.890093 24238 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.49.126:33935 every 8 connection(s)
I20251209 08:58:18.890091 24171 rpc_server.cc:307] RPC server started. Bound to: 127.20.49.126:33935
I20251209 08:58:18.891213 24171 server_base.cc:1179] Dumped server information to /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1765270668420570-20677-0/minicluster-data/master-0/data/info.pb
I20251209 08:58:18.894008 24239 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251209 08:58:18.899462 24239 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 54b7c6d45b484a56ba559b144ffac9d5: Bootstrap starting.
I20251209 08:58:18.900739 20677 external_mini_cluster.cc:1428] Started /tmp/dist-test-task8XuDZV/build/debug/bin/kudu as pid 24171
I20251209 08:58:18.900868 20677 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1765270668420570-20677-0/minicluster-data/master-0/wal/instance
I20251209 08:58:18.901142 20677 external_mini_cluster.cc:1468] Setting key ac72336ce3ab24bcc20dfd275c52c41f
I20251209 08:58:18.901998 24239 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 54b7c6d45b484a56ba559b144ffac9d5: Neither blocks nor log segments found. Creating new log.
I20251209 08:58:18.902804 24239 log.cc:826] T 00000000000000000000000000000000 P 54b7c6d45b484a56ba559b144ffac9d5: Log is configured to *not* fsync() on all Append() calls
I20251209 08:58:18.904893 24239 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 54b7c6d45b484a56ba559b144ffac9d5: No bootstrap required, opened a new log
I20251209 08:58:18.908118 24239 raft_consensus.cc:359] T 00000000000000000000000000000000 P 54b7c6d45b484a56ba559b144ffac9d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54b7c6d45b484a56ba559b144ffac9d5" member_type: VOTER last_known_addr { host: "127.20.49.126" port: 33935 } }
I20251209 08:58:18.908372 24239 raft_consensus.cc:385] T 00000000000000000000000000000000 P 54b7c6d45b484a56ba559b144ffac9d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251209 08:58:18.908442 24239 raft_consensus.cc:740] T 00000000000000000000000000000000 P 54b7c6d45b484a56ba559b144ffac9d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54b7c6d45b484a56ba559b144ffac9d5, State: Initialized, Role: FOLLOWER
Dec 09 08:58:18 dist-test-slave-8w6c krb5kdc[24155](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765270698, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.49.126@KRBTEST.COM
I20251209 08:58:18.908861 24239 consensus_queue.cc:260] T 00000000000000000000000000000000 P 54b7c6d45b484a56ba559b144ffac9d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54b7c6d45b484a56ba559b144ffac9d5" member_type: VOTER last_known_addr { host: "127.20.49.126" port: 33935 } }
I20251209 08:58:18.908985 24239 raft_consensus.cc:399] T 00000000000000000000000000000000 P 54b7c6d45b484a56ba559b144ffac9d5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251209 08:58:18.909054 24239 raft_consensus.cc:493] T 00000000000000000000000000000000 P 54b7c6d45b484a56ba559b144ffac9d5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251209 08:58:18.909149 24239 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 54b7c6d45b484a56ba559b144ffac9d5 [term 0 FOLLOWER]: Advancing to term 1
I20251209 08:58:18.910050 24239 raft_consensus.cc:515] T 00000000000000000000000000000000 P 54b7c6d45b484a56ba559b144ffac9d5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54b7c6d45b484a56ba559b144ffac9d5" member_type: VOTER last_known_addr { host: "127.20.49.126" port: 33935 } }
I20251209 08:58:18.910382 24239 leader_election.cc:304] T 00000000000000000000000000000000 P 54b7c6d45b484a56ba559b144ffac9d5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 54b7c6d45b484a56ba559b144ffac9d5; no voters:
I20251209 08:58:18.910661 24239 leader_election.cc:290] T 00000000000000000000000000000000 P 54b7c6d45b484a56ba559b144ffac9d5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20251209 08:58:18.910794 24244 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 54b7c6d45b484a56ba559b144ffac9d5 [term 1 FOLLOWER]: Leader election won for term 1
I20251209 08:58:18.911039 24244 raft_consensus.cc:697] T 00000000000000000000000000000000 P 54b7c6d45b484a56ba559b144ffac9d5 [term 1 LEADER]: Becoming Leader. State: Replica: 54b7c6d45b484a56ba559b144ffac9d5, State: Running, Role: LEADER
I20251209 08:58:18.911350 24244 consensus_queue.cc:237] T 00000000000000000000000000000000 P 54b7c6d45b484a56ba559b144ffac9d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54b7c6d45b484a56ba559b144ffac9d5" member_type: VOTER last_known_addr { host: "127.20.49.126" port: 33935 } }
I20251209 08:58:18.911994 24239 sys_catalog.cc:565] T 00000000000000000000000000000000 P 54b7c6d45b484a56ba559b144ffac9d5 [sys.catalog]: configured and running, proceeding with master startup.
I20251209 08:58:18.912993 24245 sys_catalog.cc:455] T 00000000000000000000000000000000 P 54b7c6d45b484a56ba559b144ffac9d5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "54b7c6d45b484a56ba559b144ffac9d5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54b7c6d45b484a56ba559b144ffac9d5" member_type: VOTER last_known_addr { host: "127.20.49.126" port: 33935 } } }
I20251209 08:58:18.913106 24245 sys_catalog.cc:458] T 00000000000000000000000000000000 P 54b7c6d45b484a56ba559b144ffac9d5 [sys.catalog]: This master's current role is: LEADER
I20251209 08:58:18.913030 24246 sys_catalog.cc:455] T 00000000000000000000000000000000 P 54b7c6d45b484a56ba559b144ffac9d5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 54b7c6d45b484a56ba559b144ffac9d5. Latest consensus state: current_term: 1 leader_uuid: "54b7c6d45b484a56ba559b144ffac9d5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54b7c6d45b484a56ba559b144ffac9d5" member_type: VOTER last_known_addr { host: "127.20.49.126" port: 33935 } } }
I20251209 08:58:18.913321 24246 sys_catalog.cc:458] T 00000000000000000000000000000000 P 54b7c6d45b484a56ba559b144ffac9d5 [sys.catalog]: This master's current role is: LEADER
I20251209 08:58:18.913599 24253 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251209 08:58:18.916090 24242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:58:18.902156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59980 (local address 127.20.49.126:33935)
1209 08:58:18.902550 (+ 394us) server_negotiation.cc:207] Beginning negotiation
1209 08:58:18.902560 (+ 10us) server_negotiation.cc:400] Waiting for connection header
1209 08:58:18.902597 (+ 37us) server_negotiation.cc:408] Connection header received
1209 08:58:18.903197 (+ 600us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1209 08:58:18.903217 (+ 20us) server_negotiation.cc:462] Received NEGOTIATE request from client
1209 08:58:18.903504 (+ 287us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1209 08:58:18.903784 (+ 280us) server_negotiation.cc:227] Negotiated authn=SASL
1209 08:58:18.904879 (+ 1095us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:58:18.905753 (+ 874us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1209 08:58:18.906637 (+ 884us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:58:18.906907 (+ 270us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:58:18.909540 (+ 2633us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1209 08:58:18.909560 (+ 20us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1209 08:58:18.909571 (+ 11us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1209 08:58:18.909604 (+ 33us) server_negotiation.cc:923] calling sasl_server_start()
1209 08:58:18.912500 (+ 2896us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1209 08:58:18.913121 (+ 621us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1209 08:58:18.913127 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1209 08:58:18.913132 (+ 5us) server_negotiation.cc:968] calling sasl_server_step()
1209 08:58:18.913229 (+ 97us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1209 08:58:18.913530 (+ 301us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1209 08:58:18.913533 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1209 08:58:18.913535 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1209 08:58:18.914051 (+ 516us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1209 08:58:18.914885 (+ 834us) server_negotiation.cc:1036] Waiting for connection context
1209 08:58:18.915066 (+ 181us) server_negotiation.cc:300] Negotiation successful
1209 08:58:18.915342 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":256,"thread_start_us":120,"threads_started":1}
I20251209 08:58:18.916923 24253 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251209 08:58:18.919124 20677 external_mini_cluster.cc:949] 0 TS(s) registered with all masters
I20251209 08:58:18.924043 24253 catalog_manager.cc:1357] Generated new cluster ID: 66a84e7d5d2b4e968b1b18e0cc956674
I20251209 08:58:18.924155 24253 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251209 08:58:18.936919 24253 catalog_manager.cc:1380] Generated new certificate authority record
I20251209 08:58:18.937577 24242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:58:18.932544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59986 (local address 127.20.49.126:33935)
1209 08:58:18.932685 (+ 141us) server_negotiation.cc:207] Beginning negotiation
1209 08:58:18.932690 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1209 08:58:18.933015 (+ 325us) server_negotiation.cc:408] Connection header received
1209 08:58:18.933155 (+ 140us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1209 08:58:18.933158 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1209 08:58:18.933213 (+ 55us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1209 08:58:18.933300 (+ 87us) server_negotiation.cc:227] Negotiated authn=SASL
1209 08:58:18.934154 (+ 854us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:58:18.934841 (+ 687us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1209 08:58:18.935756 (+ 915us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:58:18.935961 (+ 205us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:58:18.936497 (+ 536us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1209 08:58:18.936515 (+ 18us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1209 08:58:18.936518 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1209 08:58:18.937121 (+ 603us) server_negotiation.cc:923] calling sasl_server_start()
1209 08:58:18.937154 (+ 33us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1209 08:58:18.937173 (+ 19us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1209 08:58:18.937266 (+ 93us) server_negotiation.cc:1036] Waiting for connection context
1209 08:58:18.937392 (+ 126us) server_negotiation.cc:300] Negotiation successful
1209 08:58:18.937447 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20251209 08:58:18.938179 24253 catalog_manager.cc:1514] Loading token signing keys...
I20251209 08:58:18.949256 24253 catalog_manager.cc:6027] T 00000000000000000000000000000000 P 54b7c6d45b484a56ba559b144ffac9d5: Generated new TSK 0
I20251209 08:58:18.950068 24253 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251209 08:58:18.955597 20677 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task8XuDZV/build/debug/bin/kudu with pid 24171
2025-12-09T08:58:18Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (2364 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765270668420570-20677-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Dec 09 08:58:19 dist-test-slave-8w6c krb5kdc[24272](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Dec 09 08:58:19 dist-test-slave-8w6c krb5kdc[24272](info): set up 2 sockets
Dec 09 08:58:19 dist-test-slave-8w6c krb5kdc[24272](info): commencing operation
krb5kdc: starting...
W20251209 08:58:20.946414 20677 mini_kdc.cc:121] Time spent starting KDC: real 1.970s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765270668420570-20677-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765270668420570-20677-0/krb5kdc/test-user.keytab.
Dec 09 08:58:21 dist-test-slave-8w6c krb5kdc[24272](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765270701, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-12-09T08:58:21Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-12-09T08:58:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.49.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.49.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.49.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765270668420570-20677-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.20.49.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765270668420570-20677-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.20.49.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.20.49.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.20.49.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765270668420570-20677-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.20.49.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765270668420570-20677-0/krb5kdc/kudu.keytab.
I20251209 08:58:21.088467 20677 external_mini_cluster.cc:1366] Running /tmp/dist-test-task8XuDZV/build/debug/bin/kudu
/tmp/dist-test-task8XuDZV/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765270668420570-20677-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765270668420570-20677-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765270668420570-20677-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765270668420570-20677-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.49.126:46193
--webserver_interface=127.20.49.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765270668420570-20677-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.49.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.20.49.84:39715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.49.126:46193
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765270668420570-20677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765270668420570-20677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765270668420570-20677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251209 08:58:21.198189 24288 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251209 08:58:21.198475 24288 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251209 08:58:21.198575 24288 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251209 08:58:21.201907 24288 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251209 08:58:21.201983 24288 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251209 08:58:21.202023 24288 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251209 08:58:21.202068 24288 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251209 08:58:21.202111 24288 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251209 08:58:21.206626 24288 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.49.84:39715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765270668420570-20677-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765270668420570-20677-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.49.126:46193
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765270668420570-20677-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.49.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.49.126:46193
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765270668420570-20677-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.49.126
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24288
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765270668420570-20677-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 468897755969d686fb08386f42f7835685e7953a
build type DEBUG
built by None at 09 Dec 2025 08:43:15 UTC on 24a791456cd2
build id 9426
I20251209 08:58:21.207727 24288 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251209 08:58:21.208596 24288 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251209 08:58:21.214174 24293 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251209 08:58:21.214184 24296 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251209 08:58:21.214299 24288 server_base.cc:1047] running on GCE node
W20251209 08:58:21.214184 24294 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251209 08:58:21.214855 24288 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251209 08:58:21.215981 24288 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251209 08:58:21.217168 24288 hybrid_clock.cc:648] HybridClock initialized: now 1765270701217160 us; error 31 us; skew 500 ppm
Dec 09 08:58:21 dist-test-slave-8w6c krb5kdc[24272](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765270701, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.49.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251209 08:58:21.220193 24288 init.cc:377] Logged in from keytab as kudu/127.20.49.126@KRBTEST.COM (short username kudu)
I20251209 08:58:21.221360 24288 webserver.cc:492] Webserver started at http://127.20.49.126:39393/ using document root <none> and password file <none>
I20251209 08:58:21.222084 24288 fs_manager.cc:362] Metadata directory not provided
I20251209 08:58:21.222170 24288 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251209 08:58:21.222347 24288 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251209 08:58:21.224156 24288 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765270668420570-20677-0/minicluster-data/master-0/data/instance:
uuid: "92bf8e07223a4848baa19b91cfdb87cd"
format_stamp: "Formatted at 2025-12-09 08:58:21 on dist-test-slave-8w6c"
server_key: "5ffd6e67215a326f29b1798ef64c00dd"
server_key_iv: "aca7695b7f9b4d15f48f899bd3d3b087"
server_key_version: "encryptionkey@0"
I20251209 08:58:21.224614 24288 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765270668420570-20677-0/minicluster-data/master-0/wal/instance:
uuid: "92bf8e07223a4848baa19b91cfdb87cd"
format_stamp: "Formatted at 2025-12-09 08:58:21 on dist-test-slave-8w6c"
server_key: "5ffd6e67215a326f29b1798ef64c00dd"
server_key_iv: "aca7695b7f9b4d15f48f899bd3d3b087"
server_key_version: "encryptionkey@0"
I20251209 08:58:21.228101 24288 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20251209 08:58:21.230366 24303 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251209 08:58:21.231410 24288 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20251209 08:58:21.231515 24288 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765270668420570-20677-0/minicluster-data/master-0/data,/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765270668420570-20677-0/minicluster-data/master-0/wal
uuid: "92bf8e07223a4848baa19b91cfdb87cd"
format_stamp: "Formatted at 2025-12-09 08:58:21 on dist-test-slave-8w6c"
server_key: "5ffd6e67215a326f29b1798ef64c00dd"
server_key_iv: "aca7695b7f9b4d15f48f899bd3d3b087"
server_key_version: "encryptionkey@0"
I20251209 08:58:21.231598 24288 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765270668420570-20677-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765270668420570-20677-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765270668420570-20677-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251209 08:58:21.250819 24288 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251209 08:58:21.257416 24288 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251209 08:58:21.257596 24288 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251209 08:58:21.265097 24288 rpc_server.cc:307] RPC server started. Bound to: 127.20.49.126:46193
I20251209 08:58:21.265102 24355 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.49.126:46193 every 8 connection(s)
I20251209 08:58:21.266184 24288 server_base.cc:1179] Dumped server information to /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765270668420570-20677-0/minicluster-data/master-0/data/info.pb
I20251209 08:58:21.268963 24356 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251209 08:58:21.275007 24356 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 92bf8e07223a4848baa19b91cfdb87cd: Bootstrap starting.
I20251209 08:58:21.275005 20677 external_mini_cluster.cc:1428] Started /tmp/dist-test-task8XuDZV/build/debug/bin/kudu as pid 24288
I20251209 08:58:21.275161 20677 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765270668420570-20677-0/minicluster-data/master-0/wal/instance
I20251209 08:58:21.275388 20677 external_mini_cluster.cc:1468] Setting key 75d7444d0b701845039b53a4dc662af7
I20251209 08:58:21.277427 24356 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 92bf8e07223a4848baa19b91cfdb87cd: Neither blocks nor log segments found. Creating new log.
I20251209 08:58:21.278209 24356 log.cc:826] T 00000000000000000000000000000000 P 92bf8e07223a4848baa19b91cfdb87cd: Log is configured to *not* fsync() on all Append() calls
I20251209 08:58:21.279910 24356 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 92bf8e07223a4848baa19b91cfdb87cd: No bootstrap required, opened a new log
Dec 09 08:58:21 dist-test-slave-8w6c krb5kdc[24272](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765270701, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.49.126@KRBTEST.COM
I20251209 08:58:21.282732 24356 raft_consensus.cc:359] T 00000000000000000000000000000000 P 92bf8e07223a4848baa19b91cfdb87cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92bf8e07223a4848baa19b91cfdb87cd" member_type: VOTER last_known_addr { host: "127.20.49.126" port: 46193 } }
I20251209 08:58:21.282919 24356 raft_consensus.cc:385] T 00000000000000000000000000000000 P 92bf8e07223a4848baa19b91cfdb87cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251209 08:58:21.282974 24356 raft_consensus.cc:740] T 00000000000000000000000000000000 P 92bf8e07223a4848baa19b91cfdb87cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92bf8e07223a4848baa19b91cfdb87cd, State: Initialized, Role: FOLLOWER
I20251209 08:58:21.283425 24356 consensus_queue.cc:260] T 00000000000000000000000000000000 P 92bf8e07223a4848baa19b91cfdb87cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92bf8e07223a4848baa19b91cfdb87cd" member_type: VOTER last_known_addr { host: "127.20.49.126" port: 46193 } }
I20251209 08:58:21.283587 24356 raft_consensus.cc:399] T 00000000000000000000000000000000 P 92bf8e07223a4848baa19b91cfdb87cd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251209 08:58:21.283668 24356 raft_consensus.cc:493] T 00000000000000000000000000000000 P 92bf8e07223a4848baa19b91cfdb87cd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251209 08:58:21.283771 24356 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 92bf8e07223a4848baa19b91cfdb87cd [term 0 FOLLOWER]: Advancing to term 1
I20251209 08:58:21.284869 24356 raft_consensus.cc:515] T 00000000000000000000000000000000 P 92bf8e07223a4848baa19b91cfdb87cd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92bf8e07223a4848baa19b91cfdb87cd" member_type: VOTER last_known_addr { host: "127.20.49.126" port: 46193 } }
I20251209 08:58:21.285275 24356 leader_election.cc:304] T 00000000000000000000000000000000 P 92bf8e07223a4848baa19b91cfdb87cd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 92bf8e07223a4848baa19b91cfdb87cd; no voters:
I20251209 08:58:21.285624 24356 leader_election.cc:290] T 00000000000000000000000000000000 P 92bf8e07223a4848baa19b91cfdb87cd [CANDIDATE]: Term 1 election: Requested vote from peers
I20251209 08:58:21.286051 24361 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 92bf8e07223a4848baa19b91cfdb87cd [term 1 FOLLOWER]: Leader election won for term 1
I20251209 08:58:21.286266 24361 raft_consensus.cc:697] T 00000000000000000000000000000000 P 92bf8e07223a4848baa19b91cfdb87cd [term 1 LEADER]: Becoming Leader. State: Replica: 92bf8e07223a4848baa19b91cfdb87cd, State: Running, Role: LEADER
I20251209 08:58:21.286577 24361 consensus_queue.cc:237] T 00000000000000000000000000000000 P 92bf8e07223a4848baa19b91cfdb87cd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92bf8e07223a4848baa19b91cfdb87cd" member_type: VOTER last_known_addr { host: "127.20.49.126" port: 46193 } }
I20251209 08:58:21.286935 24356 sys_catalog.cc:565] T 00000000000000000000000000000000 P 92bf8e07223a4848baa19b91cfdb87cd [sys.catalog]: configured and running, proceeding with master startup.
I20251209 08:58:21.287775 24361 sys_catalog.cc:455] T 00000000000000000000000000000000 P 92bf8e07223a4848baa19b91cfdb87cd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 92bf8e07223a4848baa19b91cfdb87cd. Latest consensus state: current_term: 1 leader_uuid: "92bf8e07223a4848baa19b91cfdb87cd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92bf8e07223a4848baa19b91cfdb87cd" member_type: VOTER last_known_addr { host: "127.20.49.126" port: 46193 } } }
I20251209 08:58:21.287897 24361 sys_catalog.cc:458] T 00000000000000000000000000000000 P 92bf8e07223a4848baa19b91cfdb87cd [sys.catalog]: This master's current role is: LEADER
I20251209 08:58:21.288173 24362 sys_catalog.cc:455] T 00000000000000000000000000000000 P 92bf8e07223a4848baa19b91cfdb87cd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "92bf8e07223a4848baa19b91cfdb87cd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92bf8e07223a4848baa19b91cfdb87cd" member_type: VOTER last_known_addr { host: "127.20.49.126" port: 46193 } } }
I20251209 08:58:21.288282 24362 sys_catalog.cc:458] T 00000000000000000000000000000000 P 92bf8e07223a4848baa19b91cfdb87cd [sys.catalog]: This master's current role is: LEADER
I20251209 08:58:21.289144 24359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:58:21.276544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53194 (local address 127.20.49.126:46193)
1209 08:58:21.276982 (+ 438us) server_negotiation.cc:207] Beginning negotiation
1209 08:58:21.276992 (+ 10us) server_negotiation.cc:400] Waiting for connection header
1209 08:58:21.277029 (+ 37us) server_negotiation.cc:408] Connection header received
1209 08:58:21.277573 (+ 544us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1209 08:58:21.277600 (+ 27us) server_negotiation.cc:462] Received NEGOTIATE request from client
1209 08:58:21.277976 (+ 376us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1209 08:58:21.278287 (+ 311us) server_negotiation.cc:227] Negotiated authn=SASL
1209 08:58:21.279201 (+ 914us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:58:21.279942 (+ 741us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1209 08:58:21.280600 (+ 658us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:58:21.280912 (+ 312us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:58:21.283335 (+ 2423us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1209 08:58:21.283357 (+ 22us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1209 08:58:21.283369 (+ 12us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1209 08:58:21.283409 (+ 40us) server_negotiation.cc:923] calling sasl_server_start()
1209 08:58:21.285628 (+ 2219us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1209 08:58:21.286265 (+ 637us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1209 08:58:21.286270 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1209 08:58:21.286278 (+ 8us) server_negotiation.cc:968] calling sasl_server_step()
1209 08:58:21.286379 (+ 101us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1209 08:58:21.286664 (+ 285us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1209 08:58:21.286667 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1209 08:58:21.286669 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1209 08:58:21.287143 (+ 474us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1209 08:58:21.287373 (+ 230us) server_negotiation.cc:1036] Waiting for connection context
1209 08:58:21.288189 (+ 816us) server_negotiation.cc:300] Negotiation successful
1209 08:58:21.288458 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":303,"thread_start_us":139,"threads_started":1}
W20251209 08:58:21.292358 24376 catalog_manager.cc:1568] T 00000000000000000000000000000000 P 92bf8e07223a4848baa19b91cfdb87cd: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20251209 08:58:21.292446 24376 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20251209 08:58:21.292510 24368 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251209 08:58:21.293431 24368 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251209 08:58:21.299142 24368 catalog_manager.cc:1357] Generated new cluster ID: 0c3e4861733646588e80abc8e005ba91
I20251209 08:58:21.299228 24368 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251209 08:58:21.327991 24368 catalog_manager.cc:1380] Generated new certificate authority record
I20251209 08:58:21.329048 24368 catalog_manager.cc:1514] Loading token signing keys...
I20251209 08:58:21.338632 24368 catalog_manager.cc:6027] T 00000000000000000000000000000000 P 92bf8e07223a4848baa19b91cfdb87cd: Generated new TSK 0
I20251209 08:58:21.339380 24368 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.20.49.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.49.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.49.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765270668420570-20677-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.20.49.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765270668420570-20677-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.20.49.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.20.49.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.20.49.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765270668420570-20677-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.20.49.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765270668420570-20677-0/krb5kdc/kudu.keytab.
I20251209 08:58:21.404488 20677 external_mini_cluster.cc:1366] Running /tmp/dist-test-task8XuDZV/build/debug/bin/kudu
/tmp/dist-test-task8XuDZV/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765270668420570-20677-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765270668420570-20677-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765270668420570-20677-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765270668420570-20677-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.49.65:0
--local_ip_for_outbound_sockets=127.20.49.65
--webserver_interface=127.20.49.65
--webserver_port=0
--tserver_master_addrs=127.20.49.126:46193
--keytab_file=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765270668420570-20677-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.49.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.20.49.84:39715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765270668420570-20677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765270668420570-20677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765270668420570-20677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251209 08:58:21.538379 24384 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251209 08:58:21.538654 24384 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251209 08:58:21.538726 24384 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251209 08:58:21.542209 24384 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251209 08:58:21.542289 24384 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251209 08:58:21.542382 24384 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.49.65
I20251209 08:58:21.546849 24384 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.49.84:39715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765270668420570-20677-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765270668420570-20677-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765270668420570-20677-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.49.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.49.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765270668420570-20677-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.49.65
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.20.49.126:46193
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24384
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.49.65
--log_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765270668420570-20677-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 468897755969d686fb08386f42f7835685e7953a
build type DEBUG
built by None at 09 Dec 2025 08:43:15 UTC on 24a791456cd2
build id 9426
I20251209 08:58:21.547992 24384 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251209 08:58:21.548909 24384 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251209 08:58:21.556048 24390 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251209 08:58:21.556151 24384 server_base.cc:1047] running on GCE node
W20251209 08:58:21.556039 24389 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251209 08:58:21.556039 24392 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251209 08:58:21.557816 24384 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251209 08:58:21.558413 24384 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251209 08:58:21.559609 24384 hybrid_clock.cc:648] HybridClock initialized: now 1765270701559596 us; error 28 us; skew 500 ppm
Dec 09 08:58:21 dist-test-slave-8w6c krb5kdc[24272](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765270701, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.49.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251209 08:58:21.562505 24384 init.cc:377] Logged in from keytab as kudu/127.20.49.65@KRBTEST.COM (short username kudu)
I20251209 08:58:21.563632 24384 webserver.cc:492] Webserver started at http://127.20.49.65:43275/ using document root <none> and password file <none>
I20251209 08:58:21.564200 24384 fs_manager.cc:362] Metadata directory not provided
I20251209 08:58:21.564256 24384 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251209 08:58:21.564442 24384 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251209 08:58:21.566203 24384 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765270668420570-20677-0/minicluster-data/ts-0/data/instance:
uuid: "5c3e8778ceeb4574b5c7ff1d50006306"
format_stamp: "Formatted at 2025-12-09 08:58:21 on dist-test-slave-8w6c"
server_key: "1fc3d8d3b9a61ebf14cbd7c91f101c5b"
server_key_iv: "3a392b4e80e8fb00415295be61e6af60"
server_key_version: "encryptionkey@0"
I20251209 08:58:21.566668 24384 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765270668420570-20677-0/minicluster-data/ts-0/wal/instance:
uuid: "5c3e8778ceeb4574b5c7ff1d50006306"
format_stamp: "Formatted at 2025-12-09 08:58:21 on dist-test-slave-8w6c"
server_key: "1fc3d8d3b9a61ebf14cbd7c91f101c5b"
server_key_iv: "3a392b4e80e8fb00415295be61e6af60"
server_key_version: "encryptionkey@0"
I20251209 08:58:21.570179 24384 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20251209 08:58:21.572386 24399 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251209 08:58:21.573348 24384 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20251209 08:58:21.573455 24384 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765270668420570-20677-0/minicluster-data/ts-0/data,/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765270668420570-20677-0/minicluster-data/ts-0/wal
uuid: "5c3e8778ceeb4574b5c7ff1d50006306"
format_stamp: "Formatted at 2025-12-09 08:58:21 on dist-test-slave-8w6c"
server_key: "1fc3d8d3b9a61ebf14cbd7c91f101c5b"
server_key_iv: "3a392b4e80e8fb00415295be61e6af60"
server_key_version: "encryptionkey@0"
I20251209 08:58:21.573544 24384 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765270668420570-20677-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765270668420570-20677-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765270668420570-20677-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251209 08:58:21.599694 24384 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251209 08:58:21.603051 24384 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251209 08:58:21.603258 24384 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251209 08:58:21.603850 24384 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251209 08:58:21.604799 24384 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251209 08:58:21.604854 24384 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251209 08:58:21.604916 24384 ts_tablet_manager.cc:616] Registered 0 tablets
I20251209 08:58:21.604944 24384 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251209 08:58:21.615154 24384 rpc_server.cc:307] RPC server started. Bound to: 127.20.49.65:37209
I20251209 08:58:21.615208 24512 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.49.65:37209 every 8 connection(s)
I20251209 08:58:21.616580 24384 server_base.cc:1179] Dumped server information to /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765270668420570-20677-0/minicluster-data/ts-0/data/info.pb
I20251209 08:58:21.622601 20677 external_mini_cluster.cc:1428] Started /tmp/dist-test-task8XuDZV/build/debug/bin/kudu as pid 24384
I20251209 08:58:21.622743 20677 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765270668420570-20677-0/minicluster-data/ts-0/wal/instance
I20251209 08:58:21.623020 20677 external_mini_cluster.cc:1468] Setting key 35e9f2f9938c34953ee1fde3353a3671
Dec 09 08:58:21 dist-test-slave-8w6c krb5kdc[24272](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765270701, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.49.65@KRBTEST.COM for kudu/127.20.49.126@KRBTEST.COM
I20251209 08:58:21.632346 24359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:58:21.618947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.49.65:51351 (local address 127.20.49.126:46193)
1209 08:58:21.619499 (+ 552us) server_negotiation.cc:207] Beginning negotiation
1209 08:58:21.619506 (+ 7us) server_negotiation.cc:400] Waiting for connection header
1209 08:58:21.620816 (+ 1310us) server_negotiation.cc:408] Connection header received
1209 08:58:21.622075 (+ 1259us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1209 08:58:21.622078 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1209 08:58:21.622147 (+ 69us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1209 08:58:21.622221 (+ 74us) server_negotiation.cc:227] Negotiated authn=SASL
1209 08:58:21.624586 (+ 2365us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:58:21.625353 (+ 767us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1209 08:58:21.626219 (+ 866us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:58:21.626403 (+ 184us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:58:21.628692 (+ 2289us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1209 08:58:21.628713 (+ 21us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1209 08:58:21.628716 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1209 08:58:21.628751 (+ 35us) server_negotiation.cc:923] calling sasl_server_start()
1209 08:58:21.630349 (+ 1598us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1209 08:58:21.630862 (+ 513us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1209 08:58:21.630866 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1209 08:58:21.630867 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1209 08:58:21.630913 (+ 46us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1209 08:58:21.631214 (+ 301us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1209 08:58:21.631218 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1209 08:58:21.631220 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1209 08:58:21.631381 (+ 161us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1209 08:58:21.631472 (+ 91us) server_negotiation.cc:1036] Waiting for connection context
1209 08:58:21.631996 (+ 524us) server_negotiation.cc:300] Negotiation successful
1209 08:58:21.632168 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":421}
I20251209 08:58:21.633026 24515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:58:21.619296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.49.126:46193 (local address 127.20.49.65:51351)
1209 08:58:21.620559 (+ 1263us) negotiation.cc:107] Waiting for socket to connect
1209 08:58:21.620620 (+ 61us) client_negotiation.cc:174] Beginning negotiation
1209 08:58:21.621749 (+ 1129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1209 08:58:21.622568 (+ 819us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1209 08:58:21.622580 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
1209 08:58:21.623205 (+ 625us) client_negotiation.cc:189] Negotiated authn=SASL
1209 08:58:21.624429 (+ 1224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:58:21.624445 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:58:21.625525 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1209 08:58:21.625529 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1209 08:58:21.626092 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:58:21.626102 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:58:21.626341 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:58:21.627008 (+ 667us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1209 08:58:21.627036 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
1209 08:58:21.628527 (+ 1491us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1209 08:58:21.630481 (+ 1954us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1209 08:58:21.630486 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1209 08:58:21.630500 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
1209 08:58:21.630754 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1209 08:58:21.631000 (+ 246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1209 08:58:21.631005 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1209 08:58:21.631007 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1209 08:58:21.631122 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1209 08:58:21.631488 (+ 366us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1209 08:58:21.631497 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1209 08:58:21.631750 (+ 253us) client_negotiation.cc:769] Sending connection context
1209 08:58:21.631954 (+ 204us) client_negotiation.cc:240] Negotiation successful
1209 08:58:21.632264 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1020,"thread_start_us":125,"threads_started":1}
WARNING: no policy specified for kudu/127.20.49.66@KRBTEST.COM; defaulting to no policy
I20251209 08:58:21.634430 24513 heartbeater.cc:344] Connected to a master server at 127.20.49.126:46193
I20251209 08:58:21.634691 24513 heartbeater.cc:461] Registering TS with master...
I20251209 08:58:21.635272 24513 heartbeater.cc:507] Master 127.20.49.126:46193 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.49.66@KRBTEST.COM" created.
I20251209 08:58:21.636991 24320 ts_manager.cc:194] Registered new tserver with Master: 5c3e8778ceeb4574b5c7ff1d50006306 (127.20.49.65:37209)
I20251209 08:58:21.638244 24320 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.49.65@KRBTEST.COM'} at 127.20.49.65:51351
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.49.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765270668420570-20677-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.20.49.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765270668420570-20677-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.20.49.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.20.49.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.20.49.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765270668420570-20677-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.20.49.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765270668420570-20677-0/krb5kdc/kudu.keytab.
I20251209 08:58:21.677052 20677 external_mini_cluster.cc:1366] Running /tmp/dist-test-task8XuDZV/build/debug/bin/kudu
/tmp/dist-test-task8XuDZV/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765270668420570-20677-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765270668420570-20677-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765270668420570-20677-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765270668420570-20677-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.49.66:0
--local_ip_for_outbound_sockets=127.20.49.66
--webserver_interface=127.20.49.66
--webserver_port=0
--tserver_master_addrs=127.20.49.126:46193
--keytab_file=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765270668420570-20677-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.49.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.20.49.84:39715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765270668420570-20677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765270668420570-20677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765270668420570-20677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251209 08:58:21.790268 24520 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251209 08:58:21.790522 24520 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251209 08:58:21.790608 24520 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251209 08:58:21.794059 24520 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251209 08:58:21.794143 24520 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251209 08:58:21.794253 24520 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.49.66
I20251209 08:58:21.798695 24520 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.49.84:39715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765270668420570-20677-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765270668420570-20677-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765270668420570-20677-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.49.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.49.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765270668420570-20677-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.49.66
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.20.49.126:46193
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24520
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.49.66
--log_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765270668420570-20677-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 468897755969d686fb08386f42f7835685e7953a
build type DEBUG
built by None at 09 Dec 2025 08:43:15 UTC on 24a791456cd2
build id 9426
I20251209 08:58:21.799822 24520 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251209 08:58:21.800704 24520 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251209 08:58:21.807581 24525 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251209 08:58:21.807607 24528 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251209 08:58:21.807621 24526 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251209 08:58:21.807945 24520 server_base.cc:1047] running on GCE node
I20251209 08:58:21.808290 24520 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251209 08:58:21.808869 24520 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251209 08:58:21.810022 24520 hybrid_clock.cc:648] HybridClock initialized: now 1765270701809997 us; error 42 us; skew 500 ppm
Dec 09 08:58:21 dist-test-slave-8w6c krb5kdc[24272](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765270701, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.49.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251209 08:58:21.812891 24520 init.cc:377] Logged in from keytab as kudu/127.20.49.66@KRBTEST.COM (short username kudu)
I20251209 08:58:21.814123 24520 webserver.cc:492] Webserver started at http://127.20.49.66:40809/ using document root <none> and password file <none>
I20251209 08:58:21.814687 24520 fs_manager.cc:362] Metadata directory not provided
I20251209 08:58:21.814747 24520 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251209 08:58:21.814934 24520 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251209 08:58:21.816696 24520 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765270668420570-20677-0/minicluster-data/ts-1/data/instance:
uuid: "513ed56276a84382ae6d0b7177bc7380"
format_stamp: "Formatted at 2025-12-09 08:58:21 on dist-test-slave-8w6c"
server_key: "19f8412b70cf01b2388e18e84401c168"
server_key_iv: "c76a9db340cd47afa22a62d94e9a5a23"
server_key_version: "encryptionkey@0"
I20251209 08:58:21.817193 24520 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765270668420570-20677-0/minicluster-data/ts-1/wal/instance:
uuid: "513ed56276a84382ae6d0b7177bc7380"
format_stamp: "Formatted at 2025-12-09 08:58:21 on dist-test-slave-8w6c"
server_key: "19f8412b70cf01b2388e18e84401c168"
server_key_iv: "c76a9db340cd47afa22a62d94e9a5a23"
server_key_version: "encryptionkey@0"
I20251209 08:58:21.820873 24520 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20251209 08:58:21.823347 24535 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251209 08:58:21.824429 24520 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20251209 08:58:21.824537 24520 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765270668420570-20677-0/minicluster-data/ts-1/data,/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765270668420570-20677-0/minicluster-data/ts-1/wal
uuid: "513ed56276a84382ae6d0b7177bc7380"
format_stamp: "Formatted at 2025-12-09 08:58:21 on dist-test-slave-8w6c"
server_key: "19f8412b70cf01b2388e18e84401c168"
server_key_iv: "c76a9db340cd47afa22a62d94e9a5a23"
server_key_version: "encryptionkey@0"
I20251209 08:58:21.824672 24520 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765270668420570-20677-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765270668420570-20677-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765270668420570-20677-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251209 08:58:21.846392 24520 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251209 08:58:21.849558 24520 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251209 08:58:21.849794 24520 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251209 08:58:21.850415 24520 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251209 08:58:21.851329 24520 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251209 08:58:21.851377 24520 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251209 08:58:21.851418 24520 ts_tablet_manager.cc:616] Registered 0 tablets
I20251209 08:58:21.851434 24520 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251209 08:58:21.860699 24520 rpc_server.cc:307] RPC server started. Bound to: 127.20.49.66:42293
I20251209 08:58:21.860733 24648 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.49.66:42293 every 8 connection(s)
I20251209 08:58:21.861698 24520 server_base.cc:1179] Dumped server information to /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765270668420570-20677-0/minicluster-data/ts-1/data/info.pb
I20251209 08:58:21.864060 20677 external_mini_cluster.cc:1428] Started /tmp/dist-test-task8XuDZV/build/debug/bin/kudu as pid 24520
I20251209 08:58:21.864176 20677 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765270668420570-20677-0/minicluster-data/ts-1/wal/instance
I20251209 08:58:21.864444 20677 external_mini_cluster.cc:1468] Setting key 33d26b015ae52b9812a432c26e2beb42
Dec 09 08:58:21 dist-test-slave-8w6c krb5kdc[24272](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765270701, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.49.66@KRBTEST.COM for kudu/127.20.49.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.20.49.67@KRBTEST.COM; defaulting to no policy
I20251209 08:58:21.874740 24359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:58:21.863476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.49.66:55409 (local address 127.20.49.126:46193)
1209 08:58:21.863619 (+ 143us) server_negotiation.cc:207] Beginning negotiation
1209 08:58:21.863623 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1209 08:58:21.864345 (+ 722us) server_negotiation.cc:408] Connection header received
1209 08:58:21.865203 (+ 858us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1209 08:58:21.865206 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1209 08:58:21.865263 (+ 57us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1209 08:58:21.865345 (+ 82us) server_negotiation.cc:227] Negotiated authn=SASL
1209 08:58:21.867100 (+ 1755us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:58:21.867591 (+ 491us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1209 08:58:21.868240 (+ 649us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:58:21.868421 (+ 181us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:58:21.870837 (+ 2416us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1209 08:58:21.870857 (+ 20us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1209 08:58:21.870859 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1209 08:58:21.870887 (+ 28us) server_negotiation.cc:923] calling sasl_server_start()
1209 08:58:21.872541 (+ 1654us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1209 08:58:21.873116 (+ 575us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1209 08:58:21.873119 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1209 08:58:21.873120 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1209 08:58:21.873167 (+ 47us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1209 08:58:21.873462 (+ 295us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1209 08:58:21.873466 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1209 08:58:21.873468 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1209 08:58:21.873637 (+ 169us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1209 08:58:21.873787 (+ 150us) server_negotiation.cc:1036] Waiting for connection context
1209 08:58:21.874393 (+ 606us) server_negotiation.cc:300] Negotiation successful
1209 08:58:21.874571 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20251209 08:58:21.875327 24651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:58:21.863726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.49.126:46193 (local address 127.20.49.66:55409)
1209 08:58:21.864181 (+ 455us) negotiation.cc:107] Waiting for socket to connect
1209 08:58:21.864228 (+ 47us) client_negotiation.cc:174] Beginning negotiation
1209 08:58:21.865000 (+ 772us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1209 08:58:21.865510 (+ 510us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1209 08:58:21.865518 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
1209 08:58:21.865965 (+ 447us) client_negotiation.cc:189] Negotiated authn=SASL
1209 08:58:21.866938 (+ 973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:58:21.866953 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:58:21.867719 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1209 08:58:21.867723 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1209 08:58:21.868117 (+ 394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:58:21.868123 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:58:21.868343 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:58:21.869077 (+ 734us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1209 08:58:21.869103 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
1209 08:58:21.870659 (+ 1556us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1209 08:58:21.872682 (+ 2023us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1209 08:58:21.872688 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1209 08:58:21.872702 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
1209 08:58:21.873006 (+ 304us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1209 08:58:21.873258 (+ 252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1209 08:58:21.873262 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1209 08:58:21.873263 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1209 08:58:21.873372 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1209 08:58:21.873806 (+ 434us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1209 08:58:21.873817 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1209 08:58:21.874110 (+ 293us) client_negotiation.cc:769] Sending connection context
1209 08:58:21.874314 (+ 204us) client_negotiation.cc:240] Negotiation successful
1209 08:58:21.874578 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":109,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.49.67@KRBTEST.COM" created.
I20251209 08:58:21.876524 24649 heartbeater.cc:344] Connected to a master server at 127.20.49.126:46193
I20251209 08:58:21.876788 24649 heartbeater.cc:461] Registering TS with master...
I20251209 08:58:21.877331 24649 heartbeater.cc:507] Master 127.20.49.126:46193 requested a full tablet report, sending...
I20251209 08:58:21.878434 24320 ts_manager.cc:194] Registered new tserver with Master: 513ed56276a84382ae6d0b7177bc7380 (127.20.49.66:42293)
I20251209 08:58:21.879009 24320 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.49.66@KRBTEST.COM'} at 127.20.49.66:55409
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.49.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765270668420570-20677-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.20.49.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765270668420570-20677-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.20.49.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.20.49.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.20.49.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765270668420570-20677-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.20.49.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765270668420570-20677-0/krb5kdc/kudu.keytab.
I20251209 08:58:21.917491 20677 external_mini_cluster.cc:1366] Running /tmp/dist-test-task8XuDZV/build/debug/bin/kudu
/tmp/dist-test-task8XuDZV/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765270668420570-20677-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765270668420570-20677-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765270668420570-20677-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765270668420570-20677-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.49.67:0
--local_ip_for_outbound_sockets=127.20.49.67
--webserver_interface=127.20.49.67
--webserver_port=0
--tserver_master_addrs=127.20.49.126:46193
--keytab_file=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765270668420570-20677-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.49.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.20.49.84:39715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765270668420570-20677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765270668420570-20677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765270668420570-20677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251209 08:58:22.026737 24656 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251209 08:58:22.027009 24656 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251209 08:58:22.027104 24656 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251209 08:58:22.030592 24656 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251209 08:58:22.030673 24656 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251209 08:58:22.030776 24656 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.49.67
I20251209 08:58:22.035091 24656 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.49.84:39715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765270668420570-20677-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765270668420570-20677-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765270668420570-20677-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.49.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.49.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765270668420570-20677-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.49.67
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.20.49.126:46193
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24656
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.49.67
--log_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765270668420570-20677-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 468897755969d686fb08386f42f7835685e7953a
build type DEBUG
built by None at 09 Dec 2025 08:43:15 UTC on 24a791456cd2
build id 9426
I20251209 08:58:22.036196 24656 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251209 08:58:22.037060 24656 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251209 08:58:22.043663 24664 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251209 08:58:22.043691 24662 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251209 08:58:22.043664 24661 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251209 08:58:22.044013 24656 server_base.cc:1047] running on GCE node
I20251209 08:58:22.044415 24656 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251209 08:58:22.045003 24656 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251209 08:58:22.046169 24656 hybrid_clock.cc:648] HybridClock initialized: now 1765270702046159 us; error 27 us; skew 500 ppm
Dec 09 08:58:22 dist-test-slave-8w6c krb5kdc[24272](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765270702, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.49.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251209 08:58:22.049022 24656 init.cc:377] Logged in from keytab as kudu/127.20.49.67@KRBTEST.COM (short username kudu)
I20251209 08:58:22.050202 24656 webserver.cc:492] Webserver started at http://127.20.49.67:35599/ using document root <none> and password file <none>
I20251209 08:58:22.050781 24656 fs_manager.cc:362] Metadata directory not provided
I20251209 08:58:22.050839 24656 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251209 08:58:22.051033 24656 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251209 08:58:22.052793 24656 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765270668420570-20677-0/minicluster-data/ts-2/data/instance:
uuid: "a565149237674fd2a9538699ded75641"
format_stamp: "Formatted at 2025-12-09 08:58:22 on dist-test-slave-8w6c"
server_key: "b6159cc34d67e6f1f13f6556fd9dfafa"
server_key_iv: "16ce0628b3064edf909d096fc41662ca"
server_key_version: "encryptionkey@0"
I20251209 08:58:22.053329 24656 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765270668420570-20677-0/minicluster-data/ts-2/wal/instance:
uuid: "a565149237674fd2a9538699ded75641"
format_stamp: "Formatted at 2025-12-09 08:58:22 on dist-test-slave-8w6c"
server_key: "b6159cc34d67e6f1f13f6556fd9dfafa"
server_key_iv: "16ce0628b3064edf909d096fc41662ca"
server_key_version: "encryptionkey@0"
I20251209 08:58:22.056867 24656 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20251209 08:58:22.059304 24671 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251209 08:58:22.060341 24656 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20251209 08:58:22.060449 24656 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765270668420570-20677-0/minicluster-data/ts-2/data,/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765270668420570-20677-0/minicluster-data/ts-2/wal
uuid: "a565149237674fd2a9538699ded75641"
format_stamp: "Formatted at 2025-12-09 08:58:22 on dist-test-slave-8w6c"
server_key: "b6159cc34d67e6f1f13f6556fd9dfafa"
server_key_iv: "16ce0628b3064edf909d096fc41662ca"
server_key_version: "encryptionkey@0"
I20251209 08:58:22.060559 24656 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765270668420570-20677-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765270668420570-20677-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765270668420570-20677-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251209 08:58:22.082971 24656 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251209 08:58:22.086246 24656 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251209 08:58:22.086441 24656 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251209 08:58:22.087026 24656 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251209 08:58:22.087926 24656 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251209 08:58:22.087977 24656 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251209 08:58:22.088024 24656 ts_tablet_manager.cc:616] Registered 0 tablets
I20251209 08:58:22.088055 24656 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251209 08:58:22.097646 24656 rpc_server.cc:307] RPC server started. Bound to: 127.20.49.67:33653
I20251209 08:58:22.097721 24784 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.49.67:33653 every 8 connection(s)
I20251209 08:58:22.098690 24656 server_base.cc:1179] Dumped server information to /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765270668420570-20677-0/minicluster-data/ts-2/data/info.pb
I20251209 08:58:22.104076 20677 external_mini_cluster.cc:1428] Started /tmp/dist-test-task8XuDZV/build/debug/bin/kudu as pid 24656
I20251209 08:58:22.104211 20677 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1765270668420570-20677-0/minicluster-data/ts-2/wal/instance
I20251209 08:58:22.104490 20677 external_mini_cluster.cc:1468] Setting key 9c3fb6e9674dccdbdb154f7cd7b7d0d0
Dec 09 08:58:22 dist-test-slave-8w6c krb5kdc[24272](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765270702, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.49.67@KRBTEST.COM for kudu/127.20.49.126@KRBTEST.COM
I20251209 08:58:22.111958 24359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:58:22.100607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.49.67:33377 (local address 127.20.49.126:46193)
1209 08:58:22.100826 (+ 219us) server_negotiation.cc:207] Beginning negotiation
1209 08:58:22.100831 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1209 08:58:22.101515 (+ 684us) server_negotiation.cc:408] Connection header received
1209 08:58:22.102437 (+ 922us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1209 08:58:22.102441 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1209 08:58:22.102494 (+ 53us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1209 08:58:22.102576 (+ 82us) server_negotiation.cc:227] Negotiated authn=SASL
1209 08:58:22.104331 (+ 1755us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:58:22.105037 (+ 706us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1209 08:58:22.105944 (+ 907us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:58:22.106143 (+ 199us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:58:22.108612 (+ 2469us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1209 08:58:22.108628 (+ 16us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1209 08:58:22.108632 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1209 08:58:22.108663 (+ 31us) server_negotiation.cc:923] calling sasl_server_start()
1209 08:58:22.110047 (+ 1384us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1209 08:58:22.110544 (+ 497us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1209 08:58:22.110547 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1209 08:58:22.110548 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1209 08:58:22.110594 (+ 46us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1209 08:58:22.110882 (+ 288us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1209 08:58:22.110885 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1209 08:58:22.110887 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1209 08:58:22.111043 (+ 156us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1209 08:58:22.111135 (+ 92us) server_negotiation.cc:1036] Waiting for connection context
1209 08:58:22.111634 (+ 499us) server_negotiation.cc:300] Negotiation successful
1209 08:58:22.111778 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20251209 08:58:22.112650 24787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:58:22.100860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.49.126:46193 (local address 127.20.49.67:33377)
1209 08:58:22.101358 (+ 498us) negotiation.cc:107] Waiting for socket to connect
1209 08:58:22.101398 (+ 40us) client_negotiation.cc:174] Beginning negotiation
1209 08:58:22.102219 (+ 821us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1209 08:58:22.102734 (+ 515us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1209 08:58:22.102743 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
1209 08:58:22.103177 (+ 434us) client_negotiation.cc:189] Negotiated authn=SASL
1209 08:58:22.104155 (+ 978us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:58:22.104170 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:58:22.105184 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1209 08:58:22.105189 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1209 08:58:22.105779 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:58:22.105791 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:58:22.106054 (+ 263us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:58:22.106740 (+ 686us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1209 08:58:22.106782 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
1209 08:58:22.108457 (+ 1675us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1209 08:58:22.110169 (+ 1712us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1209 08:58:22.110175 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1209 08:58:22.110189 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
1209 08:58:22.110433 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1209 08:58:22.110679 (+ 246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1209 08:58:22.110684 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1209 08:58:22.110686 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1209 08:58:22.110792 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1209 08:58:22.111149 (+ 357us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1209 08:58:22.111158 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1209 08:58:22.111396 (+ 238us) client_negotiation.cc:769] Sending connection context
1209 08:58:22.111592 (+ 196us) client_negotiation.cc:240] Negotiation successful
1209 08:58:22.111879 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":313,"thread_start_us":119,"threads_started":1}
I20251209 08:58:22.113672 24785 heartbeater.cc:344] Connected to a master server at 127.20.49.126:46193
I20251209 08:58:22.113970 24785 heartbeater.cc:461] Registering TS with master...
I20251209 08:58:22.114499 24785 heartbeater.cc:507] Master 127.20.49.126:46193 requested a full tablet report, sending...
I20251209 08:58:22.115510 24320 ts_manager.cc:194] Registered new tserver with Master: a565149237674fd2a9538699ded75641 (127.20.49.67:33653)
I20251209 08:58:22.116094 24320 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.49.67@KRBTEST.COM'} at 127.20.49.67:33377
I20251209 08:58:22.119709 20677 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
Dec 09 08:58:22 dist-test-slave-8w6c krb5kdc[24272](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765270702, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Dec 09 08:58:22 dist-test-slave-8w6c krb5kdc[24272](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765270702, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.20.49.126@KRBTEST.COM
I20251209 08:58:22.144131 24359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:58:22.136427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53202 (local address 127.20.49.126:46193)
1209 08:58:22.136584 (+ 157us) server_negotiation.cc:207] Beginning negotiation
1209 08:58:22.136589 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1209 08:58:22.136624 (+ 35us) server_negotiation.cc:408] Connection header received
1209 08:58:22.136740 (+ 116us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1209 08:58:22.136743 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1209 08:58:22.136821 (+ 78us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1209 08:58:22.136889 (+ 68us) server_negotiation.cc:227] Negotiated authn=SASL
1209 08:58:22.137781 (+ 892us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:58:22.138303 (+ 522us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1209 08:58:22.138984 (+ 681us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:58:22.139136 (+ 152us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:58:22.141324 (+ 2188us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1209 08:58:22.141345 (+ 21us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1209 08:58:22.141348 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1209 08:58:22.141377 (+ 29us) server_negotiation.cc:923] calling sasl_server_start()
1209 08:58:22.142730 (+ 1353us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1209 08:58:22.143120 (+ 390us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1209 08:58:22.143124 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1209 08:58:22.143125 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1209 08:58:22.143174 (+ 49us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1209 08:58:22.143376 (+ 202us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1209 08:58:22.143379 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1209 08:58:22.143381 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1209 08:58:22.143569 (+ 188us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1209 08:58:22.143675 (+ 106us) server_negotiation.cc:1036] Waiting for connection context
1209 08:58:22.143821 (+ 146us) server_negotiation.cc:300] Negotiation successful
1209 08:58:22.143921 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
W20251209 08:58:22.144536 24320 server_base.cc:1129] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:53202
I20251209 08:58:22.145352 20677 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task8XuDZV/build/debug/bin/kudu with pid 24384
I20251209 08:58:22.151630 20677 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task8XuDZV/build/debug/bin/kudu with pid 24520
I20251209 08:58:22.157639 20677 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task8XuDZV/build/debug/bin/kudu with pid 24656
I20251209 08:58:22.163347 20677 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task8XuDZV/build/debug/bin/kudu with pid 24288
2025-12-09T08:58:22Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (3210 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765270668420570-20677-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Dec 09 08:58:22 dist-test-slave-8w6c krb5kdc[24797](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Dec 09 08:58:22 dist-test-slave-8w6c krb5kdc[24797](info): set up 2 sockets
Dec 09 08:58:22 dist-test-slave-8w6c krb5kdc[24797](info): commencing operation
krb5kdc: starting...
W20251209 08:58:24.154393 20677 mini_kdc.cc:121] Time spent starting KDC: real 1.968s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765270668420570-20677-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765270668420570-20677-0/krb5kdc/test-user.keytab.
Dec 09 08:58:24 dist-test-slave-8w6c krb5kdc[24797](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765270704, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-12-09T08:58:24Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-12-09T08:58:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.49.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.49.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.49.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765270668420570-20677-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.20.49.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765270668420570-20677-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.20.49.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.20.49.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.20.49.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765270668420570-20677-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.20.49.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765270668420570-20677-0/krb5kdc/kudu.keytab.
I20251209 08:58:24.294026 20677 external_mini_cluster.cc:1366] Running /tmp/dist-test-task8XuDZV/build/debug/bin/kudu
/tmp/dist-test-task8XuDZV/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765270668420570-20677-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765270668420570-20677-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765270668420570-20677-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765270668420570-20677-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.49.126:34643
--webserver_interface=127.20.49.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765270668420570-20677-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.49.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.20.49.84:39967
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.49.126:34643
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765270668420570-20677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765270668420570-20677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765270668420570-20677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251209 08:58:24.403612 24813 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251209 08:58:24.403870 24813 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251209 08:58:24.403929 24813 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251209 08:58:24.407284 24813 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251209 08:58:24.407357 24813 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251209 08:58:24.407380 24813 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251209 08:58:24.407423 24813 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251209 08:58:24.407459 24813 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251209 08:58:24.411835 24813 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.49.84:39967
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765270668420570-20677-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765270668420570-20677-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.49.126:34643
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765270668420570-20677-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.49.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.49.126:34643
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765270668420570-20677-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.49.126
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24813
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765270668420570-20677-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 468897755969d686fb08386f42f7835685e7953a
build type DEBUG
built by None at 09 Dec 2025 08:43:15 UTC on 24a791456cd2
build id 9426
I20251209 08:58:24.413048 24813 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251209 08:58:24.414000 24813 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251209 08:58:24.419759 24821 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251209 08:58:24.419785 24818 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251209 08:58:24.419759 24819 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251209 08:58:24.420154 24813 server_base.cc:1047] running on GCE node
I20251209 08:58:24.420670 24813 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251209 08:58:24.421655 24813 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251209 08:58:24.422931 24813 hybrid_clock.cc:648] HybridClock initialized: now 1765270704422907 us; error 41 us; skew 500 ppm
Dec 09 08:58:24 dist-test-slave-8w6c krb5kdc[24797](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765270704, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.49.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251209 08:58:24.425783 24813 init.cc:377] Logged in from keytab as kudu/127.20.49.126@KRBTEST.COM (short username kudu)
I20251209 08:58:24.426859 24813 webserver.cc:492] Webserver started at http://127.20.49.126:38149/ using document root <none> and password file <none>
I20251209 08:58:24.427407 24813 fs_manager.cc:362] Metadata directory not provided
I20251209 08:58:24.427456 24813 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251209 08:58:24.427618 24813 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251209 08:58:24.429358 24813 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765270668420570-20677-0/minicluster-data/master-0/data/instance:
uuid: "7a3ac58975f64d55ac62d7175b25ae3f"
format_stamp: "Formatted at 2025-12-09 08:58:24 on dist-test-slave-8w6c"
server_key: "2adf1eb7d8da3adb85d1a67f35446e52"
server_key_iv: "f013d6d7d179f409f9d2b198aadbd7e6"
server_key_version: "encryptionkey@0"
I20251209 08:58:24.429865 24813 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765270668420570-20677-0/minicluster-data/master-0/wal/instance:
uuid: "7a3ac58975f64d55ac62d7175b25ae3f"
format_stamp: "Formatted at 2025-12-09 08:58:24 on dist-test-slave-8w6c"
server_key: "2adf1eb7d8da3adb85d1a67f35446e52"
server_key_iv: "f013d6d7d179f409f9d2b198aadbd7e6"
server_key_version: "encryptionkey@0"
I20251209 08:58:24.433265 24813 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20251209 08:58:24.435500 24828 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251209 08:58:24.436568 24813 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20251209 08:58:24.436669 24813 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765270668420570-20677-0/minicluster-data/master-0/data,/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765270668420570-20677-0/minicluster-data/master-0/wal
uuid: "7a3ac58975f64d55ac62d7175b25ae3f"
format_stamp: "Formatted at 2025-12-09 08:58:24 on dist-test-slave-8w6c"
server_key: "2adf1eb7d8da3adb85d1a67f35446e52"
server_key_iv: "f013d6d7d179f409f9d2b198aadbd7e6"
server_key_version: "encryptionkey@0"
I20251209 08:58:24.436779 24813 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765270668420570-20677-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765270668420570-20677-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765270668420570-20677-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251209 08:58:24.450994 24813 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251209 08:58:24.454141 24813 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251209 08:58:24.454325 24813 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251209 08:58:24.461449 24813 rpc_server.cc:307] RPC server started. Bound to: 127.20.49.126:34643
I20251209 08:58:24.461460 24880 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.49.126:34643 every 8 connection(s)
I20251209 08:58:24.462517 24813 server_base.cc:1179] Dumped server information to /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765270668420570-20677-0/minicluster-data/master-0/data/info.pb
I20251209 08:58:24.465268 24881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251209 08:58:24.469805 20677 external_mini_cluster.cc:1428] Started /tmp/dist-test-task8XuDZV/build/debug/bin/kudu as pid 24813
I20251209 08:58:24.469951 20677 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765270668420570-20677-0/minicluster-data/master-0/wal/instance
I20251209 08:58:24.470222 20677 external_mini_cluster.cc:1468] Setting key 00f5349df2f010f1affb8c551f6e4478
I20251209 08:58:24.471009 24881 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7a3ac58975f64d55ac62d7175b25ae3f: Bootstrap starting.
I20251209 08:58:24.473291 24881 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7a3ac58975f64d55ac62d7175b25ae3f: Neither blocks nor log segments found. Creating new log.
I20251209 08:58:24.474233 24881 log.cc:826] T 00000000000000000000000000000000 P 7a3ac58975f64d55ac62d7175b25ae3f: Log is configured to *not* fsync() on all Append() calls
I20251209 08:58:24.476220 24881 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7a3ac58975f64d55ac62d7175b25ae3f: No bootstrap required, opened a new log
Dec 09 08:58:24 dist-test-slave-8w6c krb5kdc[24797](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765270704, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.49.126@KRBTEST.COM
I20251209 08:58:24.479161 24881 raft_consensus.cc:359] T 00000000000000000000000000000000 P 7a3ac58975f64d55ac62d7175b25ae3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a3ac58975f64d55ac62d7175b25ae3f" member_type: VOTER last_known_addr { host: "127.20.49.126" port: 34643 } }
I20251209 08:58:24.479346 24881 raft_consensus.cc:385] T 00000000000000000000000000000000 P 7a3ac58975f64d55ac62d7175b25ae3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251209 08:58:24.479414 24881 raft_consensus.cc:740] T 00000000000000000000000000000000 P 7a3ac58975f64d55ac62d7175b25ae3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a3ac58975f64d55ac62d7175b25ae3f, State: Initialized, Role: FOLLOWER
I20251209 08:58:24.479905 24881 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7a3ac58975f64d55ac62d7175b25ae3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a3ac58975f64d55ac62d7175b25ae3f" member_type: VOTER last_known_addr { host: "127.20.49.126" port: 34643 } }
I20251209 08:58:24.480050 24881 raft_consensus.cc:399] T 00000000000000000000000000000000 P 7a3ac58975f64d55ac62d7175b25ae3f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251209 08:58:24.480129 24881 raft_consensus.cc:493] T 00000000000000000000000000000000 P 7a3ac58975f64d55ac62d7175b25ae3f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251209 08:58:24.480223 24881 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 7a3ac58975f64d55ac62d7175b25ae3f [term 0 FOLLOWER]: Advancing to term 1
I20251209 08:58:24.481026 24881 raft_consensus.cc:515] T 00000000000000000000000000000000 P 7a3ac58975f64d55ac62d7175b25ae3f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a3ac58975f64d55ac62d7175b25ae3f" member_type: VOTER last_known_addr { host: "127.20.49.126" port: 34643 } }
I20251209 08:58:24.481364 24881 leader_election.cc:304] T 00000000000000000000000000000000 P 7a3ac58975f64d55ac62d7175b25ae3f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7a3ac58975f64d55ac62d7175b25ae3f; no voters:
I20251209 08:58:24.481662 24881 leader_election.cc:290] T 00000000000000000000000000000000 P 7a3ac58975f64d55ac62d7175b25ae3f [CANDIDATE]: Term 1 election: Requested vote from peers
I20251209 08:58:24.481801 24886 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 7a3ac58975f64d55ac62d7175b25ae3f [term 1 FOLLOWER]: Leader election won for term 1
I20251209 08:58:24.482327 24886 raft_consensus.cc:697] T 00000000000000000000000000000000 P 7a3ac58975f64d55ac62d7175b25ae3f [term 1 LEADER]: Becoming Leader. State: Replica: 7a3ac58975f64d55ac62d7175b25ae3f, State: Running, Role: LEADER
I20251209 08:58:24.482676 24886 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7a3ac58975f64d55ac62d7175b25ae3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a3ac58975f64d55ac62d7175b25ae3f" member_type: VOTER last_known_addr { host: "127.20.49.126" port: 34643 } }
I20251209 08:58:24.482945 24881 sys_catalog.cc:565] T 00000000000000000000000000000000 P 7a3ac58975f64d55ac62d7175b25ae3f [sys.catalog]: configured and running, proceeding with master startup.
I20251209 08:58:24.483580 24884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:58:24.471538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58810 (local address 127.20.49.126:34643)
1209 08:58:24.471943 (+ 405us) server_negotiation.cc:207] Beginning negotiation
1209 08:58:24.471953 (+ 10us) server_negotiation.cc:400] Waiting for connection header
1209 08:58:24.471987 (+ 34us) server_negotiation.cc:408] Connection header received
1209 08:58:24.472516 (+ 529us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1209 08:58:24.472533 (+ 17us) server_negotiation.cc:462] Received NEGOTIATE request from client
1209 08:58:24.472817 (+ 284us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1209 08:58:24.473090 (+ 273us) server_negotiation.cc:227] Negotiated authn=SASL
1209 08:58:24.474246 (+ 1156us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:58:24.475023 (+ 777us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1209 08:58:24.475863 (+ 840us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:58:24.476069 (+ 206us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:58:24.478612 (+ 2543us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1209 08:58:24.478631 (+ 19us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1209 08:58:24.478641 (+ 10us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1209 08:58:24.478674 (+ 33us) server_negotiation.cc:923] calling sasl_server_start()
1209 08:58:24.480716 (+ 2042us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1209 08:58:24.481161 (+ 445us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1209 08:58:24.481169 (+ 8us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1209 08:58:24.481175 (+ 6us) server_negotiation.cc:968] calling sasl_server_step()
1209 08:58:24.481269 (+ 94us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1209 08:58:24.481499 (+ 230us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1209 08:58:24.481502 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1209 08:58:24.481504 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1209 08:58:24.481963 (+ 459us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1209 08:58:24.482185 (+ 222us) server_negotiation.cc:1036] Waiting for connection context
1209 08:58:24.482646 (+ 461us) server_negotiation.cc:300] Negotiation successful
1209 08:58:24.482946 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":277,"thread_start_us":130,"threads_started":1}
I20251209 08:58:24.483894 24886 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a3ac58975f64d55ac62d7175b25ae3f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7a3ac58975f64d55ac62d7175b25ae3f. Latest consensus state: current_term: 1 leader_uuid: "7a3ac58975f64d55ac62d7175b25ae3f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a3ac58975f64d55ac62d7175b25ae3f" member_type: VOTER last_known_addr { host: "127.20.49.126" port: 34643 } } }
I20251209 08:58:24.484018 24886 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a3ac58975f64d55ac62d7175b25ae3f [sys.catalog]: This master's current role is: LEADER
I20251209 08:58:24.484499 24887 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a3ac58975f64d55ac62d7175b25ae3f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7a3ac58975f64d55ac62d7175b25ae3f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a3ac58975f64d55ac62d7175b25ae3f" member_type: VOTER last_known_addr { host: "127.20.49.126" port: 34643 } } }
I20251209 08:58:24.484829 24887 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a3ac58975f64d55ac62d7175b25ae3f [sys.catalog]: This master's current role is: LEADER
I20251209 08:58:24.485616 24896 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251209 08:58:24.488198 24896 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251209 08:58:24.493405 24896 catalog_manager.cc:1357] Generated new cluster ID: 0d77cd8558d44ae5bb65d8558b94f8ae
I20251209 08:58:24.493486 24896 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251209 08:58:24.518293 24896 catalog_manager.cc:1380] Generated new certificate authority record
I20251209 08:58:24.519611 24896 catalog_manager.cc:1514] Loading token signing keys...
I20251209 08:58:24.532473 24896 catalog_manager.cc:6027] T 00000000000000000000000000000000 P 7a3ac58975f64d55ac62d7175b25ae3f: Generated new TSK 0
I20251209 08:58:24.533434 24896 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.20.49.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.49.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.49.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765270668420570-20677-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.20.49.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765270668420570-20677-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.20.49.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.20.49.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.20.49.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765270668420570-20677-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.20.49.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765270668420570-20677-0/krb5kdc/kudu.keytab.
I20251209 08:58:24.589018 20677 external_mini_cluster.cc:1366] Running /tmp/dist-test-task8XuDZV/build/debug/bin/kudu
/tmp/dist-test-task8XuDZV/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765270668420570-20677-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765270668420570-20677-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765270668420570-20677-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765270668420570-20677-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.49.65:0
--local_ip_for_outbound_sockets=127.20.49.65
--webserver_interface=127.20.49.65
--webserver_port=0
--tserver_master_addrs=127.20.49.126:34643
--keytab_file=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765270668420570-20677-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.49.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.20.49.84:39967
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765270668420570-20677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765270668420570-20677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765270668420570-20677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251209 08:58:24.715204 24909 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251209 08:58:24.715440 24909 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251209 08:58:24.715502 24909 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251209 08:58:24.719043 24909 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251209 08:58:24.719123 24909 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251209 08:58:24.719206 24909 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.49.65
I20251209 08:58:24.723672 24909 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.49.84:39967
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765270668420570-20677-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765270668420570-20677-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765270668420570-20677-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.49.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.49.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765270668420570-20677-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.49.65
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.20.49.126:34643
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24909
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.49.65
--log_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765270668420570-20677-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 468897755969d686fb08386f42f7835685e7953a
build type DEBUG
built by None at 09 Dec 2025 08:43:15 UTC on 24a791456cd2
build id 9426
I20251209 08:58:24.724818 24909 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251209 08:58:24.725759 24909 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251209 08:58:24.732532 24917 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251209 08:58:24.732532 24915 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251209 08:58:24.732532 24914 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251209 08:58:24.732893 24909 server_base.cc:1047] running on GCE node
I20251209 08:58:24.733273 24909 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251209 08:58:24.733952 24909 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251209 08:58:24.735131 24909 hybrid_clock.cc:648] HybridClock initialized: now 1765270704735105 us; error 49 us; skew 500 ppm
Dec 09 08:58:24 dist-test-slave-8w6c krb5kdc[24797](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765270704, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.49.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251209 08:58:24.738168 24909 init.cc:377] Logged in from keytab as kudu/127.20.49.65@KRBTEST.COM (short username kudu)
I20251209 08:58:24.739370 24909 webserver.cc:492] Webserver started at http://127.20.49.65:44157/ using document root <none> and password file <none>
I20251209 08:58:24.739964 24909 fs_manager.cc:362] Metadata directory not provided
I20251209 08:58:24.740021 24909 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251209 08:58:24.740206 24909 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251209 08:58:24.742048 24909 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765270668420570-20677-0/minicluster-data/ts-0/data/instance:
uuid: "e549e2ebf41f469f92cc60d570d56475"
format_stamp: "Formatted at 2025-12-09 08:58:24 on dist-test-slave-8w6c"
server_key: "0483fdd021d663e68cdaa1fd75a35dc8"
server_key_iv: "75b5b1f240c8d271d08cdcb49c69b5b7"
server_key_version: "encryptionkey@0"
I20251209 08:58:24.742538 24909 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765270668420570-20677-0/minicluster-data/ts-0/wal/instance:
uuid: "e549e2ebf41f469f92cc60d570d56475"
format_stamp: "Formatted at 2025-12-09 08:58:24 on dist-test-slave-8w6c"
server_key: "0483fdd021d663e68cdaa1fd75a35dc8"
server_key_iv: "75b5b1f240c8d271d08cdcb49c69b5b7"
server_key_version: "encryptionkey@0"
I20251209 08:58:24.746156 24909 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20251209 08:58:24.748562 24924 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251209 08:58:24.749799 24909 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20251209 08:58:24.749950 24909 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765270668420570-20677-0/minicluster-data/ts-0/data,/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765270668420570-20677-0/minicluster-data/ts-0/wal
uuid: "e549e2ebf41f469f92cc60d570d56475"
format_stamp: "Formatted at 2025-12-09 08:58:24 on dist-test-slave-8w6c"
server_key: "0483fdd021d663e68cdaa1fd75a35dc8"
server_key_iv: "75b5b1f240c8d271d08cdcb49c69b5b7"
server_key_version: "encryptionkey@0"
I20251209 08:58:24.750049 24909 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765270668420570-20677-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765270668420570-20677-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765270668420570-20677-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251209 08:58:24.768424 24909 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251209 08:58:24.771682 24909 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251209 08:58:24.771906 24909 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251209 08:58:24.772500 24909 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251209 08:58:24.773425 24909 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251209 08:58:24.773479 24909 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251209 08:58:24.773543 24909 ts_tablet_manager.cc:616] Registered 0 tablets
I20251209 08:58:24.773576 24909 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251209 08:58:24.783073 24909 rpc_server.cc:307] RPC server started. Bound to: 127.20.49.65:38891
I20251209 08:58:24.783097 25037 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.49.65:38891 every 8 connection(s)
I20251209 08:58:24.784137 24909 server_base.cc:1179] Dumped server information to /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765270668420570-20677-0/minicluster-data/ts-0/data/info.pb
I20251209 08:58:24.785974 20677 external_mini_cluster.cc:1428] Started /tmp/dist-test-task8XuDZV/build/debug/bin/kudu as pid 24909
I20251209 08:58:24.786103 20677 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765270668420570-20677-0/minicluster-data/ts-0/wal/instance
I20251209 08:58:24.786372 20677 external_mini_cluster.cc:1468] Setting key 2ea9d7fa0bfc49cca6f08bd75f8977e2
Dec 09 08:58:24 dist-test-slave-8w6c krb5kdc[24797](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765270704, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.49.65@KRBTEST.COM for kudu/127.20.49.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.20.49.66@KRBTEST.COM; defaulting to no policy
I20251209 08:58:24.798578 24884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:58:24.786021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.49.65:40689 (local address 127.20.49.126:34643)
1209 08:58:24.786213 (+ 192us) server_negotiation.cc:207] Beginning negotiation
1209 08:58:24.786218 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1209 08:58:24.786978 (+ 760us) server_negotiation.cc:408] Connection header received
1209 08:58:24.788026 (+ 1048us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1209 08:58:24.788029 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1209 08:58:24.788079 (+ 50us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1209 08:58:24.788158 (+ 79us) server_negotiation.cc:227] Negotiated authn=SASL
1209 08:58:24.790517 (+ 2359us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:58:24.791368 (+ 851us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1209 08:58:24.792183 (+ 815us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:58:24.792433 (+ 250us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:58:24.794561 (+ 2128us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1209 08:58:24.794587 (+ 26us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1209 08:58:24.794593 (+ 6us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1209 08:58:24.794628 (+ 35us) server_negotiation.cc:923] calling sasl_server_start()
1209 08:58:24.796561 (+ 1933us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1209 08:58:24.797113 (+ 552us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1209 08:58:24.797116 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1209 08:58:24.797118 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1209 08:58:24.797166 (+ 48us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1209 08:58:24.797414 (+ 248us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1209 08:58:24.797416 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1209 08:58:24.797417 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1209 08:58:24.797627 (+ 210us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1209 08:58:24.797760 (+ 133us) server_negotiation.cc:1036] Waiting for connection context
1209 08:58:24.798281 (+ 521us) server_negotiation.cc:300] Negotiation successful
1209 08:58:24.798417 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.49.66@KRBTEST.COM" created.
I20251209 08:58:24.799259 25040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:58:24.786317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.49.126:34643 (local address 127.20.49.65:40689)
1209 08:58:24.786822 (+ 505us) negotiation.cc:107] Waiting for socket to connect
1209 08:58:24.786860 (+ 38us) client_negotiation.cc:174] Beginning negotiation
1209 08:58:24.787757 (+ 897us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1209 08:58:24.788374 (+ 617us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1209 08:58:24.788389 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
1209 08:58:24.788995 (+ 606us) client_negotiation.cc:189] Negotiated authn=SASL
1209 08:58:24.790321 (+ 1326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:58:24.790337 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:58:24.791549 (+ 1212us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1209 08:58:24.791552 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1209 08:58:24.792022 (+ 470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:58:24.792030 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:58:24.792307 (+ 277us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:58:24.792936 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1209 08:58:24.792960 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
1209 08:58:24.794405 (+ 1445us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1209 08:58:24.796713 (+ 2308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1209 08:58:24.796718 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1209 08:58:24.796728 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
1209 08:58:24.796979 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1209 08:58:24.797254 (+ 275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1209 08:58:24.797256 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1209 08:58:24.797258 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1209 08:58:24.797345 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1209 08:58:24.797761 (+ 416us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1209 08:58:24.797766 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1209 08:58:24.798036 (+ 270us) client_negotiation.cc:769] Sending connection context
1209 08:58:24.798222 (+ 186us) client_negotiation.cc:240] Negotiation successful
1209 08:58:24.798488 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":282,"thread_start_us":123,"threads_started":1}
I20251209 08:58:24.800544 25038 heartbeater.cc:344] Connected to a master server at 127.20.49.126:34643
I20251209 08:58:24.800808 25038 heartbeater.cc:461] Registering TS with master...
I20251209 08:58:24.801517 25038 heartbeater.cc:507] Master 127.20.49.126:34643 requested a full tablet report, sending...
I20251209 08:58:24.803261 24845 ts_manager.cc:194] Registered new tserver with Master: e549e2ebf41f469f92cc60d570d56475 (127.20.49.65:38891)
I20251209 08:58:24.805004 24845 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.49.65@KRBTEST.COM'} at 127.20.49.65:40689
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.49.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765270668420570-20677-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.20.49.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765270668420570-20677-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.20.49.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.20.49.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.20.49.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765270668420570-20677-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.20.49.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765270668420570-20677-0/krb5kdc/kudu.keytab.
I20251209 08:58:24.838701 20677 external_mini_cluster.cc:1366] Running /tmp/dist-test-task8XuDZV/build/debug/bin/kudu
/tmp/dist-test-task8XuDZV/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765270668420570-20677-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765270668420570-20677-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765270668420570-20677-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765270668420570-20677-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.49.66:0
--local_ip_for_outbound_sockets=127.20.49.66
--webserver_interface=127.20.49.66
--webserver_port=0
--tserver_master_addrs=127.20.49.126:34643
--keytab_file=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765270668420570-20677-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.49.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.20.49.84:39967
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765270668420570-20677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765270668420570-20677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765270668420570-20677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251209 08:58:24.948778 25045 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251209 08:58:24.949026 25045 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251209 08:58:24.949093 25045 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251209 08:58:24.952466 25045 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251209 08:58:24.952546 25045 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251209 08:58:24.952628 25045 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.49.66
I20251209 08:58:24.957000 25045 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.49.84:39967
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765270668420570-20677-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765270668420570-20677-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765270668420570-20677-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.49.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.49.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765270668420570-20677-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.49.66
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.20.49.126:34643
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25045
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.49.66
--log_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765270668420570-20677-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 468897755969d686fb08386f42f7835685e7953a
build type DEBUG
built by None at 09 Dec 2025 08:43:15 UTC on 24a791456cd2
build id 9426
I20251209 08:58:24.958195 25045 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251209 08:58:24.959007 25045 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251209 08:58:24.965655 25053 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251209 08:58:24.965655 25050 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251209 08:58:24.965718 25051 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251209 08:58:24.965941 25045 server_base.cc:1047] running on GCE node
I20251209 08:58:24.966336 25045 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251209 08:58:24.966887 25045 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251209 08:58:24.968047 25045 hybrid_clock.cc:648] HybridClock initialized: now 1765270704968023 us; error 39 us; skew 500 ppm
Dec 09 08:58:24 dist-test-slave-8w6c krb5kdc[24797](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765270704, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.49.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251209 08:58:24.970916 25045 init.cc:377] Logged in from keytab as kudu/127.20.49.66@KRBTEST.COM (short username kudu)
I20251209 08:58:24.972074 25045 webserver.cc:492] Webserver started at http://127.20.49.66:35217/ using document root <none> and password file <none>
I20251209 08:58:24.972649 25045 fs_manager.cc:362] Metadata directory not provided
I20251209 08:58:24.972711 25045 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251209 08:58:24.972901 25045 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251209 08:58:24.974781 25045 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765270668420570-20677-0/minicluster-data/ts-1/data/instance:
uuid: "6dd437fc0a904e55a5a1e3a6dff19046"
format_stamp: "Formatted at 2025-12-09 08:58:24 on dist-test-slave-8w6c"
server_key: "3e4c9fb3adca4d618f97f1c05fc98421"
server_key_iv: "c71ccfaf29ebbd739e8b939ca1f9a60f"
server_key_version: "encryptionkey@0"
I20251209 08:58:24.975283 25045 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765270668420570-20677-0/minicluster-data/ts-1/wal/instance:
uuid: "6dd437fc0a904e55a5a1e3a6dff19046"
format_stamp: "Formatted at 2025-12-09 08:58:24 on dist-test-slave-8w6c"
server_key: "3e4c9fb3adca4d618f97f1c05fc98421"
server_key_iv: "c71ccfaf29ebbd739e8b939ca1f9a60f"
server_key_version: "encryptionkey@0"
I20251209 08:58:24.978730 25045 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20251209 08:58:24.980909 25060 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251209 08:58:24.981951 25045 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20251209 08:58:24.982062 25045 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765270668420570-20677-0/minicluster-data/ts-1/data,/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765270668420570-20677-0/minicluster-data/ts-1/wal
uuid: "6dd437fc0a904e55a5a1e3a6dff19046"
format_stamp: "Formatted at 2025-12-09 08:58:24 on dist-test-slave-8w6c"
server_key: "3e4c9fb3adca4d618f97f1c05fc98421"
server_key_iv: "c71ccfaf29ebbd739e8b939ca1f9a60f"
server_key_version: "encryptionkey@0"
I20251209 08:58:24.982158 25045 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765270668420570-20677-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765270668420570-20677-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765270668420570-20677-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251209 08:58:25.005990 25045 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251209 08:58:25.008883 25045 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251209 08:58:25.009080 25045 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251209 08:58:25.009657 25045 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251209 08:58:25.010612 25045 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251209 08:58:25.010664 25045 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251209 08:58:25.010726 25045 ts_tablet_manager.cc:616] Registered 0 tablets
I20251209 08:58:25.010759 25045 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251209 08:58:25.020047 25045 rpc_server.cc:307] RPC server started. Bound to: 127.20.49.66:44487
I20251209 08:58:25.020135 25173 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.49.66:44487 every 8 connection(s)
I20251209 08:58:25.021044 25045 server_base.cc:1179] Dumped server information to /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765270668420570-20677-0/minicluster-data/ts-1/data/info.pb
I20251209 08:58:25.025516 20677 external_mini_cluster.cc:1428] Started /tmp/dist-test-task8XuDZV/build/debug/bin/kudu as pid 25045
I20251209 08:58:25.025650 20677 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765270668420570-20677-0/minicluster-data/ts-1/wal/instance
I20251209 08:58:25.025944 20677 external_mini_cluster.cc:1468] Setting key 1466b59987e0674ba5bddbea75e3ae0b
Dec 09 08:58:25 dist-test-slave-8w6c krb5kdc[24797](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765270704, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.49.66@KRBTEST.COM for kudu/127.20.49.126@KRBTEST.COM
I20251209 08:58:25.035226 24884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:58:25.022864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.49.66:43695 (local address 127.20.49.126:34643)
1209 08:58:25.023033 (+ 169us) server_negotiation.cc:207] Beginning negotiation
1209 08:58:25.023037 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1209 08:58:25.023713 (+ 676us) server_negotiation.cc:408] Connection header received
1209 08:58:25.024597 (+ 884us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1209 08:58:25.024601 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1209 08:58:25.024664 (+ 63us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1209 08:58:25.024746 (+ 82us) server_negotiation.cc:227] Negotiated authn=SASL
1209 08:58:25.026882 (+ 2136us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:58:25.027386 (+ 504us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1209 08:58:25.028080 (+ 694us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:58:25.028262 (+ 182us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:58:25.031039 (+ 2777us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1209 08:58:25.031056 (+ 17us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1209 08:58:25.031058 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1209 08:58:25.031086 (+ 28us) server_negotiation.cc:923] calling sasl_server_start()
1209 08:58:25.032965 (+ 1879us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1209 08:58:25.033613 (+ 648us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1209 08:58:25.033616 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1209 08:58:25.033617 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1209 08:58:25.033664 (+ 47us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1209 08:58:25.033990 (+ 326us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1209 08:58:25.033992 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1209 08:58:25.033994 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1209 08:58:25.034145 (+ 151us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1209 08:58:25.034227 (+ 82us) server_negotiation.cc:1036] Waiting for connection context
1209 08:58:25.034948 (+ 721us) server_negotiation.cc:300] Negotiation successful
1209 08:58:25.035081 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
WARNING: no policy specified for kudu/127.20.49.67@KRBTEST.COM; defaulting to no policy
I20251209 08:58:25.036149 25176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:58:25.023123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.49.126:34643 (local address 127.20.49.66:43695)
1209 08:58:25.023571 (+ 448us) negotiation.cc:107] Waiting for socket to connect
1209 08:58:25.023608 (+ 37us) client_negotiation.cc:174] Beginning negotiation
1209 08:58:25.024388 (+ 780us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1209 08:58:25.024923 (+ 535us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1209 08:58:25.024932 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
1209 08:58:25.025423 (+ 491us) client_negotiation.cc:189] Negotiated authn=SASL
1209 08:58:25.026735 (+ 1312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:58:25.026756 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:58:25.027515 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1209 08:58:25.027521 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1209 08:58:25.027964 (+ 443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:58:25.027972 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:58:25.028207 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:58:25.028946 (+ 739us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1209 08:58:25.028971 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
1209 08:58:25.030854 (+ 1883us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1209 08:58:25.033099 (+ 2245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1209 08:58:25.033109 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1209 08:58:25.033124 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
1209 08:58:25.033494 (+ 370us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1209 08:58:25.033764 (+ 270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1209 08:58:25.033768 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1209 08:58:25.033770 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1209 08:58:25.033899 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1209 08:58:25.034280 (+ 381us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1209 08:58:25.034288 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1209 08:58:25.034622 (+ 334us) client_negotiation.cc:769] Sending connection context
1209 08:58:25.034885 (+ 263us) client_negotiation.cc:240] Negotiation successful
1209 08:58:25.035224 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":112,"threads_started":1}
I20251209 08:58:25.037360 25174 heartbeater.cc:344] Connected to a master server at 127.20.49.126:34643
I20251209 08:58:25.037616 25174 heartbeater.cc:461] Registering TS with master...
I20251209 08:58:25.038278 25174 heartbeater.cc:507] Master 127.20.49.126:34643 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.49.67@KRBTEST.COM" created.
I20251209 08:58:25.039402 24845 ts_manager.cc:194] Registered new tserver with Master: 6dd437fc0a904e55a5a1e3a6dff19046 (127.20.49.66:44487)
I20251209 08:58:25.039978 24845 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.49.66@KRBTEST.COM'} at 127.20.49.66:43695
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.49.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765270668420570-20677-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.20.49.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765270668420570-20677-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.20.49.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.20.49.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.20.49.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765270668420570-20677-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.20.49.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765270668420570-20677-0/krb5kdc/kudu.keytab.
I20251209 08:58:25.079229 20677 external_mini_cluster.cc:1366] Running /tmp/dist-test-task8XuDZV/build/debug/bin/kudu
/tmp/dist-test-task8XuDZV/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765270668420570-20677-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765270668420570-20677-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765270668420570-20677-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765270668420570-20677-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.49.67:0
--local_ip_for_outbound_sockets=127.20.49.67
--webserver_interface=127.20.49.67
--webserver_port=0
--tserver_master_addrs=127.20.49.126:34643
--keytab_file=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765270668420570-20677-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.49.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.20.49.84:39967
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765270668420570-20677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765270668420570-20677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765270668420570-20677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251209 08:58:25.189111 25181 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251209 08:58:25.189358 25181 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251209 08:58:25.189451 25181 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251209 08:58:25.192768 25181 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251209 08:58:25.192857 25181 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251209 08:58:25.192975 25181 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.49.67
I20251209 08:58:25.197178 25181 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.49.84:39967
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765270668420570-20677-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765270668420570-20677-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765270668420570-20677-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.49.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.49.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765270668420570-20677-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.49.67
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.20.49.126:34643
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25181
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.49.67
--log_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765270668420570-20677-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 468897755969d686fb08386f42f7835685e7953a
build type DEBUG
built by None at 09 Dec 2025 08:43:15 UTC on 24a791456cd2
build id 9426
I20251209 08:58:25.198325 25181 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251209 08:58:25.199240 25181 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251209 08:58:25.206230 25186 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251209 08:58:25.206257 25187 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251209 08:58:25.206450 25181 server_base.cc:1047] running on GCE node
W20251209 08:58:25.206395 25189 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251209 08:58:25.206828 25181 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251209 08:58:25.207404 25181 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251209 08:58:25.208590 25181 hybrid_clock.cc:648] HybridClock initialized: now 1765270705208560 us; error 43 us; skew 500 ppm
Dec 09 08:58:25 dist-test-slave-8w6c krb5kdc[24797](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765270705, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.49.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251209 08:58:25.211500 25181 init.cc:377] Logged in from keytab as kudu/127.20.49.67@KRBTEST.COM (short username kudu)
I20251209 08:58:25.212770 25181 webserver.cc:492] Webserver started at http://127.20.49.67:37655/ using document root <none> and password file <none>
I20251209 08:58:25.213367 25181 fs_manager.cc:362] Metadata directory not provided
I20251209 08:58:25.213428 25181 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251209 08:58:25.213622 25181 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251209 08:58:25.215505 25181 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765270668420570-20677-0/minicluster-data/ts-2/data/instance:
uuid: "39e3edd9650845b789b10fe4ba685548"
format_stamp: "Formatted at 2025-12-09 08:58:25 on dist-test-slave-8w6c"
server_key: "80431c4b7b74b81511dd3764c751b892"
server_key_iv: "e7e210c2c814238e52bb86a41121ca45"
server_key_version: "encryptionkey@0"
I20251209 08:58:25.216018 25181 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765270668420570-20677-0/minicluster-data/ts-2/wal/instance:
uuid: "39e3edd9650845b789b10fe4ba685548"
format_stamp: "Formatted at 2025-12-09 08:58:25 on dist-test-slave-8w6c"
server_key: "80431c4b7b74b81511dd3764c751b892"
server_key_iv: "e7e210c2c814238e52bb86a41121ca45"
server_key_version: "encryptionkey@0"
I20251209 08:58:25.219686 25181 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20251209 08:58:25.222033 25196 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251209 08:58:25.223228 25181 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20251209 08:58:25.223353 25181 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765270668420570-20677-0/minicluster-data/ts-2/data,/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765270668420570-20677-0/minicluster-data/ts-2/wal
uuid: "39e3edd9650845b789b10fe4ba685548"
format_stamp: "Formatted at 2025-12-09 08:58:25 on dist-test-slave-8w6c"
server_key: "80431c4b7b74b81511dd3764c751b892"
server_key_iv: "e7e210c2c814238e52bb86a41121ca45"
server_key_version: "encryptionkey@0"
I20251209 08:58:25.223456 25181 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765270668420570-20677-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765270668420570-20677-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765270668420570-20677-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251209 08:58:25.234660 25181 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251209 08:58:25.237591 25181 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251209 08:58:25.237809 25181 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251209 08:58:25.238389 25181 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251209 08:58:25.239265 25181 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251209 08:58:25.239317 25181 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251209 08:58:25.239380 25181 ts_tablet_manager.cc:616] Registered 0 tablets
I20251209 08:58:25.239408 25181 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251209 08:58:25.248781 25181 rpc_server.cc:307] RPC server started. Bound to: 127.20.49.67:42097
I20251209 08:58:25.248822 25309 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.49.67:42097 every 8 connection(s)
I20251209 08:58:25.249815 25181 server_base.cc:1179] Dumped server information to /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765270668420570-20677-0/minicluster-data/ts-2/data/info.pb
I20251209 08:58:25.255422 20677 external_mini_cluster.cc:1428] Started /tmp/dist-test-task8XuDZV/build/debug/bin/kudu as pid 25181
I20251209 08:58:25.255560 20677 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1765270668420570-20677-0/minicluster-data/ts-2/wal/instance
I20251209 08:58:25.255802 20677 external_mini_cluster.cc:1468] Setting key aa693661515e923f3bf71d4eed7b92b8
Dec 09 08:58:25 dist-test-slave-8w6c krb5kdc[24797](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765270705, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.49.67@KRBTEST.COM for kudu/127.20.49.126@KRBTEST.COM
I20251209 08:58:25.262619 24884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:58:25.251625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.49.67:40823 (local address 127.20.49.126:34643)
1209 08:58:25.251816 (+ 191us) server_negotiation.cc:207] Beginning negotiation
1209 08:58:25.251822 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1209 08:58:25.252583 (+ 761us) server_negotiation.cc:408] Connection header received
1209 08:58:25.253407 (+ 824us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1209 08:58:25.253410 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1209 08:58:25.253462 (+ 52us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1209 08:58:25.253552 (+ 90us) server_negotiation.cc:227] Negotiated authn=SASL
1209 08:58:25.255737 (+ 2185us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:58:25.256313 (+ 576us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1209 08:58:25.257094 (+ 781us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:58:25.257302 (+ 208us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:58:25.259656 (+ 2354us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1209 08:58:25.259672 (+ 16us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1209 08:58:25.259674 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1209 08:58:25.259701 (+ 27us) server_negotiation.cc:923] calling sasl_server_start()
1209 08:58:25.261000 (+ 1299us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1209 08:58:25.261441 (+ 441us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1209 08:58:25.261443 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1209 08:58:25.261444 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1209 08:58:25.261485 (+ 41us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1209 08:58:25.261720 (+ 235us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1209 08:58:25.261726 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1209 08:58:25.261731 (+ 5us) server_negotiation.cc:968] calling sasl_server_step()
1209 08:58:25.261881 (+ 150us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1209 08:58:25.261960 (+ 79us) server_negotiation.cc:1036] Waiting for connection context
1209 08:58:25.262398 (+ 438us) server_negotiation.cc:300] Negotiation successful
1209 08:58:25.262492 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20251209 08:58:25.263296 25312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:58:25.251897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.49.126:34643 (local address 127.20.49.67:40823)
1209 08:58:25.252418 (+ 521us) negotiation.cc:107] Waiting for socket to connect
1209 08:58:25.252455 (+ 37us) client_negotiation.cc:174] Beginning negotiation
1209 08:58:25.253206 (+ 751us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1209 08:58:25.253717 (+ 511us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1209 08:58:25.253726 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
1209 08:58:25.254410 (+ 684us) client_negotiation.cc:189] Negotiated authn=SASL
1209 08:58:25.255545 (+ 1135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:58:25.255567 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:58:25.256448 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1209 08:58:25.256451 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1209 08:58:25.256964 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:58:25.256979 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:58:25.257219 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:58:25.257983 (+ 764us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1209 08:58:25.258017 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
1209 08:58:25.259511 (+ 1494us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1209 08:58:25.261108 (+ 1597us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1209 08:58:25.261113 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1209 08:58:25.261122 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
1209 08:58:25.261358 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1209 08:58:25.261549 (+ 191us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1209 08:58:25.261551 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1209 08:58:25.261553 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1209 08:58:25.261632 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1209 08:58:25.261962 (+ 330us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1209 08:58:25.261967 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1209 08:58:25.262199 (+ 232us) client_negotiation.cc:769] Sending connection context
1209 08:58:25.262372 (+ 173us) client_negotiation.cc:240] Negotiation successful
1209 08:58:25.262584 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":335,"thread_start_us":137,"threads_started":1}
I20251209 08:58:25.264359 25310 heartbeater.cc:344] Connected to a master server at 127.20.49.126:34643
I20251209 08:58:25.264608 25310 heartbeater.cc:461] Registering TS with master...
I20251209 08:58:25.265134 25310 heartbeater.cc:507] Master 127.20.49.126:34643 requested a full tablet report, sending...
I20251209 08:58:25.266156 24845 ts_manager.cc:194] Registered new tserver with Master: 39e3edd9650845b789b10fe4ba685548 (127.20.49.67:42097)
I20251209 08:58:25.266706 24845 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.49.67@KRBTEST.COM'} at 127.20.49.67:40823
I20251209 08:58:25.269582 20677 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
Dec 09 08:58:25 dist-test-slave-8w6c krb5kdc[24797](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765270705, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Dec 09 08:58:25 dist-test-slave-8w6c krb5kdc[24797](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765270705, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.49.65@KRBTEST.COM
I20251209 08:58:25.294224 25317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:58:25.285539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37234 (local address 127.20.49.65:38891)
1209 08:58:25.285847 (+ 308us) server_negotiation.cc:207] Beginning negotiation
1209 08:58:25.285853 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1209 08:58:25.285879 (+ 26us) server_negotiation.cc:408] Connection header received
1209 08:58:25.286021 (+ 142us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1209 08:58:25.286026 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1209 08:58:25.286212 (+ 186us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1209 08:58:25.286344 (+ 132us) server_negotiation.cc:227] Negotiated authn=SASL
1209 08:58:25.287250 (+ 906us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:58:25.287934 (+ 684us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1209 08:58:25.288580 (+ 646us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:58:25.288752 (+ 172us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:58:25.290883 (+ 2131us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1209 08:58:25.290907 (+ 24us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1209 08:58:25.290919 (+ 12us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1209 08:58:25.290945 (+ 26us) server_negotiation.cc:923] calling sasl_server_start()
1209 08:58:25.292740 (+ 1795us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1209 08:58:25.293108 (+ 368us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1209 08:58:25.293113 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1209 08:58:25.293114 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1209 08:58:25.293179 (+ 65us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1209 08:58:25.293348 (+ 169us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1209 08:58:25.293350 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1209 08:58:25.293351 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1209 08:58:25.293521 (+ 170us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1209 08:58:25.293667 (+ 146us) server_negotiation.cc:1036] Waiting for connection context
1209 08:58:25.293858 (+ 191us) server_negotiation.cc:300] Negotiation successful
1209 08:58:25.294003 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":206,"thread_start_us":76,"threads_started":1}
Dec 09 08:58:25 dist-test-slave-8w6c krb5kdc[24797](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765270705, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.49.126@KRBTEST.COM
I20251209 08:58:25.303913 24884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:58:25.296816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58824 (local address 127.20.49.126:34643)
1209 08:58:25.296934 (+ 118us) server_negotiation.cc:207] Beginning negotiation
1209 08:58:25.296938 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1209 08:58:25.296951 (+ 13us) server_negotiation.cc:408] Connection header received
1209 08:58:25.297053 (+ 102us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1209 08:58:25.297055 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1209 08:58:25.297100 (+ 45us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1209 08:58:25.297178 (+ 78us) server_negotiation.cc:227] Negotiated authn=SASL
1209 08:58:25.297833 (+ 655us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:58:25.298324 (+ 491us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1209 08:58:25.298868 (+ 544us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:58:25.298987 (+ 119us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:58:25.301053 (+ 2066us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1209 08:58:25.301074 (+ 21us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1209 08:58:25.301077 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1209 08:58:25.301104 (+ 27us) server_negotiation.cc:923] calling sasl_server_start()
1209 08:58:25.302598 (+ 1494us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1209 08:58:25.302975 (+ 377us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1209 08:58:25.302978 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1209 08:58:25.302979 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1209 08:58:25.303016 (+ 37us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1209 08:58:25.303191 (+ 175us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1209 08:58:25.303194 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1209 08:58:25.303195 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1209 08:58:25.303359 (+ 164us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1209 08:58:25.303458 (+ 99us) server_negotiation.cc:1036] Waiting for connection context
1209 08:58:25.303622 (+ 164us) server_negotiation.cc:300] Negotiation successful
1209 08:58:25.303718 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
W20251209 08:58:25.304307 24845 server_base.cc:1129] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58824
I20251209 08:58:25.304965 20677 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task8XuDZV/build/debug/bin/kudu with pid 24909
I20251209 08:58:25.311282 20677 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task8XuDZV/build/debug/bin/kudu with pid 25045
I20251209 08:58:25.317373 20677 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task8XuDZV/build/debug/bin/kudu with pid 25181
I20251209 08:58:25.322999 20677 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task8XuDZV/build/debug/bin/kudu with pid 24813
2025-12-09T08:58:25Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (3159 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765270668420570-20677-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Dec 09 08:58:25 dist-test-slave-8w6c krb5kdc[25322](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Dec 09 08:58:25 dist-test-slave-8w6c krb5kdc[25322](info): set up 2 sockets
Dec 09 08:58:25 dist-test-slave-8w6c krb5kdc[25322](info): commencing operation
krb5kdc: starting...
W20251209 08:58:27.330412 20677 mini_kdc.cc:121] Time spent starting KDC: real 1.984s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765270668420570-20677-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765270668420570-20677-0/krb5kdc/test-user.keytab.
Dec 09 08:58:27 dist-test-slave-8w6c krb5kdc[25322](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765270707, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-12-09T08:58:27Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-12-09T08:58:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.49.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.49.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.49.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765270668420570-20677-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.20.49.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765270668420570-20677-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.20.49.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.20.49.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.20.49.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765270668420570-20677-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.20.49.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765270668420570-20677-0/krb5kdc/kudu.keytab.
I20251209 08:58:27.475350 20677 external_mini_cluster.cc:1366] Running /tmp/dist-test-task8XuDZV/build/debug/bin/kudu
/tmp/dist-test-task8XuDZV/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765270668420570-20677-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765270668420570-20677-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765270668420570-20677-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765270668420570-20677-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.49.126:35043
--webserver_interface=127.20.49.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765270668420570-20677-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.49.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.20.49.84:34197
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.49.126:35043
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765270668420570-20677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765270668420570-20677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765270668420570-20677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251209 08:58:27.586432 25338 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251209 08:58:27.586675 25338 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251209 08:58:27.586735 25338 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251209 08:58:27.589965 25338 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251209 08:58:27.590031 25338 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251209 08:58:27.590055 25338 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251209 08:58:27.590073 25338 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251209 08:58:27.590091 25338 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251209 08:58:27.594404 25338 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.49.84:34197
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765270668420570-20677-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765270668420570-20677-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.49.126:35043
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765270668420570-20677-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.49.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.49.126:35043
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765270668420570-20677-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.49.126
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25338
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765270668420570-20677-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 468897755969d686fb08386f42f7835685e7953a
build type DEBUG
built by None at 09 Dec 2025 08:43:15 UTC on 24a791456cd2
build id 9426
I20251209 08:58:27.595499 25338 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251209 08:58:27.596345 25338 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251209 08:58:27.601850 25343 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251209 08:58:27.601879 25346 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251209 08:58:27.602074 25338 server_base.cc:1047] running on GCE node
W20251209 08:58:27.601850 25344 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251209 08:58:27.602743 25338 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251209 08:58:27.603659 25338 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251209 08:58:27.604813 25338 hybrid_clock.cc:648] HybridClock initialized: now 1765270707604800 us; error 34 us; skew 500 ppm
Dec 09 08:58:27 dist-test-slave-8w6c krb5kdc[25322](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765270707, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.49.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251209 08:58:27.607445 25338 init.cc:377] Logged in from keytab as kudu/127.20.49.126@KRBTEST.COM (short username kudu)
I20251209 08:58:27.608124 25338 fs_manager.cc:362] Metadata directory not provided
I20251209 08:58:27.608188 25338 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251209 08:58:27.608347 25338 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251209 08:58:27.610116 25338 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765270668420570-20677-0/minicluster-data/master-0/data/instance:
uuid: "091f13f46e6444a197412a75e5e612eb"
format_stamp: "Formatted at 2025-12-09 08:58:27 on dist-test-slave-8w6c"
server_key: "aadbe97f0bef7915f6a048a1bc3a6c59"
server_key_iv: "fcb5bab7f5104c915eda3799f567565b"
server_key_version: "encryptionkey@0"
I20251209 08:58:27.610558 25338 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765270668420570-20677-0/minicluster-data/master-0/wal/instance:
uuid: "091f13f46e6444a197412a75e5e612eb"
format_stamp: "Formatted at 2025-12-09 08:58:27 on dist-test-slave-8w6c"
server_key: "aadbe97f0bef7915f6a048a1bc3a6c59"
server_key_iv: "fcb5bab7f5104c915eda3799f567565b"
server_key_version: "encryptionkey@0"
I20251209 08:58:27.613919 25338 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20251209 08:58:27.618052 25352 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251209 08:58:27.619115 25338 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20251209 08:58:27.619231 25338 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765270668420570-20677-0/minicluster-data/master-0/data,/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765270668420570-20677-0/minicluster-data/master-0/wal
uuid: "091f13f46e6444a197412a75e5e612eb"
format_stamp: "Formatted at 2025-12-09 08:58:27 on dist-test-slave-8w6c"
server_key: "aadbe97f0bef7915f6a048a1bc3a6c59"
server_key_iv: "fcb5bab7f5104c915eda3799f567565b"
server_key_version: "encryptionkey@0"
I20251209 08:58:27.619343 25338 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765270668420570-20677-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765270668420570-20677-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765270668420570-20677-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251209 08:58:27.643384 25338 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251209 08:58:27.647059 25338 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251209 08:58:27.647275 25338 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251209 08:58:27.654834 25404 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.49.126:35043 every 8 connection(s)
I20251209 08:58:27.654842 25338 rpc_server.cc:307] RPC server started. Bound to: 127.20.49.126:35043
I20251209 08:58:27.655543 25338 server_base.cc:1179] Dumped server information to /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765270668420570-20677-0/minicluster-data/master-0/data/info.pb
I20251209 08:58:27.658386 25405 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251209 08:58:27.662051 20677 external_mini_cluster.cc:1428] Started /tmp/dist-test-task8XuDZV/build/debug/bin/kudu as pid 25338
I20251209 08:58:27.662170 20677 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765270668420570-20677-0/minicluster-data/master-0/wal/instance
I20251209 08:58:27.662411 20677 external_mini_cluster.cc:1468] Setting key 80f1c35521c5533fdc8a628b96104673
I20251209 08:58:27.664170 25405 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 091f13f46e6444a197412a75e5e612eb: Bootstrap starting.
I20251209 08:58:27.666723 25405 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 091f13f46e6444a197412a75e5e612eb: Neither blocks nor log segments found. Creating new log.
I20251209 08:58:27.667553 25405 log.cc:826] T 00000000000000000000000000000000 P 091f13f46e6444a197412a75e5e612eb: Log is configured to *not* fsync() on all Append() calls
I20251209 08:58:27.669607 25405 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 091f13f46e6444a197412a75e5e612eb: No bootstrap required, opened a new log
Dec 09 08:58:27 dist-test-slave-8w6c krb5kdc[25322](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765270707, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.49.126@KRBTEST.COM
I20251209 08:58:27.672780 25405 raft_consensus.cc:359] T 00000000000000000000000000000000 P 091f13f46e6444a197412a75e5e612eb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "091f13f46e6444a197412a75e5e612eb" member_type: VOTER last_known_addr { host: "127.20.49.126" port: 35043 } }
I20251209 08:58:27.673027 25405 raft_consensus.cc:385] T 00000000000000000000000000000000 P 091f13f46e6444a197412a75e5e612eb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251209 08:58:27.673091 25405 raft_consensus.cc:740] T 00000000000000000000000000000000 P 091f13f46e6444a197412a75e5e612eb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 091f13f46e6444a197412a75e5e612eb, State: Initialized, Role: FOLLOWER
I20251209 08:58:27.673538 25405 consensus_queue.cc:260] T 00000000000000000000000000000000 P 091f13f46e6444a197412a75e5e612eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "091f13f46e6444a197412a75e5e612eb" member_type: VOTER last_known_addr { host: "127.20.49.126" port: 35043 } }
I20251209 08:58:27.673707 25405 raft_consensus.cc:399] T 00000000000000000000000000000000 P 091f13f46e6444a197412a75e5e612eb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251209 08:58:27.673787 25405 raft_consensus.cc:493] T 00000000000000000000000000000000 P 091f13f46e6444a197412a75e5e612eb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251209 08:58:27.673884 25405 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 091f13f46e6444a197412a75e5e612eb [term 0 FOLLOWER]: Advancing to term 1
I20251209 08:58:27.674789 25405 raft_consensus.cc:515] T 00000000000000000000000000000000 P 091f13f46e6444a197412a75e5e612eb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "091f13f46e6444a197412a75e5e612eb" member_type: VOTER last_known_addr { host: "127.20.49.126" port: 35043 } }
I20251209 08:58:27.675191 25405 leader_election.cc:304] T 00000000000000000000000000000000 P 091f13f46e6444a197412a75e5e612eb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 091f13f46e6444a197412a75e5e612eb; no voters:
I20251209 08:58:27.675495 25405 leader_election.cc:290] T 00000000000000000000000000000000 P 091f13f46e6444a197412a75e5e612eb [CANDIDATE]: Term 1 election: Requested vote from peers
I20251209 08:58:27.675765 25408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:58:27.663687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52736 (local address 127.20.49.126:35043)
1209 08:58:27.664104 (+ 417us) server_negotiation.cc:207] Beginning negotiation
1209 08:58:27.664116 (+ 12us) server_negotiation.cc:400] Waiting for connection header
1209 08:58:27.664164 (+ 48us) server_negotiation.cc:408] Connection header received
1209 08:58:27.664803 (+ 639us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1209 08:58:27.664826 (+ 23us) server_negotiation.cc:462] Received NEGOTIATE request from client
1209 08:58:27.665195 (+ 369us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1209 08:58:27.665580 (+ 385us) server_negotiation.cc:227] Negotiated authn=SASL
1209 08:58:27.666493 (+ 913us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:58:27.667645 (+ 1152us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1209 08:58:27.668493 (+ 848us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:58:27.668762 (+ 269us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:58:27.670882 (+ 2120us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1209 08:58:27.670905 (+ 23us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1209 08:58:27.670919 (+ 14us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1209 08:58:27.670961 (+ 42us) server_negotiation.cc:923] calling sasl_server_start()
1209 08:58:27.673050 (+ 2089us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1209 08:58:27.673530 (+ 480us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1209 08:58:27.673535 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1209 08:58:27.673540 (+ 5us) server_negotiation.cc:968] calling sasl_server_step()
1209 08:58:27.673635 (+ 95us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1209 08:58:27.674032 (+ 397us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1209 08:58:27.674035 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1209 08:58:27.674036 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1209 08:58:27.674375 (+ 339us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1209 08:58:27.674541 (+ 166us) server_negotiation.cc:1036] Waiting for connection context
1209 08:58:27.674824 (+ 283us) server_negotiation.cc:300] Negotiation successful
1209 08:58:27.675084 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":276,"thread_start_us":115,"threads_started":1}
I20251209 08:58:27.675818 25410 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 091f13f46e6444a197412a75e5e612eb [term 1 FOLLOWER]: Leader election won for term 1
I20251209 08:58:27.675985 25410 raft_consensus.cc:697] T 00000000000000000000000000000000 P 091f13f46e6444a197412a75e5e612eb [term 1 LEADER]: Becoming Leader. State: Replica: 091f13f46e6444a197412a75e5e612eb, State: Running, Role: LEADER
I20251209 08:58:27.676271 25410 consensus_queue.cc:237] T 00000000000000000000000000000000 P 091f13f46e6444a197412a75e5e612eb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "091f13f46e6444a197412a75e5e612eb" member_type: VOTER last_known_addr { host: "127.20.49.126" port: 35043 } }
I20251209 08:58:27.676736 25405 sys_catalog.cc:565] T 00000000000000000000000000000000 P 091f13f46e6444a197412a75e5e612eb [sys.catalog]: configured and running, proceeding with master startup.
I20251209 08:58:27.677412 25411 sys_catalog.cc:455] T 00000000000000000000000000000000 P 091f13f46e6444a197412a75e5e612eb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "091f13f46e6444a197412a75e5e612eb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "091f13f46e6444a197412a75e5e612eb" member_type: VOTER last_known_addr { host: "127.20.49.126" port: 35043 } } }
I20251209 08:58:27.677532 25411 sys_catalog.cc:458] T 00000000000000000000000000000000 P 091f13f46e6444a197412a75e5e612eb [sys.catalog]: This master's current role is: LEADER
I20251209 08:58:27.677802 25410 sys_catalog.cc:455] T 00000000000000000000000000000000 P 091f13f46e6444a197412a75e5e612eb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 091f13f46e6444a197412a75e5e612eb. Latest consensus state: current_term: 1 leader_uuid: "091f13f46e6444a197412a75e5e612eb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "091f13f46e6444a197412a75e5e612eb" member_type: VOTER last_known_addr { host: "127.20.49.126" port: 35043 } } }
I20251209 08:58:27.677909 25410 sys_catalog.cc:458] T 00000000000000000000000000000000 P 091f13f46e6444a197412a75e5e612eb [sys.catalog]: This master's current role is: LEADER
I20251209 08:58:27.678375 25416 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251209 08:58:27.681324 25416 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251209 08:58:27.686437 25416 catalog_manager.cc:1357] Generated new cluster ID: 4c969b5b06d640db94f4799272252539
I20251209 08:58:27.686509 25416 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251209 08:58:27.705096 25416 catalog_manager.cc:1380] Generated new certificate authority record
I20251209 08:58:27.706180 25416 catalog_manager.cc:1514] Loading token signing keys...
I20251209 08:58:27.721989 25416 catalog_manager.cc:6027] T 00000000000000000000000000000000 P 091f13f46e6444a197412a75e5e612eb: Generated new TSK 0
I20251209 08:58:27.722785 25416 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.20.49.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.49.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.49.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765270668420570-20677-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.20.49.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765270668420570-20677-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.20.49.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.20.49.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.20.49.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765270668420570-20677-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.20.49.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765270668420570-20677-0/krb5kdc/kudu.keytab.
I20251209 08:58:27.781782 20677 external_mini_cluster.cc:1366] Running /tmp/dist-test-task8XuDZV/build/debug/bin/kudu
/tmp/dist-test-task8XuDZV/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765270668420570-20677-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765270668420570-20677-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765270668420570-20677-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765270668420570-20677-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.49.65:0
--local_ip_for_outbound_sockets=127.20.49.65
--webserver_interface=127.20.49.65
--webserver_port=0
--tserver_master_addrs=127.20.49.126:35043
--keytab_file=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765270668420570-20677-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.49.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.20.49.84:34197
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765270668420570-20677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765270668420570-20677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765270668420570-20677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251209 08:58:27.893637 25433 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251209 08:58:27.893965 25433 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251209 08:58:27.894066 25433 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251209 08:58:27.897405 25433 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251209 08:58:27.897485 25433 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251209 08:58:27.897614 25433 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.49.65
I20251209 08:58:27.902057 25433 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.49.84:34197
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765270668420570-20677-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765270668420570-20677-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765270668420570-20677-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.49.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.49.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765270668420570-20677-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.49.65
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.20.49.126:35043
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25433
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.49.65
--log_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765270668420570-20677-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 468897755969d686fb08386f42f7835685e7953a
build type DEBUG
built by None at 09 Dec 2025 08:43:15 UTC on 24a791456cd2
build id 9426
I20251209 08:58:27.903191 25433 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251209 08:58:27.904058 25433 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251209 08:58:27.910526 25439 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251209 08:58:27.910581 25441 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251209 08:58:27.910528 25438 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251209 08:58:27.910776 25433 server_base.cc:1047] running on GCE node
I20251209 08:58:27.911229 25433 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251209 08:58:27.911789 25433 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251209 08:58:27.912992 25433 hybrid_clock.cc:648] HybridClock initialized: now 1765270707912961 us; error 43 us; skew 500 ppm
Dec 09 08:58:27 dist-test-slave-8w6c krb5kdc[25322](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765270707, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.49.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251209 08:58:27.915642 25433 init.cc:377] Logged in from keytab as kudu/127.20.49.65@KRBTEST.COM (short username kudu)
I20251209 08:58:27.916424 25433 fs_manager.cc:362] Metadata directory not provided
I20251209 08:58:27.916492 25433 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251209 08:58:27.916679 25433 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251209 08:58:27.918543 25433 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765270668420570-20677-0/minicluster-data/ts-0/data/instance:
uuid: "136ace646aa240a99c03a01c7db8dc5f"
format_stamp: "Formatted at 2025-12-09 08:58:27 on dist-test-slave-8w6c"
server_key: "9b293af71fbaa5b95c399739a9eaf203"
server_key_iv: "97d7218eba5dca947f323a3593dca114"
server_key_version: "encryptionkey@0"
I20251209 08:58:27.919039 25433 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765270668420570-20677-0/minicluster-data/ts-0/wal/instance:
uuid: "136ace646aa240a99c03a01c7db8dc5f"
format_stamp: "Formatted at 2025-12-09 08:58:27 on dist-test-slave-8w6c"
server_key: "9b293af71fbaa5b95c399739a9eaf203"
server_key_iv: "97d7218eba5dca947f323a3593dca114"
server_key_version: "encryptionkey@0"
I20251209 08:58:27.922565 25433 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20251209 08:58:27.924829 25447 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251209 08:58:27.925925 25433 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20251209 08:58:27.926044 25433 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765270668420570-20677-0/minicluster-data/ts-0/data,/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765270668420570-20677-0/minicluster-data/ts-0/wal
uuid: "136ace646aa240a99c03a01c7db8dc5f"
format_stamp: "Formatted at 2025-12-09 08:58:27 on dist-test-slave-8w6c"
server_key: "9b293af71fbaa5b95c399739a9eaf203"
server_key_iv: "97d7218eba5dca947f323a3593dca114"
server_key_version: "encryptionkey@0"
I20251209 08:58:27.926136 25433 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765270668420570-20677-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765270668420570-20677-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765270668420570-20677-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251209 08:58:27.951802 25433 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251209 08:58:27.954938 25433 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251209 08:58:27.955144 25433 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251209 08:58:27.955739 25433 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251209 08:58:27.956676 25433 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251209 08:58:27.956729 25433 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251209 08:58:27.956794 25433 ts_tablet_manager.cc:616] Registered 0 tablets
I20251209 08:58:27.956825 25433 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251209 08:58:27.966503 25433 rpc_server.cc:307] RPC server started. Bound to: 127.20.49.65:42165
I20251209 08:58:27.966583 25560 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.49.65:42165 every 8 connection(s)
I20251209 08:58:27.967113 25433 server_base.cc:1179] Dumped server information to /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765270668420570-20677-0/minicluster-data/ts-0/data/info.pb
I20251209 08:58:27.968652 20677 external_mini_cluster.cc:1428] Started /tmp/dist-test-task8XuDZV/build/debug/bin/kudu as pid 25433
I20251209 08:58:27.968765 20677 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765270668420570-20677-0/minicluster-data/ts-0/wal/instance
I20251209 08:58:27.969059 20677 external_mini_cluster.cc:1468] Setting key b10310dd35908f937613bd1383c0d829
Dec 09 08:58:27 dist-test-slave-8w6c krb5kdc[25322](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765270707, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.49.65@KRBTEST.COM for kudu/127.20.49.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.20.49.66@KRBTEST.COM; defaulting to no policy
I20251209 08:58:27.980768 25408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:58:27.968949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.49.65:53821 (local address 127.20.49.126:35043)
1209 08:58:27.969150 (+ 201us) server_negotiation.cc:207] Beginning negotiation
1209 08:58:27.969155 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1209 08:58:27.969919 (+ 764us) server_negotiation.cc:408] Connection header received
1209 08:58:27.970923 (+ 1004us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1209 08:58:27.970926 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1209 08:58:27.970973 (+ 47us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1209 08:58:27.971043 (+ 70us) server_negotiation.cc:227] Negotiated authn=SASL
1209 08:58:27.973276 (+ 2233us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:58:27.973821 (+ 545us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1209 08:58:27.974479 (+ 658us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:58:27.974652 (+ 173us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:58:27.977150 (+ 2498us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1209 08:58:27.977191 (+ 41us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1209 08:58:27.977194 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1209 08:58:27.977232 (+ 38us) server_negotiation.cc:923] calling sasl_server_start()
1209 08:58:27.978811 (+ 1579us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1209 08:58:27.979313 (+ 502us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1209 08:58:27.979316 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1209 08:58:27.979318 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1209 08:58:27.979362 (+ 44us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1209 08:58:27.979634 (+ 272us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1209 08:58:27.979638 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1209 08:58:27.979640 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1209 08:58:27.979795 (+ 155us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1209 08:58:27.979951 (+ 156us) server_negotiation.cc:1036] Waiting for connection context
1209 08:58:27.980418 (+ 467us) server_negotiation.cc:300] Negotiation successful
1209 08:58:27.980565 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.49.66@KRBTEST.COM" created.
I20251209 08:58:27.981433 25563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:58:27.969252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.49.126:35043 (local address 127.20.49.65:53821)
1209 08:58:27.969773 (+ 521us) negotiation.cc:107] Waiting for socket to connect
1209 08:58:27.969821 (+ 48us) client_negotiation.cc:174] Beginning negotiation
1209 08:58:27.970707 (+ 886us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1209 08:58:27.971215 (+ 508us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1209 08:58:27.971229 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
1209 08:58:27.971819 (+ 590us) client_negotiation.cc:189] Negotiated authn=SASL
1209 08:58:27.973113 (+ 1294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:58:27.973127 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:58:27.973940 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1209 08:58:27.973944 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1209 08:58:27.974364 (+ 420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:58:27.974371 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:58:27.974571 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:58:27.975285 (+ 714us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1209 08:58:27.975310 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
1209 08:58:27.977000 (+ 1690us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1209 08:58:27.978931 (+ 1931us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1209 08:58:27.978937 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1209 08:58:27.978951 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
1209 08:58:27.979206 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1209 08:58:27.979448 (+ 242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1209 08:58:27.979453 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1209 08:58:27.979455 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1209 08:58:27.979546 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1209 08:58:27.979925 (+ 379us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1209 08:58:27.979932 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1209 08:58:27.980177 (+ 245us) client_negotiation.cc:769] Sending connection context
1209 08:58:27.980372 (+ 195us) client_negotiation.cc:240] Negotiation successful
1209 08:58:27.980646 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":286,"thread_start_us":137,"threads_started":1}
I20251209 08:58:27.982646 25561 heartbeater.cc:344] Connected to a master server at 127.20.49.126:35043
I20251209 08:58:27.982894 25561 heartbeater.cc:461] Registering TS with master...
I20251209 08:58:27.983392 25561 heartbeater.cc:507] Master 127.20.49.126:35043 requested a full tablet report, sending...
I20251209 08:58:27.984829 25369 ts_manager.cc:194] Registered new tserver with Master: 136ace646aa240a99c03a01c7db8dc5f (127.20.49.65:42165)
I20251209 08:58:27.986044 25369 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.49.65@KRBTEST.COM'} at 127.20.49.65:53821
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.49.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765270668420570-20677-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.20.49.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765270668420570-20677-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.20.49.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.20.49.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.20.49.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765270668420570-20677-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.20.49.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765270668420570-20677-0/krb5kdc/kudu.keytab.
I20251209 08:58:28.020295 20677 external_mini_cluster.cc:1366] Running /tmp/dist-test-task8XuDZV/build/debug/bin/kudu
/tmp/dist-test-task8XuDZV/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765270668420570-20677-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765270668420570-20677-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765270668420570-20677-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765270668420570-20677-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.49.66:0
--local_ip_for_outbound_sockets=127.20.49.66
--webserver_interface=127.20.49.66
--webserver_port=0
--tserver_master_addrs=127.20.49.126:35043
--keytab_file=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765270668420570-20677-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.49.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.20.49.84:34197
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765270668420570-20677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765270668420570-20677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765270668420570-20677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251209 08:58:28.131873 25568 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251209 08:58:28.132095 25568 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251209 08:58:28.132155 25568 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251209 08:58:28.135345 25568 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251209 08:58:28.135408 25568 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251209 08:58:28.135489 25568 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.49.66
I20251209 08:58:28.139685 25568 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.49.84:34197
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765270668420570-20677-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765270668420570-20677-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765270668420570-20677-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.49.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.49.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765270668420570-20677-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.49.66
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.20.49.126:35043
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25568
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.49.66
--log_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765270668420570-20677-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 468897755969d686fb08386f42f7835685e7953a
build type DEBUG
built by None at 09 Dec 2025 08:43:15 UTC on 24a791456cd2
build id 9426
I20251209 08:58:28.140753 25568 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251209 08:58:28.141597 25568 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251209 08:58:28.148028 25573 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251209 08:58:28.148020 25576 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251209 08:58:28.148020 25574 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251209 08:58:28.148384 25568 server_base.cc:1047] running on GCE node
I20251209 08:58:28.148794 25568 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251209 08:58:28.149396 25568 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251209 08:58:28.150547 25568 hybrid_clock.cc:648] HybridClock initialized: now 1765270708150537 us; error 31 us; skew 500 ppm
Dec 09 08:58:28 dist-test-slave-8w6c krb5kdc[25322](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765270708, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.49.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251209 08:58:28.153237 25568 init.cc:377] Logged in from keytab as kudu/127.20.49.66@KRBTEST.COM (short username kudu)
I20251209 08:58:28.154006 25568 fs_manager.cc:362] Metadata directory not provided
I20251209 08:58:28.154073 25568 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251209 08:58:28.154258 25568 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251209 08:58:28.155992 25568 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765270668420570-20677-0/minicluster-data/ts-1/data/instance:
uuid: "22c6d3fe795545ff91525c965e0a75b4"
format_stamp: "Formatted at 2025-12-09 08:58:28 on dist-test-slave-8w6c"
server_key: "8bbf2e28cc9936fa11288d2cecde945d"
server_key_iv: "1adbd81575507952a45f56b56f9a28d1"
server_key_version: "encryptionkey@0"
I20251209 08:58:28.156479 25568 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765270668420570-20677-0/minicluster-data/ts-1/wal/instance:
uuid: "22c6d3fe795545ff91525c965e0a75b4"
format_stamp: "Formatted at 2025-12-09 08:58:28 on dist-test-slave-8w6c"
server_key: "8bbf2e28cc9936fa11288d2cecde945d"
server_key_iv: "1adbd81575507952a45f56b56f9a28d1"
server_key_version: "encryptionkey@0"
I20251209 08:58:28.160308 25568 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20251209 08:58:28.162586 25582 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251209 08:58:28.163589 25568 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20251209 08:58:28.163704 25568 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765270668420570-20677-0/minicluster-data/ts-1/data,/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765270668420570-20677-0/minicluster-data/ts-1/wal
uuid: "22c6d3fe795545ff91525c965e0a75b4"
format_stamp: "Formatted at 2025-12-09 08:58:28 on dist-test-slave-8w6c"
server_key: "8bbf2e28cc9936fa11288d2cecde945d"
server_key_iv: "1adbd81575507952a45f56b56f9a28d1"
server_key_version: "encryptionkey@0"
I20251209 08:58:28.163797 25568 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765270668420570-20677-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765270668420570-20677-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765270668420570-20677-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251209 08:58:28.197016 25568 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251209 08:58:28.199877 25568 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251209 08:58:28.200073 25568 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251209 08:58:28.200634 25568 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251209 08:58:28.201586 25568 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251209 08:58:28.201638 25568 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251209 08:58:28.201720 25568 ts_tablet_manager.cc:616] Registered 0 tablets
I20251209 08:58:28.201756 25568 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251209 08:58:28.210813 25568 rpc_server.cc:307] RPC server started. Bound to: 127.20.49.66:45461
I20251209 08:58:28.210852 25695 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.49.66:45461 every 8 connection(s)
I20251209 08:58:28.211372 25568 server_base.cc:1179] Dumped server information to /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765270668420570-20677-0/minicluster-data/ts-1/data/info.pb
I20251209 08:58:28.217420 20677 external_mini_cluster.cc:1428] Started /tmp/dist-test-task8XuDZV/build/debug/bin/kudu as pid 25568
I20251209 08:58:28.217526 20677 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765270668420570-20677-0/minicluster-data/ts-1/wal/instance
I20251209 08:58:28.217837 20677 external_mini_cluster.cc:1468] Setting key a1950402e6b31cd03b02a706c6f4be77
Dec 09 08:58:28 dist-test-slave-8w6c krb5kdc[25322](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765270708, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.49.66@KRBTEST.COM for kudu/127.20.49.126@KRBTEST.COM
I20251209 08:58:28.225054 25408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:58:28.213082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.49.66:36859 (local address 127.20.49.126:35043)
1209 08:58:28.213260 (+ 178us) server_negotiation.cc:207] Beginning negotiation
1209 08:58:28.213264 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1209 08:58:28.213928 (+ 664us) server_negotiation.cc:408] Connection header received
1209 08:58:28.214843 (+ 915us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1209 08:58:28.214847 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1209 08:58:28.214915 (+ 68us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1209 08:58:28.214999 (+ 84us) server_negotiation.cc:227] Negotiated authn=SASL
1209 08:58:28.216711 (+ 1712us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:58:28.217252 (+ 541us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1209 08:58:28.218059 (+ 807us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:58:28.218256 (+ 197us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:58:28.220940 (+ 2684us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1209 08:58:28.220956 (+ 16us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1209 08:58:28.220958 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1209 08:58:28.220987 (+ 29us) server_negotiation.cc:923] calling sasl_server_start()
1209 08:58:28.222704 (+ 1717us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1209 08:58:28.223354 (+ 650us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1209 08:58:28.223356 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1209 08:58:28.223358 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1209 08:58:28.223397 (+ 39us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1209 08:58:28.223753 (+ 356us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1209 08:58:28.223756 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1209 08:58:28.223757 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1209 08:58:28.223911 (+ 154us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1209 08:58:28.223995 (+ 84us) server_negotiation.cc:1036] Waiting for connection context
1209 08:58:28.224643 (+ 648us) server_negotiation.cc:300] Negotiation successful
1209 08:58:28.224807 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20251209 08:58:28.225606 25698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:58:28.213323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.49.126:35043 (local address 127.20.49.66:36859)
1209 08:58:28.213772 (+ 449us) negotiation.cc:107] Waiting for socket to connect
1209 08:58:28.213809 (+ 37us) client_negotiation.cc:174] Beginning negotiation
1209 08:58:28.214615 (+ 806us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1209 08:58:28.215193 (+ 578us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1209 08:58:28.215202 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
1209 08:58:28.215610 (+ 408us) client_negotiation.cc:189] Negotiated authn=SASL
1209 08:58:28.216550 (+ 940us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:58:28.216563 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:58:28.217399 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1209 08:58:28.217403 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1209 08:58:28.217936 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:58:28.217946 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:58:28.218123 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:58:28.218828 (+ 705us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1209 08:58:28.218861 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
1209 08:58:28.220730 (+ 1869us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1209 08:58:28.222832 (+ 2102us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1209 08:58:28.222842 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1209 08:58:28.222859 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
1209 08:58:28.223250 (+ 391us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1209 08:58:28.223489 (+ 239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1209 08:58:28.223496 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1209 08:58:28.223501 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
1209 08:58:28.223648 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1209 08:58:28.224020 (+ 372us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1209 08:58:28.224029 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1209 08:58:28.224364 (+ 335us) client_negotiation.cc:769] Sending connection context
1209 08:58:28.224570 (+ 206us) client_negotiation.cc:240] Negotiation successful
1209 08:58:28.224812 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":105,"threads_started":1}
I20251209 08:58:28.226897 25696 heartbeater.cc:344] Connected to a master server at 127.20.49.126:35043
I20251209 08:58:28.227175 25696 heartbeater.cc:461] Registering TS with master...
I20251209 08:58:28.227679 25696 heartbeater.cc:507] Master 127.20.49.126:35043 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.20.49.67@KRBTEST.COM; defaulting to no policy
I20251209 08:58:28.228792 25369 ts_manager.cc:194] Registered new tserver with Master: 22c6d3fe795545ff91525c965e0a75b4 (127.20.49.66:45461)
I20251209 08:58:28.229470 25369 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.49.66@KRBTEST.COM'} at 127.20.49.66:36859
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.49.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.49.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765270668420570-20677-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.20.49.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765270668420570-20677-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.20.49.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.20.49.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.20.49.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765270668420570-20677-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.20.49.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765270668420570-20677-0/krb5kdc/kudu.keytab.
I20251209 08:58:28.268817 20677 external_mini_cluster.cc:1366] Running /tmp/dist-test-task8XuDZV/build/debug/bin/kudu
/tmp/dist-test-task8XuDZV/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765270668420570-20677-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765270668420570-20677-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765270668420570-20677-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765270668420570-20677-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.49.67:0
--local_ip_for_outbound_sockets=127.20.49.67
--webserver_interface=127.20.49.67
--webserver_port=0
--tserver_master_addrs=127.20.49.126:35043
--keytab_file=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765270668420570-20677-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.49.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.20.49.84:34197
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765270668420570-20677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765270668420570-20677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765270668420570-20677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251209 08:58:28.382050 25703 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251209 08:58:28.382298 25703 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251209 08:58:28.382392 25703 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251209 08:58:28.385758 25703 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251209 08:58:28.385826 25703 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251209 08:58:28.385948 25703 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.49.67
I20251209 08:58:28.390097 25703 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.49.84:34197
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765270668420570-20677-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765270668420570-20677-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765270668420570-20677-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.49.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.49.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765270668420570-20677-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.49.67
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.20.49.126:35043
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25703
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.49.67
--log_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765270668420570-20677-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 468897755969d686fb08386f42f7835685e7953a
build type DEBUG
built by None at 09 Dec 2025 08:43:15 UTC on 24a791456cd2
build id 9426
I20251209 08:58:28.391224 25703 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251209 08:58:28.392095 25703 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251209 08:58:28.398260 25709 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251209 08:58:28.398260 25711 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251209 08:58:28.398285 25708 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251209 08:58:28.398561 25703 server_base.cc:1047] running on GCE node
I20251209 08:58:28.398896 25703 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251209 08:58:28.399443 25703 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251209 08:58:28.400595 25703 hybrid_clock.cc:648] HybridClock initialized: now 1765270708400571 us; error 40 us; skew 500 ppm
Dec 09 08:58:28 dist-test-slave-8w6c krb5kdc[25322](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765270708, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.49.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251209 08:58:28.403229 25703 init.cc:377] Logged in from keytab as kudu/127.20.49.67@KRBTEST.COM (short username kudu)
I20251209 08:58:28.403908 25703 fs_manager.cc:362] Metadata directory not provided
I20251209 08:58:28.403980 25703 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251209 08:58:28.404179 25703 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251209 08:58:28.405968 25703 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765270668420570-20677-0/minicluster-data/ts-2/data/instance:
uuid: "50c9062fc9574c329c2fad05a6bfc363"
format_stamp: "Formatted at 2025-12-09 08:58:28 on dist-test-slave-8w6c"
server_key: "1fea4eaec039b9732e95e7b2396c9336"
server_key_iv: "857acf1afe0e99bf448c08a1aa8178ee"
server_key_version: "encryptionkey@0"
I20251209 08:58:28.406445 25703 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765270668420570-20677-0/minicluster-data/ts-2/wal/instance:
uuid: "50c9062fc9574c329c2fad05a6bfc363"
format_stamp: "Formatted at 2025-12-09 08:58:28 on dist-test-slave-8w6c"
server_key: "1fea4eaec039b9732e95e7b2396c9336"
server_key_iv: "857acf1afe0e99bf448c08a1aa8178ee"
server_key_version: "encryptionkey@0"
I20251209 08:58:28.409885 25703 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20251209 08:58:28.412194 25717 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251209 08:58:28.413239 25703 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20251209 08:58:28.413352 25703 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765270668420570-20677-0/minicluster-data/ts-2/data,/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765270668420570-20677-0/minicluster-data/ts-2/wal
uuid: "50c9062fc9574c329c2fad05a6bfc363"
format_stamp: "Formatted at 2025-12-09 08:58:28 on dist-test-slave-8w6c"
server_key: "1fea4eaec039b9732e95e7b2396c9336"
server_key_iv: "857acf1afe0e99bf448c08a1aa8178ee"
server_key_version: "encryptionkey@0"
I20251209 08:58:28.413448 25703 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765270668420570-20677-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765270668420570-20677-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765270668420570-20677-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251209 08:58:28.428064 25703 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251209 08:58:28.431048 25703 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251209 08:58:28.431241 25703 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251209 08:58:28.431819 25703 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251209 08:58:28.432758 25703 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251209 08:58:28.432812 25703 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251209 08:58:28.432870 25703 ts_tablet_manager.cc:616] Registered 0 tablets
I20251209 08:58:28.432904 25703 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251209 08:58:28.441817 25703 rpc_server.cc:307] RPC server started. Bound to: 127.20.49.67:41723
I20251209 08:58:28.441854 25830 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.49.67:41723 every 8 connection(s)
I20251209 08:58:28.442386 25703 server_base.cc:1179] Dumped server information to /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765270668420570-20677-0/minicluster-data/ts-2/data/info.pb
I20251209 08:58:28.445032 20677 external_mini_cluster.cc:1428] Started /tmp/dist-test-task8XuDZV/build/debug/bin/kudu as pid 25703
I20251209 08:58:28.445158 20677 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765270668420570-20677-0/minicluster-data/ts-2/wal/instance
I20251209 08:58:28.445425 20677 external_mini_cluster.cc:1468] Setting key 35c06484ea13935904bfcd981346b91c
Dec 09 08:58:28 dist-test-slave-8w6c krb5kdc[25322](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765270708, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.49.67@KRBTEST.COM for kudu/127.20.49.126@KRBTEST.COM
I20251209 08:58:28.454767 25408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:58:28.444006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.49.67:33729 (local address 127.20.49.126:35043)
1209 08:58:28.444185 (+ 179us) server_negotiation.cc:207] Beginning negotiation
1209 08:58:28.444190 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1209 08:58:28.444896 (+ 706us) server_negotiation.cc:408] Connection header received
1209 08:58:28.446028 (+ 1132us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1209 08:58:28.446032 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1209 08:58:28.446091 (+ 59us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1209 08:58:28.446178 (+ 87us) server_negotiation.cc:227] Negotiated authn=SASL
1209 08:58:28.448074 (+ 1896us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:58:28.448573 (+ 499us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1209 08:58:28.449232 (+ 659us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:58:28.449411 (+ 179us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:58:28.451600 (+ 2189us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1209 08:58:28.451617 (+ 17us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1209 08:58:28.451619 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1209 08:58:28.451646 (+ 27us) server_negotiation.cc:923] calling sasl_server_start()
1209 08:58:28.452957 (+ 1311us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1209 08:58:28.453400 (+ 443us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1209 08:58:28.453403 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1209 08:58:28.453404 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1209 08:58:28.453444 (+ 40us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1209 08:58:28.453728 (+ 284us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1209 08:58:28.453735 (+ 7us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1209 08:58:28.453739 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
1209 08:58:28.453953 (+ 214us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1209 08:58:28.454049 (+ 96us) server_negotiation.cc:1036] Waiting for connection context
1209 08:58:28.454525 (+ 476us) server_negotiation.cc:300] Negotiation successful
1209 08:58:28.454633 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20251209 08:58:28.455391 25833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:58:28.444305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.49.126:35043 (local address 127.20.49.67:33729)
1209 08:58:28.444729 (+ 424us) negotiation.cc:107] Waiting for socket to connect
1209 08:58:28.444771 (+ 42us) client_negotiation.cc:174] Beginning negotiation
1209 08:58:28.445806 (+ 1035us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1209 08:58:28.446415 (+ 609us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1209 08:58:28.446429 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
1209 08:58:28.446897 (+ 468us) client_negotiation.cc:189] Negotiated authn=SASL
1209 08:58:28.447924 (+ 1027us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:58:28.447937 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:58:28.448698 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1209 08:58:28.448701 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1209 08:58:28.449118 (+ 417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:58:28.449125 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:58:28.449333 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:58:28.450010 (+ 677us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1209 08:58:28.450033 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
1209 08:58:28.451465 (+ 1432us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1209 08:58:28.453071 (+ 1606us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1209 08:58:28.453075 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1209 08:58:28.453085 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
1209 08:58:28.453316 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1209 08:58:28.453505 (+ 189us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1209 08:58:28.453507 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1209 08:58:28.453508 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1209 08:58:28.453585 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1209 08:58:28.454067 (+ 482us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1209 08:58:28.454072 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1209 08:58:28.454291 (+ 219us) client_negotiation.cc:769] Sending connection context
1209 08:58:28.454467 (+ 176us) client_negotiation.cc:240] Negotiation successful
1209 08:58:28.454679 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":110,"threads_started":1}
I20251209 08:58:28.456485 25831 heartbeater.cc:344] Connected to a master server at 127.20.49.126:35043
I20251209 08:58:28.456745 25831 heartbeater.cc:461] Registering TS with master...
I20251209 08:58:28.457247 25831 heartbeater.cc:507] Master 127.20.49.126:35043 requested a full tablet report, sending...
I20251209 08:58:28.458321 25369 ts_manager.cc:194] Registered new tserver with Master: 50c9062fc9574c329c2fad05a6bfc363 (127.20.49.67:41723)
I20251209 08:58:28.458947 25369 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.49.67@KRBTEST.COM'} at 127.20.49.67:33729
I20251209 08:58:28.459319 20677 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20251209 08:58:28.467319 25408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:58:28.461082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41600 (local address 127.20.49.126:35043)
1209 08:58:28.461205 (+ 123us) server_negotiation.cc:207] Beginning negotiation
1209 08:58:28.461208 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1209 08:58:28.461324 (+ 116us) server_negotiation.cc:408] Connection header received
1209 08:58:28.461420 (+ 96us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1209 08:58:28.461422 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1209 08:58:28.461460 (+ 38us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1209 08:58:28.461529 (+ 69us) server_negotiation.cc:227] Negotiated authn=SASL
1209 08:58:28.462424 (+ 895us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:58:28.462895 (+ 471us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1209 08:58:28.463487 (+ 592us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:58:28.463609 (+ 122us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:58:28.464674 (+ 1065us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1209 08:58:28.464689 (+ 15us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1209 08:58:28.464692 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1209 08:58:28.464713 (+ 21us) server_negotiation.cc:923] calling sasl_server_start()
1209 08:58:28.466154 (+ 1441us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1209 08:58:28.466483 (+ 329us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1209 08:58:28.466485 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1209 08:58:28.466487 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1209 08:58:28.466527 (+ 40us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1209 08:58:28.466689 (+ 162us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1209 08:58:28.466691 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1209 08:58:28.466693 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1209 08:58:28.466846 (+ 153us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1209 08:58:28.466940 (+ 94us) server_negotiation.cc:1036] Waiting for connection context
1209 08:58:28.467094 (+ 154us) server_negotiation.cc:300] Negotiation successful
1209 08:58:28.467187 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20251209 08:58:28.469797 25369 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41600:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251209 08:58:28.472194 25369 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251209 08:58:28.485241 25842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:58:28.480426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.49.65:42165 (local address 127.0.0.1:33602)
1209 08:58:28.480772 (+ 346us) negotiation.cc:107] Waiting for socket to connect
1209 08:58:28.480821 (+ 49us) client_negotiation.cc:174] Beginning negotiation
1209 08:58:28.480966 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1209 08:58:28.481808 (+ 842us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1209 08:58:28.481815 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
1209 08:58:28.481842 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1209 08:58:28.482164 (+ 322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:58:28.482174 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:58:28.483700 (+ 1526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1209 08:58:28.483703 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1209 08:58:28.484683 (+ 980us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:58:28.484689 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:58:28.484797 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:58:28.484854 (+ 57us) client_negotiation.cc:769] Sending connection context
1209 08:58:28.484947 (+ 93us) client_negotiation.cc:240] Negotiation successful
1209 08:58:28.485041 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":90,"threads_started":1}
I20251209 08:58:28.485674 25845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:58:28.480933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.49.66:45461 (local address 127.0.0.1:45644)
1209 08:58:28.481377 (+ 444us) negotiation.cc:107] Waiting for socket to connect
1209 08:58:28.481390 (+ 13us) client_negotiation.cc:174] Beginning negotiation
1209 08:58:28.481468 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1209 08:58:28.482205 (+ 737us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1209 08:58:28.482211 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
1209 08:58:28.482237 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1209 08:58:28.482487 (+ 250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:58:28.482499 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:58:28.483804 (+ 1305us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1209 08:58:28.483807 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1209 08:58:28.484673 (+ 866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:58:28.484684 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:58:28.484796 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:58:28.484854 (+ 58us) client_negotiation.cc:769] Sending connection context
1209 08:58:28.484947 (+ 93us) client_negotiation.cc:240] Negotiation successful
1209 08:58:28.485041 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":373,"thread_start_us":125,"threads_started":1}
I20251209 08:58:28.486147 25846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:58:28.481111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45644 (local address 127.20.49.66:45461)
1209 08:58:28.481767 (+ 656us) server_negotiation.cc:207] Beginning negotiation
1209 08:58:28.481773 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1209 08:58:28.481796 (+ 23us) server_negotiation.cc:408] Connection header received
1209 08:58:28.481878 (+ 82us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1209 08:58:28.481883 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1209 08:58:28.482054 (+ 171us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1209 08:58:28.482202 (+ 148us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1209 08:58:28.482617 (+ 415us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:58:28.483697 (+ 1080us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1209 08:58:28.485143 (+ 1446us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:58:28.485741 (+ 598us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:58:28.485855 (+ 114us) server_negotiation.cc:1036] Waiting for connection context
1209 08:58:28.485919 (+ 64us) server_negotiation.cc:300] Negotiation successful
1209 08:58:28.486017 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":503,"thread_start_us":136,"threads_started":1}
I20251209 08:58:28.486524 25843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:58:28.480603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33602 (local address 127.20.49.65:42165)
1209 08:58:28.481271 (+ 668us) server_negotiation.cc:207] Beginning negotiation
1209 08:58:28.481278 (+ 7us) server_negotiation.cc:400] Waiting for connection header
1209 08:58:28.481304 (+ 26us) server_negotiation.cc:408] Connection header received
1209 08:58:28.481399 (+ 95us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1209 08:58:28.481409 (+ 10us) server_negotiation.cc:462] Received NEGOTIATE request from client
1209 08:58:28.481614 (+ 205us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1209 08:58:28.482320 (+ 706us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1209 08:58:28.482462 (+ 142us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:58:28.483577 (+ 1115us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1209 08:58:28.485612 (+ 2035us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:58:28.486122 (+ 510us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:58:28.486247 (+ 125us) server_negotiation.cc:1036] Waiting for connection context
1209 08:58:28.486316 (+ 69us) server_negotiation.cc:300] Negotiation successful
1209 08:58:28.486407 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":550,"thread_start_us":110,"threads_started":1}
I20251209 08:58:28.488397 25844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:58:28.481144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.49.67:41723 (local address 127.0.0.1:48456)
1209 08:58:28.483827 (+ 2683us) negotiation.cc:107] Waiting for socket to connect
1209 08:58:28.483840 (+ 13us) client_negotiation.cc:174] Beginning negotiation
1209 08:58:28.483932 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1209 08:58:28.484315 (+ 383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1209 08:58:28.484318 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1209 08:58:28.484336 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1209 08:58:28.484574 (+ 238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:58:28.484583 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:58:28.485895 (+ 1312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1209 08:58:28.485900 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1209 08:58:28.486729 (+ 829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:58:28.486738 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:58:28.488111 (+ 1373us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:58:28.488137 (+ 26us) client_negotiation.cc:769] Sending connection context
1209 08:58:28.488210 (+ 73us) client_negotiation.cc:240] Negotiation successful
1209 08:58:28.488284 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2605,"thread_start_us":110,"threads_started":1}
I20251209 08:58:28.488873 25847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:58:28.481534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48456 (local address 127.20.49.67:41723)
1209 08:58:28.482472 (+ 938us) server_negotiation.cc:207] Beginning negotiation
1209 08:58:28.482478 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1209 08:58:28.483929 (+ 1451us) server_negotiation.cc:408] Connection header received
1209 08:58:28.484067 (+ 138us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1209 08:58:28.484071 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1209 08:58:28.484192 (+ 121us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1209 08:58:28.484315 (+ 123us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1209 08:58:28.484706 (+ 391us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:58:28.485760 (+ 1054us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1209 08:58:28.486879 (+ 1119us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:58:28.487403 (+ 524us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:58:28.487503 (+ 100us) server_negotiation.cc:1036] Waiting for connection context
1209 08:58:28.488643 (+ 1140us) server_negotiation.cc:300] Negotiation successful
1209 08:58:28.488741 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":441,"thread_start_us":156,"threads_started":1}
I20251209 08:58:28.489033 25495 tablet_service.cc:1505] Processing CreateTablet for tablet 641ed4e45a614803b9d52eb14ec682de (DEFAULT_TABLE table=test-table [id=10b773589f194049b687004173e22124]), partition=RANGE (key) PARTITION UNBOUNDED
I20251209 08:58:28.489032 25630 tablet_service.cc:1505] Processing CreateTablet for tablet 641ed4e45a614803b9d52eb14ec682de (DEFAULT_TABLE table=test-table [id=10b773589f194049b687004173e22124]), partition=RANGE (key) PARTITION UNBOUNDED
I20251209 08:58:28.490005 25630 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 641ed4e45a614803b9d52eb14ec682de. 1 dirs total, 0 dirs full, 0 dirs failed
I20251209 08:58:28.490022 25495 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 641ed4e45a614803b9d52eb14ec682de. 1 dirs total, 0 dirs full, 0 dirs failed
I20251209 08:58:28.491662 25765 tablet_service.cc:1505] Processing CreateTablet for tablet 641ed4e45a614803b9d52eb14ec682de (DEFAULT_TABLE table=test-table [id=10b773589f194049b687004173e22124]), partition=RANGE (key) PARTITION UNBOUNDED
I20251209 08:58:28.492602 25765 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 641ed4e45a614803b9d52eb14ec682de. 1 dirs total, 0 dirs full, 0 dirs failed
I20251209 08:58:28.495441 25848 tablet_bootstrap.cc:492] T 641ed4e45a614803b9d52eb14ec682de P 22c6d3fe795545ff91525c965e0a75b4: Bootstrap starting.
I20251209 08:58:28.496716 25849 tablet_bootstrap.cc:492] T 641ed4e45a614803b9d52eb14ec682de P 136ace646aa240a99c03a01c7db8dc5f: Bootstrap starting.
I20251209 08:58:28.498011 25848 tablet_bootstrap.cc:654] T 641ed4e45a614803b9d52eb14ec682de P 22c6d3fe795545ff91525c965e0a75b4: Neither blocks nor log segments found. Creating new log.
I20251209 08:58:28.498644 25850 tablet_bootstrap.cc:492] T 641ed4e45a614803b9d52eb14ec682de P 50c9062fc9574c329c2fad05a6bfc363: Bootstrap starting.
I20251209 08:58:28.498956 25849 tablet_bootstrap.cc:654] T 641ed4e45a614803b9d52eb14ec682de P 136ace646aa240a99c03a01c7db8dc5f: Neither blocks nor log segments found. Creating new log.
I20251209 08:58:28.499033 25848 log.cc:826] T 641ed4e45a614803b9d52eb14ec682de P 22c6d3fe795545ff91525c965e0a75b4: Log is configured to *not* fsync() on all Append() calls
I20251209 08:58:28.499719 25849 log.cc:826] T 641ed4e45a614803b9d52eb14ec682de P 136ace646aa240a99c03a01c7db8dc5f: Log is configured to *not* fsync() on all Append() calls
I20251209 08:58:28.500701 25850 tablet_bootstrap.cc:654] T 641ed4e45a614803b9d52eb14ec682de P 50c9062fc9574c329c2fad05a6bfc363: Neither blocks nor log segments found. Creating new log.
I20251209 08:58:28.501216 25848 tablet_bootstrap.cc:492] T 641ed4e45a614803b9d52eb14ec682de P 22c6d3fe795545ff91525c965e0a75b4: No bootstrap required, opened a new log
I20251209 08:58:28.501356 25850 log.cc:826] T 641ed4e45a614803b9d52eb14ec682de P 50c9062fc9574c329c2fad05a6bfc363: Log is configured to *not* fsync() on all Append() calls
I20251209 08:58:28.501415 25848 ts_tablet_manager.cc:1403] T 641ed4e45a614803b9d52eb14ec682de P 22c6d3fe795545ff91525c965e0a75b4: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20251209 08:58:28.501727 25849 tablet_bootstrap.cc:492] T 641ed4e45a614803b9d52eb14ec682de P 136ace646aa240a99c03a01c7db8dc5f: No bootstrap required, opened a new log
I20251209 08:58:28.501902 25849 ts_tablet_manager.cc:1403] T 641ed4e45a614803b9d52eb14ec682de P 136ace646aa240a99c03a01c7db8dc5f: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20251209 08:58:28.503208 25850 tablet_bootstrap.cc:492] T 641ed4e45a614803b9d52eb14ec682de P 50c9062fc9574c329c2fad05a6bfc363: No bootstrap required, opened a new log
I20251209 08:58:28.503381 25850 ts_tablet_manager.cc:1403] T 641ed4e45a614803b9d52eb14ec682de P 50c9062fc9574c329c2fad05a6bfc363: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20251209 08:58:28.505203 25849 raft_consensus.cc:359] T 641ed4e45a614803b9d52eb14ec682de P 136ace646aa240a99c03a01c7db8dc5f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "136ace646aa240a99c03a01c7db8dc5f" member_type: VOTER last_known_addr { host: "127.20.49.65" port: 42165 } } peers { permanent_uuid: "50c9062fc9574c329c2fad05a6bfc363" member_type: VOTER last_known_addr { host: "127.20.49.67" port: 41723 } } peers { permanent_uuid: "22c6d3fe795545ff91525c965e0a75b4" member_type: VOTER last_known_addr { host: "127.20.49.66" port: 45461 } }
I20251209 08:58:28.505218 25848 raft_consensus.cc:359] T 641ed4e45a614803b9d52eb14ec682de P 22c6d3fe795545ff91525c965e0a75b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "136ace646aa240a99c03a01c7db8dc5f" member_type: VOTER last_known_addr { host: "127.20.49.65" port: 42165 } } peers { permanent_uuid: "50c9062fc9574c329c2fad05a6bfc363" member_type: VOTER last_known_addr { host: "127.20.49.67" port: 41723 } } peers { permanent_uuid: "22c6d3fe795545ff91525c965e0a75b4" member_type: VOTER last_known_addr { host: "127.20.49.66" port: 45461 } }
I20251209 08:58:28.505436 25849 raft_consensus.cc:385] T 641ed4e45a614803b9d52eb14ec682de P 136ace646aa240a99c03a01c7db8dc5f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251209 08:58:28.505436 25848 raft_consensus.cc:385] T 641ed4e45a614803b9d52eb14ec682de P 22c6d3fe795545ff91525c965e0a75b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251209 08:58:28.505496 25848 raft_consensus.cc:740] T 641ed4e45a614803b9d52eb14ec682de P 22c6d3fe795545ff91525c965e0a75b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22c6d3fe795545ff91525c965e0a75b4, State: Initialized, Role: FOLLOWER
I20251209 08:58:28.505503 25849 raft_consensus.cc:740] T 641ed4e45a614803b9d52eb14ec682de P 136ace646aa240a99c03a01c7db8dc5f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 136ace646aa240a99c03a01c7db8dc5f, State: Initialized, Role: FOLLOWER
I20251209 08:58:28.505981 25848 consensus_queue.cc:260] T 641ed4e45a614803b9d52eb14ec682de P 22c6d3fe795545ff91525c965e0a75b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "136ace646aa240a99c03a01c7db8dc5f" member_type: VOTER last_known_addr { host: "127.20.49.65" port: 42165 } } peers { permanent_uuid: "50c9062fc9574c329c2fad05a6bfc363" member_type: VOTER last_known_addr { host: "127.20.49.67" port: 41723 } } peers { permanent_uuid: "22c6d3fe795545ff91525c965e0a75b4" member_type: VOTER last_known_addr { host: "127.20.49.66" port: 45461 } }
I20251209 08:58:28.505981 25849 consensus_queue.cc:260] T 641ed4e45a614803b9d52eb14ec682de P 136ace646aa240a99c03a01c7db8dc5f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "136ace646aa240a99c03a01c7db8dc5f" member_type: VOTER last_known_addr { host: "127.20.49.65" port: 42165 } } peers { permanent_uuid: "50c9062fc9574c329c2fad05a6bfc363" member_type: VOTER last_known_addr { host: "127.20.49.67" port: 41723 } } peers { permanent_uuid: "22c6d3fe795545ff91525c965e0a75b4" member_type: VOTER last_known_addr { host: "127.20.49.66" port: 45461 } }
I20251209 08:58:28.506079 25850 raft_consensus.cc:359] T 641ed4e45a614803b9d52eb14ec682de P 50c9062fc9574c329c2fad05a6bfc363 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "136ace646aa240a99c03a01c7db8dc5f" member_type: VOTER last_known_addr { host: "127.20.49.65" port: 42165 } } peers { permanent_uuid: "50c9062fc9574c329c2fad05a6bfc363" member_type: VOTER last_known_addr { host: "127.20.49.67" port: 41723 } } peers { permanent_uuid: "22c6d3fe795545ff91525c965e0a75b4" member_type: VOTER last_known_addr { host: "127.20.49.66" port: 45461 } }
I20251209 08:58:28.506299 25850 raft_consensus.cc:385] T 641ed4e45a614803b9d52eb14ec682de P 50c9062fc9574c329c2fad05a6bfc363 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251209 08:58:28.506374 25850 raft_consensus.cc:740] T 641ed4e45a614803b9d52eb14ec682de P 50c9062fc9574c329c2fad05a6bfc363 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50c9062fc9574c329c2fad05a6bfc363, State: Initialized, Role: FOLLOWER
I20251209 08:58:28.506768 25850 consensus_queue.cc:260] T 641ed4e45a614803b9d52eb14ec682de P 50c9062fc9574c329c2fad05a6bfc363 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "136ace646aa240a99c03a01c7db8dc5f" member_type: VOTER last_known_addr { host: "127.20.49.65" port: 42165 } } peers { permanent_uuid: "50c9062fc9574c329c2fad05a6bfc363" member_type: VOTER last_known_addr { host: "127.20.49.67" port: 41723 } } peers { permanent_uuid: "22c6d3fe795545ff91525c965e0a75b4" member_type: VOTER last_known_addr { host: "127.20.49.66" port: 45461 } }
I20251209 08:58:28.506892 25848 ts_tablet_manager.cc:1434] T 641ed4e45a614803b9d52eb14ec682de P 22c6d3fe795545ff91525c965e0a75b4: Time spent starting tablet: real 0.005s user 0.003s sys 0.004s
I20251209 08:58:28.506892 25849 ts_tablet_manager.cc:1434] T 641ed4e45a614803b9d52eb14ec682de P 136ace646aa240a99c03a01c7db8dc5f: Time spent starting tablet: real 0.005s user 0.001s sys 0.003s
I20251209 08:58:28.507283 25561 heartbeater.cc:499] Master 127.20.49.126:35043 was elected leader, sending a full tablet report...
I20251209 08:58:28.507283 25696 heartbeater.cc:499] Master 127.20.49.126:35043 was elected leader, sending a full tablet report...
I20251209 08:58:28.507571 25850 ts_tablet_manager.cc:1434] T 641ed4e45a614803b9d52eb14ec682de P 50c9062fc9574c329c2fad05a6bfc363: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20251209 08:58:28.508347 25831 heartbeater.cc:499] Master 127.20.49.126:35043 was elected leader, sending a full tablet report...
I20251209 08:58:28.550421 25855 raft_consensus.cc:493] T 641ed4e45a614803b9d52eb14ec682de P 136ace646aa240a99c03a01c7db8dc5f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251209 08:58:28.550635 25855 raft_consensus.cc:515] T 641ed4e45a614803b9d52eb14ec682de P 136ace646aa240a99c03a01c7db8dc5f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "136ace646aa240a99c03a01c7db8dc5f" member_type: VOTER last_known_addr { host: "127.20.49.65" port: 42165 } } peers { permanent_uuid: "50c9062fc9574c329c2fad05a6bfc363" member_type: VOTER last_known_addr { host: "127.20.49.67" port: 41723 } } peers { permanent_uuid: "22c6d3fe795545ff91525c965e0a75b4" member_type: VOTER last_known_addr { host: "127.20.49.66" port: 45461 } }
I20251209 08:58:28.551882 25855 leader_election.cc:290] T 641ed4e45a614803b9d52eb14ec682de P 136ace646aa240a99c03a01c7db8dc5f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 50c9062fc9574c329c2fad05a6bfc363 (127.20.49.67:41723), 22c6d3fe795545ff91525c965e0a75b4 (127.20.49.66:45461)
I20251209 08:58:28.556190 25858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:58:28.552177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.49.66:45461 (local address 127.20.49.65:36125)
1209 08:58:28.552610 (+ 433us) negotiation.cc:107] Waiting for socket to connect
1209 08:58:28.552627 (+ 17us) client_negotiation.cc:174] Beginning negotiation
1209 08:58:28.552769 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1209 08:58:28.553095 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1209 08:58:28.553100 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
1209 08:58:28.553149 (+ 49us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1209 08:58:28.553662 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:58:28.553669 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:58:28.554672 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1209 08:58:28.554674 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1209 08:58:28.555698 (+ 1024us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:58:28.555708 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:58:28.555844 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:58:28.555866 (+ 22us) client_negotiation.cc:769] Sending connection context
1209 08:58:28.555923 (+ 57us) client_negotiation.cc:240] Negotiation successful
1209 08:58:28.555993 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":345,"spinlock_wait_cycles":384,"thread_start_us":143,"threads_started":1}
I20251209 08:58:28.556190 25857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:58:28.552180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.49.67:41723 (local address 127.20.49.65:60159)
1209 08:58:28.552676 (+ 496us) negotiation.cc:107] Waiting for socket to connect
1209 08:58:28.552690 (+ 14us) client_negotiation.cc:174] Beginning negotiation
1209 08:58:28.552776 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1209 08:58:28.553053 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1209 08:58:28.553058 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
1209 08:58:28.553078 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1209 08:58:28.553475 (+ 397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:58:28.553487 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:58:28.554653 (+ 1166us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1209 08:58:28.554660 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1209 08:58:28.555698 (+ 1038us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:58:28.555708 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:58:28.555841 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:58:28.555866 (+ 25us) client_negotiation.cc:769] Sending connection context
1209 08:58:28.555923 (+ 57us) client_negotiation.cc:240] Negotiation successful
1209 08:58:28.555995 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":411,"mutex_wait_us":56,"spinlock_wait_cycles":10880,"thread_start_us":94,"threads_started":1}
I20251209 08:58:28.556669 25846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:58:28.552304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.49.65:36125 (local address 127.20.49.66:45461)
1209 08:58:28.552427 (+ 123us) server_negotiation.cc:207] Beginning negotiation
1209 08:58:28.552432 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1209 08:58:28.552742 (+ 310us) server_negotiation.cc:408] Connection header received
1209 08:58:28.552893 (+ 151us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1209 08:58:28.552936 (+ 43us) server_negotiation.cc:462] Received NEGOTIATE request from client
1209 08:58:28.552998 (+ 62us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1209 08:58:28.553082 (+ 84us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1209 08:58:28.553823 (+ 741us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:58:28.554566 (+ 743us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1209 08:58:28.555852 (+ 1286us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:58:28.556396 (+ 544us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:58:28.556437 (+ 41us) server_negotiation.cc:1036] Waiting for connection context
1209 08:58:28.556494 (+ 57us) server_negotiation.cc:300] Negotiation successful
1209 08:58:28.556553 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20251209 08:58:28.556684 25847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:58:28.552285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.49.65:60159 (local address 127.20.49.67:41723)
1209 08:58:28.552427 (+ 142us) server_negotiation.cc:207] Beginning negotiation
1209 08:58:28.552432 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1209 08:58:28.552786 (+ 354us) server_negotiation.cc:408] Connection header received
1209 08:58:28.552864 (+ 78us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1209 08:58:28.552868 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1209 08:58:28.552934 (+ 66us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1209 08:58:28.553032 (+ 98us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1209 08:58:28.553649 (+ 617us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:58:28.554521 (+ 872us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1209 08:58:28.555852 (+ 1331us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:58:28.556396 (+ 544us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:58:28.556437 (+ 41us) server_negotiation.cc:1036] Waiting for connection context
1209 08:58:28.556494 (+ 57us) server_negotiation.cc:300] Negotiation successful
1209 08:58:28.556553 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20251209 08:58:28.557327 25650 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "641ed4e45a614803b9d52eb14ec682de" candidate_uuid: "136ace646aa240a99c03a01c7db8dc5f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22c6d3fe795545ff91525c965e0a75b4" is_pre_election: true
I20251209 08:58:28.557376 25785 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "641ed4e45a614803b9d52eb14ec682de" candidate_uuid: "136ace646aa240a99c03a01c7db8dc5f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "50c9062fc9574c329c2fad05a6bfc363" is_pre_election: true
I20251209 08:58:28.557626 25650 raft_consensus.cc:2468] T 641ed4e45a614803b9d52eb14ec682de P 22c6d3fe795545ff91525c965e0a75b4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 136ace646aa240a99c03a01c7db8dc5f in term 0.
I20251209 08:58:28.557641 25785 raft_consensus.cc:2468] T 641ed4e45a614803b9d52eb14ec682de P 50c9062fc9574c329c2fad05a6bfc363 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 136ace646aa240a99c03a01c7db8dc5f in term 0.
I20251209 08:58:28.558269 25448 leader_election.cc:304] T 641ed4e45a614803b9d52eb14ec682de P 136ace646aa240a99c03a01c7db8dc5f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 136ace646aa240a99c03a01c7db8dc5f, 50c9062fc9574c329c2fad05a6bfc363; no voters:
I20251209 08:58:28.558521 25855 raft_consensus.cc:2804] T 641ed4e45a614803b9d52eb14ec682de P 136ace646aa240a99c03a01c7db8dc5f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251209 08:58:28.558610 25855 raft_consensus.cc:493] T 641ed4e45a614803b9d52eb14ec682de P 136ace646aa240a99c03a01c7db8dc5f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251209 08:58:28.558683 25855 raft_consensus.cc:3060] T 641ed4e45a614803b9d52eb14ec682de P 136ace646aa240a99c03a01c7db8dc5f [term 0 FOLLOWER]: Advancing to term 1
I20251209 08:58:28.559677 25855 raft_consensus.cc:515] T 641ed4e45a614803b9d52eb14ec682de P 136ace646aa240a99c03a01c7db8dc5f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "136ace646aa240a99c03a01c7db8dc5f" member_type: VOTER last_known_addr { host: "127.20.49.65" port: 42165 } } peers { permanent_uuid: "50c9062fc9574c329c2fad05a6bfc363" member_type: VOTER last_known_addr { host: "127.20.49.67" port: 41723 } } peers { permanent_uuid: "22c6d3fe795545ff91525c965e0a75b4" member_type: VOTER last_known_addr { host: "127.20.49.66" port: 45461 } }
I20251209 08:58:28.560053 25855 leader_election.cc:290] T 641ed4e45a614803b9d52eb14ec682de P 136ace646aa240a99c03a01c7db8dc5f [CANDIDATE]: Term 1 election: Requested vote from peers 50c9062fc9574c329c2fad05a6bfc363 (127.20.49.67:41723), 22c6d3fe795545ff91525c965e0a75b4 (127.20.49.66:45461)
I20251209 08:58:28.560446 25785 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "641ed4e45a614803b9d52eb14ec682de" candidate_uuid: "136ace646aa240a99c03a01c7db8dc5f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "50c9062fc9574c329c2fad05a6bfc363"
I20251209 08:58:28.560444 25650 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "641ed4e45a614803b9d52eb14ec682de" candidate_uuid: "136ace646aa240a99c03a01c7db8dc5f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22c6d3fe795545ff91525c965e0a75b4"
I20251209 08:58:28.560581 25785 raft_consensus.cc:3060] T 641ed4e45a614803b9d52eb14ec682de P 50c9062fc9574c329c2fad05a6bfc363 [term 0 FOLLOWER]: Advancing to term 1
I20251209 08:58:28.560581 25650 raft_consensus.cc:3060] T 641ed4e45a614803b9d52eb14ec682de P 22c6d3fe795545ff91525c965e0a75b4 [term 0 FOLLOWER]: Advancing to term 1
I20251209 08:58:28.561934 25785 raft_consensus.cc:2468] T 641ed4e45a614803b9d52eb14ec682de P 50c9062fc9574c329c2fad05a6bfc363 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 136ace646aa240a99c03a01c7db8dc5f in term 1.
I20251209 08:58:28.561954 25650 raft_consensus.cc:2468] T 641ed4e45a614803b9d52eb14ec682de P 22c6d3fe795545ff91525c965e0a75b4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 136ace646aa240a99c03a01c7db8dc5f in term 1.
I20251209 08:58:28.562309 25449 leader_election.cc:304] T 641ed4e45a614803b9d52eb14ec682de P 136ace646aa240a99c03a01c7db8dc5f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 136ace646aa240a99c03a01c7db8dc5f, 22c6d3fe795545ff91525c965e0a75b4; no voters:
I20251209 08:58:28.562497 25855 raft_consensus.cc:2804] T 641ed4e45a614803b9d52eb14ec682de P 136ace646aa240a99c03a01c7db8dc5f [term 1 FOLLOWER]: Leader election won for term 1
I20251209 08:58:28.562738 25855 raft_consensus.cc:697] T 641ed4e45a614803b9d52eb14ec682de P 136ace646aa240a99c03a01c7db8dc5f [term 1 LEADER]: Becoming Leader. State: Replica: 136ace646aa240a99c03a01c7db8dc5f, State: Running, Role: LEADER
I20251209 08:58:28.563082 25855 consensus_queue.cc:237] T 641ed4e45a614803b9d52eb14ec682de P 136ace646aa240a99c03a01c7db8dc5f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "136ace646aa240a99c03a01c7db8dc5f" member_type: VOTER last_known_addr { host: "127.20.49.65" port: 42165 } } peers { permanent_uuid: "50c9062fc9574c329c2fad05a6bfc363" member_type: VOTER last_known_addr { host: "127.20.49.67" port: 41723 } } peers { permanent_uuid: "22c6d3fe795545ff91525c965e0a75b4" member_type: VOTER last_known_addr { host: "127.20.49.66" port: 45461 } }
I20251209 08:58:28.566226 25369 catalog_manager.cc:5654] T 641ed4e45a614803b9d52eb14ec682de P 136ace646aa240a99c03a01c7db8dc5f reported cstate change: term changed from 0 to 1, leader changed from <none> to 136ace646aa240a99c03a01c7db8dc5f (127.20.49.65). New cstate: current_term: 1 leader_uuid: "136ace646aa240a99c03a01c7db8dc5f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "136ace646aa240a99c03a01c7db8dc5f" member_type: VOTER last_known_addr { host: "127.20.49.65" port: 42165 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "50c9062fc9574c329c2fad05a6bfc363" member_type: VOTER last_known_addr { host: "127.20.49.67" port: 41723 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "22c6d3fe795545ff91525c965e0a75b4" member_type: VOTER last_known_addr { host: "127.20.49.66" port: 45461 } health_report { overall_health: UNKNOWN } } }
I20251209 08:58:28.585170 25843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:58:28.581995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33606 (local address 127.20.49.65:42165)
1209 08:58:28.582151 (+ 156us) server_negotiation.cc:207] Beginning negotiation
1209 08:58:28.582156 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1209 08:58:28.582173 (+ 17us) server_negotiation.cc:408] Connection header received
1209 08:58:28.582285 (+ 112us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1209 08:58:28.582287 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1209 08:58:28.582345 (+ 58us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1209 08:58:28.582439 (+ 94us) server_negotiation.cc:227] Negotiated authn=TOKEN
1209 08:58:28.582818 (+ 379us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:58:28.583325 (+ 507us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1209 08:58:28.583995 (+ 670us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:58:28.584189 (+ 194us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:58:28.584294 (+ 105us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1209 08:58:28.584734 (+ 440us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1209 08:58:28.584855 (+ 121us) server_negotiation.cc:1036] Waiting for connection context
1209 08:58:28.584966 (+ 111us) server_negotiation.cc:300] Negotiation successful
1209 08:58:28.585021 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20251209 08:58:28.590726 25650 raft_consensus.cc:1275] T 641ed4e45a614803b9d52eb14ec682de P 22c6d3fe795545ff91525c965e0a75b4 [term 1 FOLLOWER]: Refusing update from remote peer 136ace646aa240a99c03a01c7db8dc5f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251209 08:58:28.590725 25785 raft_consensus.cc:1275] T 641ed4e45a614803b9d52eb14ec682de P 50c9062fc9574c329c2fad05a6bfc363 [term 1 FOLLOWER]: Refusing update from remote peer 136ace646aa240a99c03a01c7db8dc5f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251209 08:58:28.591435 25855 consensus_queue.cc:1048] T 641ed4e45a614803b9d52eb14ec682de P 136ace646aa240a99c03a01c7db8dc5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "50c9062fc9574c329c2fad05a6bfc363" member_type: VOTER last_known_addr { host: "127.20.49.67" port: 41723 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251209 08:58:28.591614 25859 consensus_queue.cc:1048] T 641ed4e45a614803b9d52eb14ec682de P 136ace646aa240a99c03a01c7db8dc5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "22c6d3fe795545ff91525c965e0a75b4" member_type: VOTER last_known_addr { host: "127.20.49.66" port: 45461 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251209 08:58:28.600306 25864 mvcc.cc:204] Tried to move back new op lower bound from 7230548822380167168 to 7230548822277181440. Current Snapshot: MvccSnapshot[applied={T|T < 7230548822380167168}]
I20251209 08:58:28.601657 25863 mvcc.cc:204] Tried to move back new op lower bound from 7230548822380167168 to 7230548822277181440. Current Snapshot: MvccSnapshot[applied={T|T < 7230548822380167168}]
I20251209 08:58:28.609422 25369 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41600:
table { table_name: "test-table" } modify_external_catalogs: true
I20251209 08:58:28.609589 25369 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41600:
table { table_name: "test-table" } modify_external_catalogs: true
I20251209 08:58:28.611999 25369 catalog_manager.cc:5941] T 00000000000000000000000000000000 P 091f13f46e6444a197412a75e5e612eb: Sending DeleteTablet for 3 replicas of tablet 641ed4e45a614803b9d52eb14ec682de
I20251209 08:58:28.612759 25495 tablet_service.cc:1552] Processing DeleteTablet for tablet 641ed4e45a614803b9d52eb14ec682de with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-09 08:58:28 UTC) from {username='kudu', principal='kudu/127.20.49.126@KRBTEST.COM'} at 127.0.0.1:33602
I20251209 08:58:28.612761 25630 tablet_service.cc:1552] Processing DeleteTablet for tablet 641ed4e45a614803b9d52eb14ec682de with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-09 08:58:28 UTC) from {username='kudu', principal='kudu/127.20.49.126@KRBTEST.COM'} at 127.0.0.1:45644
I20251209 08:58:28.612879 25765 tablet_service.cc:1552] Processing DeleteTablet for tablet 641ed4e45a614803b9d52eb14ec682de with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-09 08:58:28 UTC) from {username='kudu', principal='kudu/127.20.49.126@KRBTEST.COM'} at 127.0.0.1:48456
I20251209 08:58:28.613451 25872 tablet_replica.cc:333] T 641ed4e45a614803b9d52eb14ec682de P 22c6d3fe795545ff91525c965e0a75b4: stopping tablet replica
I20251209 08:58:28.613458 25874 tablet_replica.cc:333] T 641ed4e45a614803b9d52eb14ec682de P 50c9062fc9574c329c2fad05a6bfc363: stopping tablet replica
I20251209 08:58:28.613758 25872 raft_consensus.cc:2243] T 641ed4e45a614803b9d52eb14ec682de P 22c6d3fe795545ff91525c965e0a75b4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251209 08:58:28.613816 25874 raft_consensus.cc:2243] T 641ed4e45a614803b9d52eb14ec682de P 50c9062fc9574c329c2fad05a6bfc363 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251209 08:58:28.613981 20677 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task8XuDZV/build/debug/bin/kudu with pid 25433
I20251209 08:58:28.615300 25874 raft_consensus.cc:2272] T 641ed4e45a614803b9d52eb14ec682de P 50c9062fc9574c329c2fad05a6bfc363 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251209 08:58:28.615826 25872 raft_consensus.cc:2272] T 641ed4e45a614803b9d52eb14ec682de P 22c6d3fe795545ff91525c965e0a75b4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251209 08:58:28.616832 25874 ts_tablet_manager.cc:1916] T 641ed4e45a614803b9d52eb14ec682de P 50c9062fc9574c329c2fad05a6bfc363: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251209 08:58:28.617313 25872 ts_tablet_manager.cc:1916] T 641ed4e45a614803b9d52eb14ec682de P 22c6d3fe795545ff91525c965e0a75b4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251209 08:58:28.619032 25874 ts_tablet_manager.cc:1929] T 641ed4e45a614803b9d52eb14ec682de P 50c9062fc9574c329c2fad05a6bfc363: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251209 08:58:28.619124 25874 log.cc:1199] T 641ed4e45a614803b9d52eb14ec682de P 50c9062fc9574c329c2fad05a6bfc363: Deleting WAL directory at /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765270668420570-20677-0/minicluster-data/ts-2/wal/wals/641ed4e45a614803b9d52eb14ec682de
I20251209 08:58:28.619438 25874 ts_tablet_manager.cc:1950] T 641ed4e45a614803b9d52eb14ec682de P 50c9062fc9574c329c2fad05a6bfc363: Deleting consensus metadata
I20251209 08:58:28.620536 25353 catalog_manager.cc:4985] TS 50c9062fc9574c329c2fad05a6bfc363 (127.20.49.67:41723): tablet 641ed4e45a614803b9d52eb14ec682de (table test-table [id=10b773589f194049b687004173e22124]) successfully deleted
I20251209 08:58:28.620563 25872 ts_tablet_manager.cc:1929] T 641ed4e45a614803b9d52eb14ec682de P 22c6d3fe795545ff91525c965e0a75b4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251209 08:58:28.620649 25872 log.cc:1199] T 641ed4e45a614803b9d52eb14ec682de P 22c6d3fe795545ff91525c965e0a75b4: Deleting WAL directory at /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1765270668420570-20677-0/minicluster-data/ts-1/wal/wals/641ed4e45a614803b9d52eb14ec682de
I20251209 08:58:28.620996 25872 ts_tablet_manager.cc:1950] T 641ed4e45a614803b9d52eb14ec682de P 22c6d3fe795545ff91525c965e0a75b4: Deleting consensus metadata
I20251209 08:58:28.622291 25354 catalog_manager.cc:4985] TS 22c6d3fe795545ff91525c965e0a75b4 (127.20.49.66:45461): tablet 641ed4e45a614803b9d52eb14ec682de (table test-table [id=10b773589f194049b687004173e22124]) successfully deleted
W20251209 08:58:28.622799 25353 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.20.49.65:42165 (error 108)
I20251209 08:58:28.622987 20677 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task8XuDZV/build/debug/bin/kudu with pid 25568
I20251209 08:58:28.625475 25844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:58:28.625204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.49.65:42165 (local address 127.0.0.1:33622)
1209 08:58:28.625333 (+ 129us) negotiation.cc:107] Waiting for socket to connect
1209 08:58:28.625396 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.49.65:42165: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20251209 08:58:28.625620 25353 catalog_manager.cc:4712] TS 136ace646aa240a99c03a01c7db8dc5f (127.20.49.65:42165): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 641ed4e45a614803b9d52eb14ec682de: Network error: Client connection negotiation failed: client connection to 127.20.49.65:42165: connect: Connection refused (error 111)
I20251209 08:58:28.630079 20677 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task8XuDZV/build/debug/bin/kudu with pid 25703
I20251209 08:58:28.636523 20677 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task8XuDZV/build/debug/bin/kudu with pid 25338
2025-12-09T08:58:28Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (3313 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2025-12-09T08:58:28Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-12-09T08:58:28Z Disabled control of system clock
I20251209 08:58:28.668457 20677 external_mini_cluster.cc:1366] Running /tmp/dist-test-task8XuDZV/build/debug/bin/kudu
/tmp/dist-test-task8XuDZV/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1765270668420570-20677-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1765270668420570-20677-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1765270668420570-20677-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1765270668420570-20677-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.49.126:40209
--webserver_interface=127.20.49.126
--webserver_port=0
--builtin_ntp_servers=127.20.49.84:33251
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.49.126:40209
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20251209 08:58:28.780542 25879 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251209 08:58:28.780776 25879 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251209 08:58:28.780827 25879 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251209 08:58:28.784237 25879 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251209 08:58:28.784300 25879 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251209 08:58:28.784317 25879 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251209 08:58:28.784336 25879 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251209 08:58:28.784359 25879 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251209 08:58:28.788297 25879 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.49.84:33251
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1765270668420570-20677-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1765270668420570-20677-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.49.126:40209
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.49.126:40209
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1765270668420570-20677-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.49.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25879
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1765270668420570-20677-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 468897755969d686fb08386f42f7835685e7953a
build type DEBUG
built by None at 09 Dec 2025 08:43:15 UTC on 24a791456cd2
build id 9426
I20251209 08:58:28.789390 25879 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251209 08:58:28.790534 25879 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251209 08:58:28.796088 25885 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251209 08:58:28.796110 25884 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251209 08:58:28.796113 25887 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251209 08:58:28.796525 25879 server_base.cc:1047] running on GCE node
I20251209 08:58:28.796994 25879 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251209 08:58:28.797916 25879 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251209 08:58:28.799063 25879 hybrid_clock.cc:648] HybridClock initialized: now 1765270708799045 us; error 35 us; skew 500 ppm
I20251209 08:58:28.800876 25879 webserver.cc:492] Webserver started at http://127.20.49.126:36899/ using document root <none> and password file <none>
I20251209 08:58:28.801436 25879 fs_manager.cc:362] Metadata directory not provided
I20251209 08:58:28.801493 25879 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251209 08:58:28.801715 25879 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251209 08:58:28.803422 25879 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1765270668420570-20677-0/minicluster-data/master-0/data/instance:
uuid: "22e43c42a14645dd8b752a6f6042e31c"
format_stamp: "Formatted at 2025-12-09 08:58:28 on dist-test-slave-8w6c"
server_key: "b90c544472ad9b9ee557134f3f968146"
server_key_iv: "68fac735d8b7b49508374b50d63cf1d0"
server_key_version: "encryptionkey@0"
I20251209 08:58:28.803936 25879 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1765270668420570-20677-0/minicluster-data/master-0/wal/instance:
uuid: "22e43c42a14645dd8b752a6f6042e31c"
format_stamp: "Formatted at 2025-12-09 08:58:28 on dist-test-slave-8w6c"
server_key: "b90c544472ad9b9ee557134f3f968146"
server_key_iv: "68fac735d8b7b49508374b50d63cf1d0"
server_key_version: "encryptionkey@0"
I20251209 08:58:28.807286 25879 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20251209 08:58:28.809450 25893 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251209 08:58:28.810457 25879 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20251209 08:58:28.810580 25879 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1765270668420570-20677-0/minicluster-data/master-0/data,/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1765270668420570-20677-0/minicluster-data/master-0/wal
uuid: "22e43c42a14645dd8b752a6f6042e31c"
format_stamp: "Formatted at 2025-12-09 08:58:28 on dist-test-slave-8w6c"
server_key: "b90c544472ad9b9ee557134f3f968146"
server_key_iv: "68fac735d8b7b49508374b50d63cf1d0"
server_key_version: "encryptionkey@0"
I20251209 08:58:28.810675 25879 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1765270668420570-20677-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1765270668420570-20677-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1765270668420570-20677-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251209 08:58:28.815375 25879 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251209 08:58:28.815892 25879 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251209 08:58:28.816051 25879 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251209 08:58:28.823119 25879 rpc_server.cc:307] RPC server started. Bound to: 127.20.49.126:40209
I20251209 08:58:28.823134 25945 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.49.126:40209 every 8 connection(s)
I20251209 08:58:28.824159 25879 server_base.cc:1179] Dumped server information to /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1765270668420570-20677-0/minicluster-data/master-0/data/info.pb
I20251209 08:58:28.827076 25946 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251209 08:58:28.832549 25946 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 22e43c42a14645dd8b752a6f6042e31c: Bootstrap starting.
I20251209 08:58:28.834197 20677 external_mini_cluster.cc:1428] Started /tmp/dist-test-task8XuDZV/build/debug/bin/kudu as pid 25879
I20251209 08:58:28.834331 20677 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1765270668420570-20677-0/minicluster-data/master-0/wal/instance
I20251209 08:58:28.834542 20677 external_mini_cluster.cc:1468] Setting key 93267e6e5887b1b4cf7d396515bcab6c
I20251209 08:58:28.834940 25946 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 22e43c42a14645dd8b752a6f6042e31c: Neither blocks nor log segments found. Creating new log.
I20251209 08:58:28.835783 25946 log.cc:826] T 00000000000000000000000000000000 P 22e43c42a14645dd8b752a6f6042e31c: Log is configured to *not* fsync() on all Append() calls
I20251209 08:58:28.837792 25946 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 22e43c42a14645dd8b752a6f6042e31c: No bootstrap required, opened a new log
I20251209 08:58:28.839479 25949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:58:28.835572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37824 (local address 127.20.49.126:40209)
1209 08:58:28.836241 (+ 669us) server_negotiation.cc:207] Beginning negotiation
1209 08:58:28.836253 (+ 12us) server_negotiation.cc:400] Waiting for connection header
1209 08:58:28.836293 (+ 40us) server_negotiation.cc:408] Connection header received
1209 08:58:28.836914 (+ 621us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1209 08:58:28.836934 (+ 20us) server_negotiation.cc:462] Received NEGOTIATE request from client
1209 08:58:28.837195 (+ 261us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1209 08:58:28.837461 (+ 266us) server_negotiation.cc:227] Negotiated authn=SASL
1209 08:58:28.838383 (+ 922us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1209 08:58:28.838388 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1209 08:58:28.838396 (+ 8us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1209 08:58:28.838407 (+ 11us) server_negotiation.cc:923] calling sasl_server_start()
1209 08:58:28.838449 (+ 42us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1209 08:58:28.838461 (+ 12us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1209 08:58:28.838551 (+ 90us) server_negotiation.cc:1036] Waiting for connection context
1209 08:58:28.838683 (+ 132us) server_negotiation.cc:300] Negotiation successful
1209 08:58:28.838916 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":276,"thread_start_us":129,"threads_started":1}
I20251209 08:58:28.840495 25946 raft_consensus.cc:359] T 00000000000000000000000000000000 P 22e43c42a14645dd8b752a6f6042e31c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22e43c42a14645dd8b752a6f6042e31c" member_type: VOTER last_known_addr { host: "127.20.49.126" port: 40209 } }
I20251209 08:58:28.840680 25946 raft_consensus.cc:385] T 00000000000000000000000000000000 P 22e43c42a14645dd8b752a6f6042e31c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251209 08:58:28.840746 25946 raft_consensus.cc:740] T 00000000000000000000000000000000 P 22e43c42a14645dd8b752a6f6042e31c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22e43c42a14645dd8b752a6f6042e31c, State: Initialized, Role: FOLLOWER
I20251209 08:58:28.841150 25946 consensus_queue.cc:260] T 00000000000000000000000000000000 P 22e43c42a14645dd8b752a6f6042e31c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22e43c42a14645dd8b752a6f6042e31c" member_type: VOTER last_known_addr { host: "127.20.49.126" port: 40209 } }
I20251209 08:58:28.841274 25946 raft_consensus.cc:399] T 00000000000000000000000000000000 P 22e43c42a14645dd8b752a6f6042e31c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251209 08:58:28.841333 25946 raft_consensus.cc:493] T 00000000000000000000000000000000 P 22e43c42a14645dd8b752a6f6042e31c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251209 08:58:28.841434 25946 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 22e43c42a14645dd8b752a6f6042e31c [term 0 FOLLOWER]: Advancing to term 1
I20251209 08:58:28.842280 25946 raft_consensus.cc:515] T 00000000000000000000000000000000 P 22e43c42a14645dd8b752a6f6042e31c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22e43c42a14645dd8b752a6f6042e31c" member_type: VOTER last_known_addr { host: "127.20.49.126" port: 40209 } }
I20251209 08:58:28.842599 25946 leader_election.cc:304] T 00000000000000000000000000000000 P 22e43c42a14645dd8b752a6f6042e31c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 22e43c42a14645dd8b752a6f6042e31c; no voters:
I20251209 08:58:28.842877 25946 leader_election.cc:290] T 00000000000000000000000000000000 P 22e43c42a14645dd8b752a6f6042e31c [CANDIDATE]: Term 1 election: Requested vote from peers
I20251209 08:58:28.842988 25951 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 22e43c42a14645dd8b752a6f6042e31c [term 1 FOLLOWER]: Leader election won for term 1
I20251209 08:58:28.843153 25951 raft_consensus.cc:697] T 00000000000000000000000000000000 P 22e43c42a14645dd8b752a6f6042e31c [term 1 LEADER]: Becoming Leader. State: Replica: 22e43c42a14645dd8b752a6f6042e31c, State: Running, Role: LEADER
I20251209 08:58:28.843439 25951 consensus_queue.cc:237] T 00000000000000000000000000000000 P 22e43c42a14645dd8b752a6f6042e31c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22e43c42a14645dd8b752a6f6042e31c" member_type: VOTER last_known_addr { host: "127.20.49.126" port: 40209 } }
I20251209 08:58:28.844026 25946 sys_catalog.cc:565] T 00000000000000000000000000000000 P 22e43c42a14645dd8b752a6f6042e31c [sys.catalog]: configured and running, proceeding with master startup.
I20251209 08:58:28.845111 25952 sys_catalog.cc:455] T 00000000000000000000000000000000 P 22e43c42a14645dd8b752a6f6042e31c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "22e43c42a14645dd8b752a6f6042e31c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22e43c42a14645dd8b752a6f6042e31c" member_type: VOTER last_known_addr { host: "127.20.49.126" port: 40209 } } }
I20251209 08:58:28.845240 25952 sys_catalog.cc:458] T 00000000000000000000000000000000 P 22e43c42a14645dd8b752a6f6042e31c [sys.catalog]: This master's current role is: LEADER
I20251209 08:58:28.845468 25953 sys_catalog.cc:455] T 00000000000000000000000000000000 P 22e43c42a14645dd8b752a6f6042e31c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 22e43c42a14645dd8b752a6f6042e31c. Latest consensus state: current_term: 1 leader_uuid: "22e43c42a14645dd8b752a6f6042e31c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22e43c42a14645dd8b752a6f6042e31c" member_type: VOTER last_known_addr { host: "127.20.49.126" port: 40209 } } }
I20251209 08:58:28.845561 25953 sys_catalog.cc:458] T 00000000000000000000000000000000 P 22e43c42a14645dd8b752a6f6042e31c [sys.catalog]: This master's current role is: LEADER
I20251209 08:58:28.845778 25960 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251209 08:58:28.849236 25960 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251209 08:58:28.855046 25960 catalog_manager.cc:1357] Generated new cluster ID: 66e4b54f62ed45588f177d0c66610a31
I20251209 08:58:28.855142 25960 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251209 08:58:28.876627 25960 catalog_manager.cc:1380] Generated new certificate authority record
I20251209 08:58:28.876962 25960 catalog_manager.cc:1514] Loading token signing keys...
I20251209 08:58:28.882334 25960 catalog_manager.cc:6027] T 00000000000000000000000000000000 P 22e43c42a14645dd8b752a6f6042e31c: Generated new TSK 0
I20251209 08:58:28.883026 25960 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251209 08:58:28.887795 20677 external_mini_cluster.cc:1366] Running /tmp/dist-test-task8XuDZV/build/debug/bin/kudu
/tmp/dist-test-task8XuDZV/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1765270668420570-20677-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1765270668420570-20677-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1765270668420570-20677-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1765270668420570-20677-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.49.65:0
--local_ip_for_outbound_sockets=127.20.49.65
--webserver_interface=127.20.49.65
--webserver_port=0
--tserver_master_addrs=127.20.49.126:40209
--builtin_ntp_servers=127.20.49.84:33251
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20251209 08:58:29.011682 25970 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251209 08:58:29.011924 25970 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251209 08:58:29.012014 25970 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251209 08:58:29.015388 25970 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251209 08:58:29.015468 25970 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251209 08:58:29.015601 25970 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.49.65
I20251209 08:58:29.019505 25970 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.49.84:33251
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1765270668420570-20677-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1765270668420570-20677-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.49.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1765270668420570-20677-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.49.65
--webserver_port=0
--tserver_master_addrs=127.20.49.126:40209
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25970
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.49.65
--log_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1765270668420570-20677-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 468897755969d686fb08386f42f7835685e7953a
build type DEBUG
built by None at 09 Dec 2025 08:43:15 UTC on 24a791456cd2
build id 9426
I20251209 08:58:29.020607 25970 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251209 08:58:29.021790 25970 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251209 08:58:29.028270 25976 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251209 08:58:29.028277 25975 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251209 08:58:29.028659 25978 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251209 08:58:29.028946 25970 server_base.cc:1047] running on GCE node
I20251209 08:58:29.029317 25970 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251209 08:58:29.030099 25970 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251209 08:58:29.031291 25970 hybrid_clock.cc:648] HybridClock initialized: now 1765270709031264 us; error 49 us; skew 500 ppm
I20251209 08:58:29.033248 25970 webserver.cc:492] Webserver started at http://127.20.49.65:37461/ using document root <none> and password file <none>
I20251209 08:58:29.033895 25970 fs_manager.cc:362] Metadata directory not provided
I20251209 08:58:29.033955 25970 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251209 08:58:29.034157 25970 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251209 08:58:29.035919 25970 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1765270668420570-20677-0/minicluster-data/ts-0/data/instance:
uuid: "845b6c0baf6b4617a765c5894813e7d4"
format_stamp: "Formatted at 2025-12-09 08:58:29 on dist-test-slave-8w6c"
server_key: "6fc5081afb66f669205e0d36476c1e84"
server_key_iv: "65a2d035dd808912283d989bf326d393"
server_key_version: "encryptionkey@0"
I20251209 08:58:29.036438 25970 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1765270668420570-20677-0/minicluster-data/ts-0/wal/instance:
uuid: "845b6c0baf6b4617a765c5894813e7d4"
format_stamp: "Formatted at 2025-12-09 08:58:29 on dist-test-slave-8w6c"
server_key: "6fc5081afb66f669205e0d36476c1e84"
server_key_iv: "65a2d035dd808912283d989bf326d393"
server_key_version: "encryptionkey@0"
I20251209 08:58:29.039951 25970 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20251209 08:58:29.042223 25984 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251209 08:58:29.043227 25970 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20251209 08:58:29.043340 25970 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1765270668420570-20677-0/minicluster-data/ts-0/data,/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1765270668420570-20677-0/minicluster-data/ts-0/wal
uuid: "845b6c0baf6b4617a765c5894813e7d4"
format_stamp: "Formatted at 2025-12-09 08:58:29 on dist-test-slave-8w6c"
server_key: "6fc5081afb66f669205e0d36476c1e84"
server_key_iv: "65a2d035dd808912283d989bf326d393"
server_key_version: "encryptionkey@0"
I20251209 08:58:29.043443 25970 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1765270668420570-20677-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1765270668420570-20677-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1765270668420570-20677-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251209 08:58:29.048620 25970 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251209 08:58:29.049198 25970 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251209 08:58:29.049376 25970 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251209 08:58:29.049921 25970 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251209 08:58:29.050896 25970 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251209 08:58:29.050945 25970 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251209 08:58:29.050987 25970 ts_tablet_manager.cc:616] Registered 0 tablets
I20251209 08:58:29.051019 25970 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251209 08:58:29.060091 25970 rpc_server.cc:307] RPC server started. Bound to: 127.20.49.65:33127
I20251209 08:58:29.060118 26097 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.49.65:33127 every 8 connection(s)
I20251209 08:58:29.061378 25970 server_base.cc:1179] Dumped server information to /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1765270668420570-20677-0/minicluster-data/ts-0/data/info.pb
I20251209 08:58:29.064934 20677 external_mini_cluster.cc:1428] Started /tmp/dist-test-task8XuDZV/build/debug/bin/kudu as pid 25970
I20251209 08:58:29.065058 20677 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1765270668420570-20677-0/minicluster-data/ts-0/wal/instance
I20251209 08:58:29.065308 20677 external_mini_cluster.cc:1468] Setting key 45ef2230d14cdc430a74271c6d4634ae
I20251209 08:58:29.067345 25949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:58:29.063426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.49.65:38603 (local address 127.20.49.126:40209)
1209 08:58:29.063590 (+ 164us) server_negotiation.cc:207] Beginning negotiation
1209 08:58:29.063594 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1209 08:58:29.064518 (+ 924us) server_negotiation.cc:408] Connection header received
1209 08:58:29.065377 (+ 859us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1209 08:58:29.065381 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1209 08:58:29.065432 (+ 51us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1209 08:58:29.065530 (+ 98us) server_negotiation.cc:227] Negotiated authn=SASL
1209 08:58:29.066782 (+ 1252us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1209 08:58:29.066786 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1209 08:58:29.066789 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1209 08:58:29.066804 (+ 15us) server_negotiation.cc:923] calling sasl_server_start()
1209 08:58:29.066830 (+ 26us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1209 08:58:29.066840 (+ 10us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1209 08:58:29.066928 (+ 88us) server_negotiation.cc:1036] Waiting for connection context
1209 08:58:29.067106 (+ 178us) server_negotiation.cc:300] Negotiation successful
1209 08:58:29.067154 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20251209 08:58:29.067396 20677 external_mini_cluster.cc:1366] Running /tmp/dist-test-task8XuDZV/build/debug/bin/kudu
/tmp/dist-test-task8XuDZV/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1765270668420570-20677-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1765270668420570-20677-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1765270668420570-20677-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1765270668420570-20677-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.49.66:0
--local_ip_for_outbound_sockets=127.20.49.66
--webserver_interface=127.20.49.66
--webserver_port=0
--tserver_master_addrs=127.20.49.126:40209
--builtin_ntp_servers=127.20.49.84:33251
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20251209 08:58:29.068122 26100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:58:29.063716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.49.126:40209 (local address 127.20.49.65:38603)
1209 08:58:29.064372 (+ 656us) negotiation.cc:107] Waiting for socket to connect
1209 08:58:29.064407 (+ 35us) client_negotiation.cc:174] Beginning negotiation
1209 08:58:29.065169 (+ 762us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1209 08:58:29.065743 (+ 574us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1209 08:58:29.065755 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
1209 08:58:29.065869 (+ 114us) client_negotiation.cc:189] Negotiated authn=SASL
1209 08:58:29.066490 (+ 621us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
1209 08:58:29.066492 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
1209 08:58:29.066539 (+ 47us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
1209 08:58:29.066613 (+ 74us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1209 08:58:29.066933 (+ 320us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1209 08:58:29.066939 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1209 08:58:29.066944 (+ 5us) client_negotiation.cc:769] Sending connection context
1209 08:58:29.067162 (+ 218us) client_negotiation.cc:240] Negotiation successful
1209 08:58:29.067395 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":106,"threads_started":1}
I20251209 08:58:29.070132 26098 heartbeater.cc:344] Connected to a master server at 127.20.49.126:40209
I20251209 08:58:29.070374 26098 heartbeater.cc:461] Registering TS with master...
I20251209 08:58:29.070782 26098 heartbeater.cc:507] Master 127.20.49.126:40209 requested a full tablet report, sending...
I20251209 08:58:29.072532 25910 ts_manager.cc:194] Registered new tserver with Master: 845b6c0baf6b4617a765c5894813e7d4 (127.20.49.65:33127)
W20251209 08:58:29.179028 26101 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251209 08:58:29.179257 26101 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251209 08:58:29.179306 26101 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251209 08:58:29.182513 26101 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251209 08:58:29.182577 26101 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251209 08:58:29.182657 26101 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.49.66
I20251209 08:58:29.186388 26101 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.49.84:33251
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1765270668420570-20677-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1765270668420570-20677-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.49.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1765270668420570-20677-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.49.66
--webserver_port=0
--tserver_master_addrs=127.20.49.126:40209
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26101
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.49.66
--log_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1765270668420570-20677-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 468897755969d686fb08386f42f7835685e7953a
build type DEBUG
built by None at 09 Dec 2025 08:43:15 UTC on 24a791456cd2
build id 9426
I20251209 08:58:29.187438 26101 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251209 08:58:29.188537 26101 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251209 08:58:29.195171 26107 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251209 08:58:29.195433 26101 server_base.cc:1047] running on GCE node
W20251209 08:58:29.195164 26109 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251209 08:58:29.195219 26106 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251209 08:58:29.195960 26101 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251209 08:58:29.196528 26101 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251209 08:58:29.197703 26101 hybrid_clock.cc:648] HybridClock initialized: now 1765270709197667 us; error 28 us; skew 500 ppm
I20251209 08:58:29.199486 26101 webserver.cc:492] Webserver started at http://127.20.49.66:35507/ using document root <none> and password file <none>
I20251209 08:58:29.200057 26101 fs_manager.cc:362] Metadata directory not provided
I20251209 08:58:29.200121 26101 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251209 08:58:29.200301 26101 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251209 08:58:29.202045 26101 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1765270668420570-20677-0/minicluster-data/ts-1/data/instance:
uuid: "a019260f6dcf40e5a792d27d93bb86b3"
format_stamp: "Formatted at 2025-12-09 08:58:29 on dist-test-slave-8w6c"
server_key: "ff50088f82ba37f3e7cdbc5f27cbdf33"
server_key_iv: "8f1b13f148862b09fc175384db329f7a"
server_key_version: "encryptionkey@0"
I20251209 08:58:29.202521 26101 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1765270668420570-20677-0/minicluster-data/ts-1/wal/instance:
uuid: "a019260f6dcf40e5a792d27d93bb86b3"
format_stamp: "Formatted at 2025-12-09 08:58:29 on dist-test-slave-8w6c"
server_key: "ff50088f82ba37f3e7cdbc5f27cbdf33"
server_key_iv: "8f1b13f148862b09fc175384db329f7a"
server_key_version: "encryptionkey@0"
I20251209 08:58:29.205916 26101 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20251209 08:58:29.208133 26115 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251209 08:58:29.209131 26101 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20251209 08:58:29.209244 26101 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1765270668420570-20677-0/minicluster-data/ts-1/data,/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1765270668420570-20677-0/minicluster-data/ts-1/wal
uuid: "a019260f6dcf40e5a792d27d93bb86b3"
format_stamp: "Formatted at 2025-12-09 08:58:29 on dist-test-slave-8w6c"
server_key: "ff50088f82ba37f3e7cdbc5f27cbdf33"
server_key_iv: "8f1b13f148862b09fc175384db329f7a"
server_key_version: "encryptionkey@0"
I20251209 08:58:29.209336 26101 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1765270668420570-20677-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1765270668420570-20677-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1765270668420570-20677-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251209 08:58:29.214232 26101 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251209 08:58:29.214823 26101 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251209 08:58:29.215004 26101 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251209 08:58:29.215480 26101 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251209 08:58:29.216418 26101 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251209 08:58:29.216467 26101 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251209 08:58:29.216506 26101 ts_tablet_manager.cc:616] Registered 0 tablets
I20251209 08:58:29.216526 26101 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251209 08:58:29.226146 26101 rpc_server.cc:307] RPC server started. Bound to: 127.20.49.66:37777
I20251209 08:58:29.226184 26228 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.49.66:37777 every 8 connection(s)
I20251209 08:58:29.227185 26101 server_base.cc:1179] Dumped server information to /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1765270668420570-20677-0/minicluster-data/ts-1/data/info.pb
I20251209 08:58:29.232434 25949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:58:29.229037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.49.66:47373 (local address 127.20.49.126:40209)
1209 08:58:29.229220 (+ 183us) server_negotiation.cc:207] Beginning negotiation
1209 08:58:29.229225 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1209 08:58:29.230115 (+ 890us) server_negotiation.cc:408] Connection header received
1209 08:58:29.230856 (+ 741us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1209 08:58:29.230859 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1209 08:58:29.230900 (+ 41us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1209 08:58:29.230979 (+ 79us) server_negotiation.cc:227] Negotiated authn=SASL
1209 08:58:29.232072 (+ 1093us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1209 08:58:29.232075 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1209 08:58:29.232077 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1209 08:58:29.232085 (+ 8us) server_negotiation.cc:923] calling sasl_server_start()
1209 08:58:29.232100 (+ 15us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1209 08:58:29.232107 (+ 7us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1209 08:58:29.232180 (+ 73us) server_negotiation.cc:1036] Waiting for connection context
1209 08:58:29.232286 (+ 106us) server_negotiation.cc:300] Negotiation successful
1209 08:58:29.232321 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20251209 08:58:29.233209 26231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:58:29.229299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.49.126:40209 (local address 127.20.49.66:47373)
1209 08:58:29.229961 (+ 662us) negotiation.cc:107] Waiting for socket to connect
1209 08:58:29.230014 (+ 53us) client_negotiation.cc:174] Beginning negotiation
1209 08:58:29.230686 (+ 672us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1209 08:58:29.231141 (+ 455us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1209 08:58:29.231149 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
1209 08:58:29.231245 (+ 96us) client_negotiation.cc:189] Negotiated authn=SASL
1209 08:58:29.231854 (+ 609us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
1209 08:58:29.231856 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
1209 08:58:29.231889 (+ 33us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
1209 08:58:29.231945 (+ 56us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1209 08:58:29.232171 (+ 226us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1209 08:58:29.232176 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1209 08:58:29.232179 (+ 3us) client_negotiation.cc:769] Sending connection context
1209 08:58:29.232340 (+ 161us) client_negotiation.cc:240] Negotiation successful
1209 08:58:29.232521 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":101,"threads_started":1}
I20251209 08:58:29.234112 20677 external_mini_cluster.cc:1428] Started /tmp/dist-test-task8XuDZV/build/debug/bin/kudu as pid 26101
I20251209 08:58:29.234264 20677 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1765270668420570-20677-0/minicluster-data/ts-1/wal/instance
I20251209 08:58:29.234428 26229 heartbeater.cc:344] Connected to a master server at 127.20.49.126:40209
I20251209 08:58:29.234612 20677 external_mini_cluster.cc:1468] Setting key d57a22a5a8901dd9cde796750de1f519
I20251209 08:58:29.234717 26229 heartbeater.cc:461] Registering TS with master...
I20251209 08:58:29.235209 26229 heartbeater.cc:507] Master 127.20.49.126:40209 requested a full tablet report, sending...
I20251209 08:58:29.236266 25910 ts_manager.cc:194] Registered new tserver with Master: a019260f6dcf40e5a792d27d93bb86b3 (127.20.49.66:37777)
I20251209 08:58:29.236702 20677 external_mini_cluster.cc:1366] Running /tmp/dist-test-task8XuDZV/build/debug/bin/kudu
/tmp/dist-test-task8XuDZV/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1765270668420570-20677-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1765270668420570-20677-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1765270668420570-20677-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1765270668420570-20677-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.49.67:0
--local_ip_for_outbound_sockets=127.20.49.67
--webserver_interface=127.20.49.67
--webserver_port=0
--tserver_master_addrs=127.20.49.126:40209
--builtin_ntp_servers=127.20.49.84:33251
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20251209 08:58:29.347991 26232 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251209 08:58:29.348223 26232 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251209 08:58:29.348286 26232 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251209 08:58:29.351569 26232 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251209 08:58:29.351636 26232 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251209 08:58:29.351712 26232 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.49.67
I20251209 08:58:29.355614 26232 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.49.84:33251
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1765270668420570-20677-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1765270668420570-20677-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.49.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1765270668420570-20677-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.49.67
--webserver_port=0
--tserver_master_addrs=127.20.49.126:40209
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26232
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.49.67
--log_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1765270668420570-20677-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 468897755969d686fb08386f42f7835685e7953a
build type DEBUG
built by None at 09 Dec 2025 08:43:15 UTC on 24a791456cd2
build id 9426
I20251209 08:58:29.356693 26232 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251209 08:58:29.357926 26232 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251209 08:58:29.364742 26237 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251209 08:58:29.364859 26232 server_base.cc:1047] running on GCE node
W20251209 08:58:29.364737 26238 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251209 08:58:29.364735 26240 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251209 08:58:29.365545 26232 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251209 08:58:29.366186 26232 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251209 08:58:29.367348 26232 hybrid_clock.cc:648] HybridClock initialized: now 1765270709367325 us; error 38 us; skew 500 ppm
I20251209 08:58:29.369241 26232 webserver.cc:492] Webserver started at http://127.20.49.67:37315/ using document root <none> and password file <none>
I20251209 08:58:29.369851 26232 fs_manager.cc:362] Metadata directory not provided
I20251209 08:58:29.369918 26232 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251209 08:58:29.370133 26232 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251209 08:58:29.371835 26232 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1765270668420570-20677-0/minicluster-data/ts-2/data/instance:
uuid: "7e63e9c084d540329c00e5d311bc39ca"
format_stamp: "Formatted at 2025-12-09 08:58:29 on dist-test-slave-8w6c"
server_key: "cf91382aa20372c0a68de1532dd046a2"
server_key_iv: "0a5cdbe120993da05e2a323c936e79f8"
server_key_version: "encryptionkey@0"
I20251209 08:58:29.372331 26232 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1765270668420570-20677-0/minicluster-data/ts-2/wal/instance:
uuid: "7e63e9c084d540329c00e5d311bc39ca"
format_stamp: "Formatted at 2025-12-09 08:58:29 on dist-test-slave-8w6c"
server_key: "cf91382aa20372c0a68de1532dd046a2"
server_key_iv: "0a5cdbe120993da05e2a323c936e79f8"
server_key_version: "encryptionkey@0"
I20251209 08:58:29.375718 26232 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20251209 08:58:29.377938 26246 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251209 08:58:29.378964 26232 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20251209 08:58:29.379062 26232 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1765270668420570-20677-0/minicluster-data/ts-2/data,/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1765270668420570-20677-0/minicluster-data/ts-2/wal
uuid: "7e63e9c084d540329c00e5d311bc39ca"
format_stamp: "Formatted at 2025-12-09 08:58:29 on dist-test-slave-8w6c"
server_key: "cf91382aa20372c0a68de1532dd046a2"
server_key_iv: "0a5cdbe120993da05e2a323c936e79f8"
server_key_version: "encryptionkey@0"
I20251209 08:58:29.379168 26232 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1765270668420570-20677-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1765270668420570-20677-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1765270668420570-20677-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251209 08:58:29.384541 26232 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251209 08:58:29.385102 26232 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251209 08:58:29.385255 26232 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251209 08:58:29.385776 26232 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251209 08:58:29.386670 26232 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251209 08:58:29.386716 26232 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251209 08:58:29.386775 26232 ts_tablet_manager.cc:616] Registered 0 tablets
I20251209 08:58:29.386807 26232 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251209 08:58:29.395850 26232 rpc_server.cc:307] RPC server started. Bound to: 127.20.49.67:33297
I20251209 08:58:29.395897 26359 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.49.67:33297 every 8 connection(s)
I20251209 08:58:29.396775 26232 server_base.cc:1179] Dumped server information to /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1765270668420570-20677-0/minicluster-data/ts-2/data/info.pb
I20251209 08:58:29.401994 25949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:58:29.398537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.49.67:33811 (local address 127.20.49.126:40209)
1209 08:58:29.398665 (+ 128us) server_negotiation.cc:207] Beginning negotiation
1209 08:58:29.398669 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1209 08:58:29.399519 (+ 850us) server_negotiation.cc:408] Connection header received
1209 08:58:29.400304 (+ 785us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1209 08:58:29.400308 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1209 08:58:29.400349 (+ 41us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1209 08:58:29.400433 (+ 84us) server_negotiation.cc:227] Negotiated authn=SASL
1209 08:58:29.401528 (+ 1095us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1209 08:58:29.401531 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1209 08:58:29.401534 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1209 08:58:29.401547 (+ 13us) server_negotiation.cc:923] calling sasl_server_start()
1209 08:58:29.401571 (+ 24us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1209 08:58:29.401579 (+ 8us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1209 08:58:29.401669 (+ 90us) server_negotiation.cc:1036] Waiting for connection context
1209 08:58:29.401844 (+ 175us) server_negotiation.cc:300] Negotiation successful
1209 08:58:29.401883 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20251209 08:58:29.402679 26362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:58:29.398780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.49.126:40209 (local address 127.20.49.67:33811)
1209 08:58:29.399383 (+ 603us) negotiation.cc:107] Waiting for socket to connect
1209 08:58:29.399417 (+ 34us) client_negotiation.cc:174] Beginning negotiation
1209 08:58:29.400099 (+ 682us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1209 08:58:29.400575 (+ 476us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1209 08:58:29.400582 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
1209 08:58:29.400666 (+ 84us) client_negotiation.cc:189] Negotiated authn=SASL
1209 08:58:29.401279 (+ 613us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
1209 08:58:29.401281 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
1209 08:58:29.401311 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
1209 08:58:29.401363 (+ 52us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1209 08:58:29.401667 (+ 304us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1209 08:58:29.401673 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1209 08:58:29.401702 (+ 29us) client_negotiation.cc:769] Sending connection context
1209 08:58:29.401889 (+ 187us) client_negotiation.cc:240] Negotiation successful
1209 08:58:29.402061 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":92,"threads_started":1}
I20251209 08:58:29.403103 20677 external_mini_cluster.cc:1428] Started /tmp/dist-test-task8XuDZV/build/debug/bin/kudu as pid 26232
I20251209 08:58:29.403288 20677 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1765270668420570-20677-0/minicluster-data/ts-2/wal/instance
I20251209 08:58:29.403597 20677 external_mini_cluster.cc:1468] Setting key e5bb1200882958ea8ca7cb7907fa6c88
I20251209 08:58:29.403934 26360 heartbeater.cc:344] Connected to a master server at 127.20.49.126:40209
I20251209 08:58:29.404181 26360 heartbeater.cc:461] Registering TS with master...
I20251209 08:58:29.404623 26360 heartbeater.cc:507] Master 127.20.49.126:40209 requested a full tablet report, sending...
I20251209 08:58:29.405611 25910 ts_manager.cc:194] Registered new tserver with Master: 7e63e9c084d540329c00e5d311bc39ca (127.20.49.67:33297)
I20251209 08:58:29.417536 20677 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20251209 08:58:29.421106 25949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:58:29.419433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37832 (local address 127.20.49.126:40209)
1209 08:58:29.419592 (+ 159us) server_negotiation.cc:207] Beginning negotiation
1209 08:58:29.419598 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1209 08:58:29.419695 (+ 97us) server_negotiation.cc:408] Connection header received
1209 08:58:29.419828 (+ 133us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1209 08:58:29.419832 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1209 08:58:29.419874 (+ 42us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1209 08:58:29.419947 (+ 73us) server_negotiation.cc:227] Negotiated authn=SASL
1209 08:58:29.420701 (+ 754us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1209 08:58:29.420704 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1209 08:58:29.420706 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1209 08:58:29.420716 (+ 10us) server_negotiation.cc:923] calling sasl_server_start()
1209 08:58:29.420733 (+ 17us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1209 08:58:29.420740 (+ 7us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1209 08:58:29.420816 (+ 76us) server_negotiation.cc:1036] Waiting for connection context
1209 08:58:29.420912 (+ 96us) server_negotiation.cc:300] Negotiation successful
1209 08:58:29.420954 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20251209 08:58:29.423698 25910 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:37832:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251209 08:58:29.426119 25910 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251209 08:58:29.438612 26371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:58:29.434489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.49.66:37777 (local address 127.0.0.1:39850)
1209 08:58:29.434854 (+ 365us) negotiation.cc:107] Waiting for socket to connect
1209 08:58:29.434895 (+ 41us) client_negotiation.cc:174] Beginning negotiation
1209 08:58:29.435004 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1209 08:58:29.436854 (+ 1850us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1209 08:58:29.436857 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1209 08:58:29.436873 (+ 16us) client_negotiation.cc:189] Negotiated authn=SASL
1209 08:58:29.437873 (+ 1000us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
1209 08:58:29.437875 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
1209 08:58:29.437885 (+ 10us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
1209 08:58:29.437920 (+ 35us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1209 08:58:29.438322 (+ 402us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1209 08:58:29.438326 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1209 08:58:29.438330 (+ 4us) client_negotiation.cc:769] Sending connection context
1209 08:58:29.438387 (+ 57us) client_negotiation.cc:240] Negotiation successful
1209 08:58:29.438443 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":80,"threads_started":1}
I20251209 08:58:29.438500 26375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:58:29.434686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.49.67:33297 (local address 127.0.0.1:44374)
1209 08:58:29.435833 (+ 1147us) negotiation.cc:107] Waiting for socket to connect
1209 08:58:29.435850 (+ 17us) client_negotiation.cc:174] Beginning negotiation
1209 08:58:29.435944 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1209 08:58:29.436727 (+ 783us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1209 08:58:29.436733 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
1209 08:58:29.436795 (+ 62us) client_negotiation.cc:189] Negotiated authn=SASL
1209 08:58:29.437600 (+ 805us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
1209 08:58:29.437602 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
1209 08:58:29.437642 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
1209 08:58:29.437737 (+ 95us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1209 08:58:29.438125 (+ 388us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1209 08:58:29.438137 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1209 08:58:29.438144 (+ 7us) client_negotiation.cc:769] Sending connection context
1209 08:58:29.438266 (+ 122us) client_negotiation.cc:240] Negotiation successful
1209 08:58:29.438333 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1041,"thread_start_us":108,"threads_started":1}
I20251209 08:58:29.438858 26372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:58:29.435545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.49.65:33127 (local address 127.0.0.1:52028)
1209 08:58:29.436171 (+ 626us) negotiation.cc:107] Waiting for socket to connect
1209 08:58:29.436187 (+ 16us) client_negotiation.cc:174] Beginning negotiation
1209 08:58:29.436251 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1209 08:58:29.437116 (+ 865us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1209 08:58:29.437122 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
1209 08:58:29.437140 (+ 18us) client_negotiation.cc:189] Negotiated authn=SASL
1209 08:58:29.438060 (+ 920us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
1209 08:58:29.438062 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
1209 08:58:29.438072 (+ 10us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
1209 08:58:29.438091 (+ 19us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1209 08:58:29.438465 (+ 374us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1209 08:58:29.438469 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1209 08:58:29.438471 (+ 2us) client_negotiation.cc:769] Sending connection context
1209 08:58:29.438544 (+ 73us) client_negotiation.cc:240] Negotiation successful
1209 08:58:29.438575 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":549,"thread_start_us":96,"threads_started":1}
I20251209 08:58:29.438908 26373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:58:29.434626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39850 (local address 127.20.49.66:37777)
1209 08:58:29.436499 (+ 1873us) server_negotiation.cc:207] Beginning negotiation
1209 08:58:29.436505 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1209 08:58:29.436525 (+ 20us) server_negotiation.cc:408] Connection header received
1209 08:58:29.436575 (+ 50us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1209 08:58:29.436580 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1209 08:58:29.436700 (+ 120us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1209 08:58:29.437106 (+ 406us) server_negotiation.cc:227] Negotiated authn=SASL
1209 08:58:29.438055 (+ 949us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1209 08:58:29.438060 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1209 08:58:29.438092 (+ 32us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1209 08:58:29.438107 (+ 15us) server_negotiation.cc:923] calling sasl_server_start()
1209 08:58:29.438157 (+ 50us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1209 08:58:29.438170 (+ 13us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1209 08:58:29.438602 (+ 432us) server_negotiation.cc:1036] Waiting for connection context
1209 08:58:29.438656 (+ 54us) server_negotiation.cc:300] Negotiation successful
1209 08:58:29.438734 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1771,"thread_start_us":166,"threads_started":1}
I20251209 08:58:29.439205 26374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:58:29.435344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44374 (local address 127.20.49.67:33297)
1209 08:58:29.436364 (+ 1020us) server_negotiation.cc:207] Beginning negotiation
1209 08:58:29.436369 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1209 08:58:29.436389 (+ 20us) server_negotiation.cc:408] Connection header received
1209 08:58:29.436435 (+ 46us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1209 08:58:29.436439 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1209 08:58:29.436570 (+ 131us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1209 08:58:29.436698 (+ 128us) server_negotiation.cc:227] Negotiated authn=SASL
1209 08:58:29.437857 (+ 1159us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1209 08:58:29.437864 (+ 7us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1209 08:58:29.437874 (+ 10us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1209 08:58:29.437885 (+ 11us) server_negotiation.cc:923] calling sasl_server_start()
1209 08:58:29.437950 (+ 65us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1209 08:58:29.437966 (+ 16us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1209 08:58:29.438081 (+ 115us) server_negotiation.cc:1036] Waiting for connection context
1209 08:58:29.438966 (+ 885us) server_negotiation.cc:300] Negotiation successful
1209 08:58:29.439051 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":890,"thread_start_us":157,"threads_started":1}
I20251209 08:58:29.439433 26376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:58:29.435983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52028 (local address 127.20.49.65:33127)
1209 08:58:29.436333 (+ 350us) server_negotiation.cc:207] Beginning negotiation
1209 08:58:29.436340 (+ 7us) server_negotiation.cc:400] Waiting for connection header
1209 08:58:29.436367 (+ 27us) server_negotiation.cc:408] Connection header received
1209 08:58:29.436427 (+ 60us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1209 08:58:29.436433 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
1209 08:58:29.436595 (+ 162us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1209 08:58:29.436730 (+ 135us) server_negotiation.cc:227] Negotiated authn=SASL
1209 08:58:29.438225 (+ 1495us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1209 08:58:29.438233 (+ 8us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1209 08:58:29.438243 (+ 10us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1209 08:58:29.438256 (+ 13us) server_negotiation.cc:923] calling sasl_server_start()
1209 08:58:29.438310 (+ 54us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1209 08:58:29.438323 (+ 13us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1209 08:58:29.438426 (+ 103us) server_negotiation.cc:1036] Waiting for connection context
1209 08:58:29.439200 (+ 774us) server_negotiation.cc:300] Negotiation successful
1209 08:58:29.439270 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":248,"thread_start_us":151,"threads_started":1}
I20251209 08:58:29.441713 26032 tablet_service.cc:1505] Processing CreateTablet for tablet 09d67a8cca7142fa9233b84c807e05e7 (DEFAULT_TABLE table=test-table [id=fbd6f12bff834b929c3d020931b1c46f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251209 08:58:29.441766 26163 tablet_service.cc:1505] Processing CreateTablet for tablet 09d67a8cca7142fa9233b84c807e05e7 (DEFAULT_TABLE table=test-table [id=fbd6f12bff834b929c3d020931b1c46f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251209 08:58:29.441763 26294 tablet_service.cc:1505] Processing CreateTablet for tablet 09d67a8cca7142fa9233b84c807e05e7 (DEFAULT_TABLE table=test-table [id=fbd6f12bff834b929c3d020931b1c46f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251209 08:58:29.442628 26032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 09d67a8cca7142fa9233b84c807e05e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251209 08:58:29.442677 26294 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 09d67a8cca7142fa9233b84c807e05e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251209 08:58:29.442677 26163 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 09d67a8cca7142fa9233b84c807e05e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251209 08:58:29.448545 26377 tablet_bootstrap.cc:492] T 09d67a8cca7142fa9233b84c807e05e7 P 845b6c0baf6b4617a765c5894813e7d4: Bootstrap starting.
I20251209 08:58:29.449537 26379 tablet_bootstrap.cc:492] T 09d67a8cca7142fa9233b84c807e05e7 P 7e63e9c084d540329c00e5d311bc39ca: Bootstrap starting.
I20251209 08:58:29.449631 26378 tablet_bootstrap.cc:492] T 09d67a8cca7142fa9233b84c807e05e7 P a019260f6dcf40e5a792d27d93bb86b3: Bootstrap starting.
I20251209 08:58:29.450873 26377 tablet_bootstrap.cc:654] T 09d67a8cca7142fa9233b84c807e05e7 P 845b6c0baf6b4617a765c5894813e7d4: Neither blocks nor log segments found. Creating new log.
I20251209 08:58:29.451704 26379 tablet_bootstrap.cc:654] T 09d67a8cca7142fa9233b84c807e05e7 P 7e63e9c084d540329c00e5d311bc39ca: Neither blocks nor log segments found. Creating new log.
I20251209 08:58:29.451704 26378 tablet_bootstrap.cc:654] T 09d67a8cca7142fa9233b84c807e05e7 P a019260f6dcf40e5a792d27d93bb86b3: Neither blocks nor log segments found. Creating new log.
I20251209 08:58:29.452068 26377 log.cc:826] T 09d67a8cca7142fa9233b84c807e05e7 P 845b6c0baf6b4617a765c5894813e7d4: Log is configured to *not* fsync() on all Append() calls
I20251209 08:58:29.452553 26378 log.cc:826] T 09d67a8cca7142fa9233b84c807e05e7 P a019260f6dcf40e5a792d27d93bb86b3: Log is configured to *not* fsync() on all Append() calls
I20251209 08:58:29.452551 26379 log.cc:826] T 09d67a8cca7142fa9233b84c807e05e7 P 7e63e9c084d540329c00e5d311bc39ca: Log is configured to *not* fsync() on all Append() calls
I20251209 08:58:29.453933 26377 tablet_bootstrap.cc:492] T 09d67a8cca7142fa9233b84c807e05e7 P 845b6c0baf6b4617a765c5894813e7d4: No bootstrap required, opened a new log
I20251209 08:58:29.454125 26377 ts_tablet_manager.cc:1403] T 09d67a8cca7142fa9233b84c807e05e7 P 845b6c0baf6b4617a765c5894813e7d4: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20251209 08:58:29.454396 26379 tablet_bootstrap.cc:492] T 09d67a8cca7142fa9233b84c807e05e7 P 7e63e9c084d540329c00e5d311bc39ca: No bootstrap required, opened a new log
I20251209 08:58:29.454396 26378 tablet_bootstrap.cc:492] T 09d67a8cca7142fa9233b84c807e05e7 P a019260f6dcf40e5a792d27d93bb86b3: No bootstrap required, opened a new log
I20251209 08:58:29.454573 26378 ts_tablet_manager.cc:1403] T 09d67a8cca7142fa9233b84c807e05e7 P a019260f6dcf40e5a792d27d93bb86b3: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20251209 08:58:29.454573 26379 ts_tablet_manager.cc:1403] T 09d67a8cca7142fa9233b84c807e05e7 P 7e63e9c084d540329c00e5d311bc39ca: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20251209 08:58:29.456524 26377 raft_consensus.cc:359] T 09d67a8cca7142fa9233b84c807e05e7 P 845b6c0baf6b4617a765c5894813e7d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a019260f6dcf40e5a792d27d93bb86b3" member_type: VOTER last_known_addr { host: "127.20.49.66" port: 37777 } } peers { permanent_uuid: "7e63e9c084d540329c00e5d311bc39ca" member_type: VOTER last_known_addr { host: "127.20.49.67" port: 33297 } } peers { permanent_uuid: "845b6c0baf6b4617a765c5894813e7d4" member_type: VOTER last_known_addr { host: "127.20.49.65" port: 33127 } }
I20251209 08:58:29.456714 26377 raft_consensus.cc:385] T 09d67a8cca7142fa9233b84c807e05e7 P 845b6c0baf6b4617a765c5894813e7d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251209 08:58:29.456771 26377 raft_consensus.cc:740] T 09d67a8cca7142fa9233b84c807e05e7 P 845b6c0baf6b4617a765c5894813e7d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 845b6c0baf6b4617a765c5894813e7d4, State: Initialized, Role: FOLLOWER
I20251209 08:58:29.457211 26377 consensus_queue.cc:260] T 09d67a8cca7142fa9233b84c807e05e7 P 845b6c0baf6b4617a765c5894813e7d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a019260f6dcf40e5a792d27d93bb86b3" member_type: VOTER last_known_addr { host: "127.20.49.66" port: 37777 } } peers { permanent_uuid: "7e63e9c084d540329c00e5d311bc39ca" member_type: VOTER last_known_addr { host: "127.20.49.67" port: 33297 } } peers { permanent_uuid: "845b6c0baf6b4617a765c5894813e7d4" member_type: VOTER last_known_addr { host: "127.20.49.65" port: 33127 } }
I20251209 08:58:29.457579 26379 raft_consensus.cc:359] T 09d67a8cca7142fa9233b84c807e05e7 P 7e63e9c084d540329c00e5d311bc39ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a019260f6dcf40e5a792d27d93bb86b3" member_type: VOTER last_known_addr { host: "127.20.49.66" port: 37777 } } peers { permanent_uuid: "7e63e9c084d540329c00e5d311bc39ca" member_type: VOTER last_known_addr { host: "127.20.49.67" port: 33297 } } peers { permanent_uuid: "845b6c0baf6b4617a765c5894813e7d4" member_type: VOTER last_known_addr { host: "127.20.49.65" port: 33127 } }
I20251209 08:58:29.457579 26378 raft_consensus.cc:359] T 09d67a8cca7142fa9233b84c807e05e7 P a019260f6dcf40e5a792d27d93bb86b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a019260f6dcf40e5a792d27d93bb86b3" member_type: VOTER last_known_addr { host: "127.20.49.66" port: 37777 } } peers { permanent_uuid: "7e63e9c084d540329c00e5d311bc39ca" member_type: VOTER last_known_addr { host: "127.20.49.67" port: 33297 } } peers { permanent_uuid: "845b6c0baf6b4617a765c5894813e7d4" member_type: VOTER last_known_addr { host: "127.20.49.65" port: 33127 } }
I20251209 08:58:29.457799 26378 raft_consensus.cc:385] T 09d67a8cca7142fa9233b84c807e05e7 P a019260f6dcf40e5a792d27d93bb86b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251209 08:58:29.457799 26379 raft_consensus.cc:385] T 09d67a8cca7142fa9233b84c807e05e7 P 7e63e9c084d540329c00e5d311bc39ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251209 08:58:29.457850 26379 raft_consensus.cc:740] T 09d67a8cca7142fa9233b84c807e05e7 P 7e63e9c084d540329c00e5d311bc39ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e63e9c084d540329c00e5d311bc39ca, State: Initialized, Role: FOLLOWER
I20251209 08:58:29.457850 26378 raft_consensus.cc:740] T 09d67a8cca7142fa9233b84c807e05e7 P a019260f6dcf40e5a792d27d93bb86b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a019260f6dcf40e5a792d27d93bb86b3, State: Initialized, Role: FOLLOWER
I20251209 08:58:29.458288 26379 consensus_queue.cc:260] T 09d67a8cca7142fa9233b84c807e05e7 P 7e63e9c084d540329c00e5d311bc39ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a019260f6dcf40e5a792d27d93bb86b3" member_type: VOTER last_known_addr { host: "127.20.49.66" port: 37777 } } peers { permanent_uuid: "7e63e9c084d540329c00e5d311bc39ca" member_type: VOTER last_known_addr { host: "127.20.49.67" port: 33297 } } peers { permanent_uuid: "845b6c0baf6b4617a765c5894813e7d4" member_type: VOTER last_known_addr { host: "127.20.49.65" port: 33127 } }
I20251209 08:58:29.458288 26378 consensus_queue.cc:260] T 09d67a8cca7142fa9233b84c807e05e7 P a019260f6dcf40e5a792d27d93bb86b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a019260f6dcf40e5a792d27d93bb86b3" member_type: VOTER last_known_addr { host: "127.20.49.66" port: 37777 } } peers { permanent_uuid: "7e63e9c084d540329c00e5d311bc39ca" member_type: VOTER last_known_addr { host: "127.20.49.67" port: 33297 } } peers { permanent_uuid: "845b6c0baf6b4617a765c5894813e7d4" member_type: VOTER last_known_addr { host: "127.20.49.65" port: 33127 } }
I20251209 08:58:29.458396 26377 ts_tablet_manager.cc:1434] T 09d67a8cca7142fa9233b84c807e05e7 P 845b6c0baf6b4617a765c5894813e7d4: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20251209 08:58:29.458760 26098 heartbeater.cc:499] Master 127.20.49.126:40209 was elected leader, sending a full tablet report...
I20251209 08:58:29.459123 26379 ts_tablet_manager.cc:1434] T 09d67a8cca7142fa9233b84c807e05e7 P 7e63e9c084d540329c00e5d311bc39ca: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20251209 08:58:29.459123 26378 ts_tablet_manager.cc:1434] T 09d67a8cca7142fa9233b84c807e05e7 P a019260f6dcf40e5a792d27d93bb86b3: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20251209 08:58:29.459182 26360 heartbeater.cc:499] Master 127.20.49.126:40209 was elected leader, sending a full tablet report...
I20251209 08:58:29.460158 26229 heartbeater.cc:499] Master 127.20.49.126:40209 was elected leader, sending a full tablet report...
W20251209 08:58:29.478188 26230 tablet.cc:2378] T 09d67a8cca7142fa9233b84c807e05e7 P a019260f6dcf40e5a792d27d93bb86b3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20251209 08:58:29.522583 26384 raft_consensus.cc:493] T 09d67a8cca7142fa9233b84c807e05e7 P a019260f6dcf40e5a792d27d93bb86b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251209 08:58:29.522812 26384 raft_consensus.cc:515] T 09d67a8cca7142fa9233b84c807e05e7 P a019260f6dcf40e5a792d27d93bb86b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a019260f6dcf40e5a792d27d93bb86b3" member_type: VOTER last_known_addr { host: "127.20.49.66" port: 37777 } } peers { permanent_uuid: "7e63e9c084d540329c00e5d311bc39ca" member_type: VOTER last_known_addr { host: "127.20.49.67" port: 33297 } } peers { permanent_uuid: "845b6c0baf6b4617a765c5894813e7d4" member_type: VOTER last_known_addr { host: "127.20.49.65" port: 33127 } }
I20251209 08:58:29.524307 26384 leader_election.cc:290] T 09d67a8cca7142fa9233b84c807e05e7 P a019260f6dcf40e5a792d27d93bb86b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7e63e9c084d540329c00e5d311bc39ca (127.20.49.67:33297), 845b6c0baf6b4617a765c5894813e7d4 (127.20.49.65:33127)
I20251209 08:58:29.526440 26374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:58:29.524705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.49.66:52925 (local address 127.20.49.67:33297)
1209 08:58:29.524842 (+ 137us) server_negotiation.cc:207] Beginning negotiation
1209 08:58:29.524847 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1209 08:58:29.524865 (+ 18us) server_negotiation.cc:408] Connection header received
1209 08:58:29.524967 (+ 102us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1209 08:58:29.524973 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
1209 08:58:29.525049 (+ 76us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1209 08:58:29.525129 (+ 80us) server_negotiation.cc:227] Negotiated authn=SASL
1209 08:58:29.526016 (+ 887us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1209 08:58:29.526021 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1209 08:58:29.526024 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1209 08:58:29.526036 (+ 12us) server_negotiation.cc:923] calling sasl_server_start()
1209 08:58:29.526058 (+ 22us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1209 08:58:29.526069 (+ 11us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1209 08:58:29.526163 (+ 94us) server_negotiation.cc:1036] Waiting for connection context
1209 08:58:29.526274 (+ 111us) server_negotiation.cc:300] Negotiation successful
1209 08:58:29.526315 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20251209 08:58:29.526440 26231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:58:29.524589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.49.67:33297 (local address 127.20.49.66:52925)
1209 08:58:29.524739 (+ 150us) negotiation.cc:107] Waiting for socket to connect
1209 08:58:29.524758 (+ 19us) client_negotiation.cc:174] Beginning negotiation
1209 08:58:29.524858 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1209 08:58:29.525228 (+ 370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1209 08:58:29.525232 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
1209 08:58:29.525260 (+ 28us) client_negotiation.cc:189] Negotiated authn=SASL
1209 08:58:29.525843 (+ 583us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
1209 08:58:29.525845 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
1209 08:58:29.525857 (+ 12us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
1209 08:58:29.525881 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1209 08:58:29.526171 (+ 290us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1209 08:58:29.526177 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1209 08:58:29.526183 (+ 6us) client_negotiation.cc:769] Sending connection context
1209 08:58:29.526278 (+ 95us) client_negotiation.cc:240] Negotiation successful
1209 08:58:29.526313 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55}
I20251209 08:58:29.526552 26376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:58:29.525049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.49.66:57083 (local address 127.20.49.65:33127)
1209 08:58:29.525223 (+ 174us) server_negotiation.cc:207] Beginning negotiation
1209 08:58:29.525229 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1209 08:58:29.525379 (+ 150us) server_negotiation.cc:408] Connection header received
1209 08:58:29.525500 (+ 121us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1209 08:58:29.525505 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1209 08:58:29.525548 (+ 43us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1209 08:58:29.525634 (+ 86us) server_negotiation.cc:227] Negotiated authn=SASL
1209 08:58:29.526162 (+ 528us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1209 08:58:29.526166 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1209 08:58:29.526169 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1209 08:58:29.526181 (+ 12us) server_negotiation.cc:923] calling sasl_server_start()
1209 08:58:29.526203 (+ 22us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1209 08:58:29.526211 (+ 8us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1209 08:58:29.526298 (+ 87us) server_negotiation.cc:1036] Waiting for connection context
1209 08:58:29.526372 (+ 74us) server_negotiation.cc:300] Negotiation successful
1209 08:58:29.526412 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20251209 08:58:29.526813 26386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:58:29.524899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.49.65:33127 (local address 127.20.49.66:57083)
1209 08:58:29.525279 (+ 380us) negotiation.cc:107] Waiting for socket to connect
1209 08:58:29.525295 (+ 16us) client_negotiation.cc:174] Beginning negotiation
1209 08:58:29.525397 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1209 08:58:29.525654 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1209 08:58:29.525657 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1209 08:58:29.525697 (+ 40us) client_negotiation.cc:189] Negotiated authn=SASL
1209 08:58:29.526031 (+ 334us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
1209 08:58:29.526032 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
1209 08:58:29.526040 (+ 8us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
1209 08:58:29.526057 (+ 17us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1209 08:58:29.526287 (+ 230us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1209 08:58:29.526290 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1209 08:58:29.526292 (+ 2us) client_negotiation.cc:769] Sending connection context
1209 08:58:29.526376 (+ 84us) client_negotiation.cc:240] Negotiation successful
1209 08:58:29.526410 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":317,"thread_start_us":147,"threads_started":1}
I20251209 08:58:29.527179 26314 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "09d67a8cca7142fa9233b84c807e05e7" candidate_uuid: "a019260f6dcf40e5a792d27d93bb86b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e63e9c084d540329c00e5d311bc39ca" is_pre_election: true
I20251209 08:58:29.527442 26052 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "09d67a8cca7142fa9233b84c807e05e7" candidate_uuid: "a019260f6dcf40e5a792d27d93bb86b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "845b6c0baf6b4617a765c5894813e7d4" is_pre_election: true
I20251209 08:58:29.527504 26314 raft_consensus.cc:2468] T 09d67a8cca7142fa9233b84c807e05e7 P 7e63e9c084d540329c00e5d311bc39ca [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a019260f6dcf40e5a792d27d93bb86b3 in term 0.
I20251209 08:58:29.527679 26052 raft_consensus.cc:2468] T 09d67a8cca7142fa9233b84c807e05e7 P 845b6c0baf6b4617a765c5894813e7d4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a019260f6dcf40e5a792d27d93bb86b3 in term 0.
I20251209 08:58:29.527964 26119 leader_election.cc:304] T 09d67a8cca7142fa9233b84c807e05e7 P a019260f6dcf40e5a792d27d93bb86b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7e63e9c084d540329c00e5d311bc39ca, a019260f6dcf40e5a792d27d93bb86b3; no voters:
I20251209 08:58:29.528259 26384 raft_consensus.cc:2804] T 09d67a8cca7142fa9233b84c807e05e7 P a019260f6dcf40e5a792d27d93bb86b3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251209 08:58:29.528339 26384 raft_consensus.cc:493] T 09d67a8cca7142fa9233b84c807e05e7 P a019260f6dcf40e5a792d27d93bb86b3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251209 08:58:29.528373 26384 raft_consensus.cc:3060] T 09d67a8cca7142fa9233b84c807e05e7 P a019260f6dcf40e5a792d27d93bb86b3 [term 0 FOLLOWER]: Advancing to term 1
I20251209 08:58:29.529408 26384 raft_consensus.cc:515] T 09d67a8cca7142fa9233b84c807e05e7 P a019260f6dcf40e5a792d27d93bb86b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a019260f6dcf40e5a792d27d93bb86b3" member_type: VOTER last_known_addr { host: "127.20.49.66" port: 37777 } } peers { permanent_uuid: "7e63e9c084d540329c00e5d311bc39ca" member_type: VOTER last_known_addr { host: "127.20.49.67" port: 33297 } } peers { permanent_uuid: "845b6c0baf6b4617a765c5894813e7d4" member_type: VOTER last_known_addr { host: "127.20.49.65" port: 33127 } }
I20251209 08:58:29.529796 26384 leader_election.cc:290] T 09d67a8cca7142fa9233b84c807e05e7 P a019260f6dcf40e5a792d27d93bb86b3 [CANDIDATE]: Term 1 election: Requested vote from peers 7e63e9c084d540329c00e5d311bc39ca (127.20.49.67:33297), 845b6c0baf6b4617a765c5894813e7d4 (127.20.49.65:33127)
I20251209 08:58:29.530200 26314 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "09d67a8cca7142fa9233b84c807e05e7" candidate_uuid: "a019260f6dcf40e5a792d27d93bb86b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e63e9c084d540329c00e5d311bc39ca"
I20251209 08:58:29.530197 26052 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "09d67a8cca7142fa9233b84c807e05e7" candidate_uuid: "a019260f6dcf40e5a792d27d93bb86b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "845b6c0baf6b4617a765c5894813e7d4"
I20251209 08:58:29.530311 26314 raft_consensus.cc:3060] T 09d67a8cca7142fa9233b84c807e05e7 P 7e63e9c084d540329c00e5d311bc39ca [term 0 FOLLOWER]: Advancing to term 1
I20251209 08:58:29.530323 26052 raft_consensus.cc:3060] T 09d67a8cca7142fa9233b84c807e05e7 P 845b6c0baf6b4617a765c5894813e7d4 [term 0 FOLLOWER]: Advancing to term 1
I20251209 08:58:29.531845 26314 raft_consensus.cc:2468] T 09d67a8cca7142fa9233b84c807e05e7 P 7e63e9c084d540329c00e5d311bc39ca [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a019260f6dcf40e5a792d27d93bb86b3 in term 1.
I20251209 08:58:29.531911 26052 raft_consensus.cc:2468] T 09d67a8cca7142fa9233b84c807e05e7 P 845b6c0baf6b4617a765c5894813e7d4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a019260f6dcf40e5a792d27d93bb86b3 in term 1.
I20251209 08:58:29.532332 26119 leader_election.cc:304] T 09d67a8cca7142fa9233b84c807e05e7 P a019260f6dcf40e5a792d27d93bb86b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 845b6c0baf6b4617a765c5894813e7d4, a019260f6dcf40e5a792d27d93bb86b3; no voters:
I20251209 08:58:29.532547 26384 raft_consensus.cc:2804] T 09d67a8cca7142fa9233b84c807e05e7 P a019260f6dcf40e5a792d27d93bb86b3 [term 1 FOLLOWER]: Leader election won for term 1
I20251209 08:58:29.532774 26384 raft_consensus.cc:697] T 09d67a8cca7142fa9233b84c807e05e7 P a019260f6dcf40e5a792d27d93bb86b3 [term 1 LEADER]: Becoming Leader. State: Replica: a019260f6dcf40e5a792d27d93bb86b3, State: Running, Role: LEADER
I20251209 08:58:29.533113 26384 consensus_queue.cc:237] T 09d67a8cca7142fa9233b84c807e05e7 P a019260f6dcf40e5a792d27d93bb86b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a019260f6dcf40e5a792d27d93bb86b3" member_type: VOTER last_known_addr { host: "127.20.49.66" port: 37777 } } peers { permanent_uuid: "7e63e9c084d540329c00e5d311bc39ca" member_type: VOTER last_known_addr { host: "127.20.49.67" port: 33297 } } peers { permanent_uuid: "845b6c0baf6b4617a765c5894813e7d4" member_type: VOTER last_known_addr { host: "127.20.49.65" port: 33127 } }
I20251209 08:58:29.536196 25910 catalog_manager.cc:5654] T 09d67a8cca7142fa9233b84c807e05e7 P a019260f6dcf40e5a792d27d93bb86b3 reported cstate change: term changed from 0 to 1, leader changed from <none> to a019260f6dcf40e5a792d27d93bb86b3 (127.20.49.66). New cstate: current_term: 1 leader_uuid: "a019260f6dcf40e5a792d27d93bb86b3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a019260f6dcf40e5a792d27d93bb86b3" member_type: VOTER last_known_addr { host: "127.20.49.66" port: 37777 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7e63e9c084d540329c00e5d311bc39ca" member_type: VOTER last_known_addr { host: "127.20.49.67" port: 33297 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "845b6c0baf6b4617a765c5894813e7d4" member_type: VOTER last_known_addr { host: "127.20.49.65" port: 33127 } health_report { overall_health: UNKNOWN } } }
I20251209 08:58:29.561774 26373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:58:29.560337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39852 (local address 127.20.49.66:37777)
1209 08:58:29.560452 (+ 115us) server_negotiation.cc:207] Beginning negotiation
1209 08:58:29.560455 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1209 08:58:29.560469 (+ 14us) server_negotiation.cc:408] Connection header received
1209 08:58:29.560598 (+ 129us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1209 08:58:29.560600 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1209 08:58:29.560661 (+ 61us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1209 08:58:29.560736 (+ 75us) server_negotiation.cc:227] Negotiated authn=SASL
1209 08:58:29.561408 (+ 672us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1209 08:58:29.561412 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1209 08:58:29.561415 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1209 08:58:29.561428 (+ 13us) server_negotiation.cc:923] calling sasl_server_start()
1209 08:58:29.561448 (+ 20us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1209 08:58:29.561456 (+ 8us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1209 08:58:29.561532 (+ 76us) server_negotiation.cc:1036] Waiting for connection context
1209 08:58:29.561583 (+ 51us) server_negotiation.cc:300] Negotiation successful
1209 08:58:29.561618 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
W20251209 08:58:29.562598 26099 tablet.cc:2378] T 09d67a8cca7142fa9233b84c807e05e7 P 845b6c0baf6b4617a765c5894813e7d4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20251209 08:58:29.567420 26052 raft_consensus.cc:1275] T 09d67a8cca7142fa9233b84c807e05e7 P 845b6c0baf6b4617a765c5894813e7d4 [term 1 FOLLOWER]: Refusing update from remote peer a019260f6dcf40e5a792d27d93bb86b3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251209 08:58:29.567420 26314 raft_consensus.cc:1275] T 09d67a8cca7142fa9233b84c807e05e7 P 7e63e9c084d540329c00e5d311bc39ca [term 1 FOLLOWER]: Refusing update from remote peer a019260f6dcf40e5a792d27d93bb86b3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251209 08:58:29.568109 26387 consensus_queue.cc:1048] T 09d67a8cca7142fa9233b84c807e05e7 P a019260f6dcf40e5a792d27d93bb86b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "845b6c0baf6b4617a765c5894813e7d4" member_type: VOTER last_known_addr { host: "127.20.49.65" port: 33127 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251209 08:58:29.568295 26384 consensus_queue.cc:1048] T 09d67a8cca7142fa9233b84c807e05e7 P a019260f6dcf40e5a792d27d93bb86b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e63e9c084d540329c00e5d311bc39ca" member_type: VOTER last_known_addr { host: "127.20.49.67" port: 33297 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251209 08:58:29.576874 26391 mvcc.cc:204] Tried to move back new op lower bound from 7230548826380800000 to 7230548826250420224. Current Snapshot: MvccSnapshot[applied={T|T < 7230548826380800000}]
I20251209 08:58:29.577621 26390 mvcc.cc:204] Tried to move back new op lower bound from 7230548826380800000 to 7230548826250420224. Current Snapshot: MvccSnapshot[applied={T|T < 7230548826380800000}]
I20251209 08:58:29.586740 25910 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:37832:
table { table_name: "test-table" } modify_external_catalogs: true
I20251209 08:58:29.586895 25910 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:37832:
table { table_name: "test-table" } modify_external_catalogs: true
I20251209 08:58:29.589419 25910 catalog_manager.cc:5941] T 00000000000000000000000000000000 P 22e43c42a14645dd8b752a6f6042e31c: Sending DeleteTablet for 3 replicas of tablet 09d67a8cca7142fa9233b84c807e05e7
I20251209 08:58:29.590353 26294 tablet_service.cc:1552] Processing DeleteTablet for tablet 09d67a8cca7142fa9233b84c807e05e7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-09 08:58:29 UTC) from {username='slave'} at 127.0.0.1:44374
I20251209 08:58:29.590404 26032 tablet_service.cc:1552] Processing DeleteTablet for tablet 09d67a8cca7142fa9233b84c807e05e7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-09 08:58:29 UTC) from {username='slave'} at 127.0.0.1:52028
I20251209 08:58:29.590934 26401 tablet_replica.cc:333] T 09d67a8cca7142fa9233b84c807e05e7 P 7e63e9c084d540329c00e5d311bc39ca: stopping tablet replica
I20251209 08:58:29.590939 26400 tablet_replica.cc:333] T 09d67a8cca7142fa9233b84c807e05e7 P 845b6c0baf6b4617a765c5894813e7d4: stopping tablet replica
I20251209 08:58:29.590919 26163 tablet_service.cc:1552] Processing DeleteTablet for tablet 09d67a8cca7142fa9233b84c807e05e7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-09 08:58:29 UTC) from {username='slave'} at 127.0.0.1:39850
I20251209 08:58:29.591241 26401 raft_consensus.cc:2243] T 09d67a8cca7142fa9233b84c807e05e7 P 7e63e9c084d540329c00e5d311bc39ca [term 1 FOLLOWER]: Raft consensus shutting down.
I20251209 08:58:29.591383 26400 raft_consensus.cc:2243] T 09d67a8cca7142fa9233b84c807e05e7 P 845b6c0baf6b4617a765c5894813e7d4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251209 08:58:29.591605 26401 raft_consensus.cc:2272] T 09d67a8cca7142fa9233b84c807e05e7 P 7e63e9c084d540329c00e5d311bc39ca [term 1 FOLLOWER]: Raft consensus is shut down!
I20251209 08:58:29.591828 26400 raft_consensus.cc:2272] T 09d67a8cca7142fa9233b84c807e05e7 P 845b6c0baf6b4617a765c5894813e7d4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251209 08:58:29.592067 20677 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task8XuDZV/build/debug/bin/kudu with pid 25970
I20251209 08:58:29.592152 26402 tablet_replica.cc:333] T 09d67a8cca7142fa9233b84c807e05e7 P a019260f6dcf40e5a792d27d93bb86b3: stopping tablet replica
I20251209 08:58:29.592398 26402 raft_consensus.cc:2243] T 09d67a8cca7142fa9233b84c807e05e7 P a019260f6dcf40e5a792d27d93bb86b3 [term 1 LEADER]: Raft consensus shutting down.
I20251209 08:58:29.592887 26402 raft_consensus.cc:2272] T 09d67a8cca7142fa9233b84c807e05e7 P a019260f6dcf40e5a792d27d93bb86b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251209 08:58:29.599400 26401 ts_tablet_manager.cc:1916] T 09d67a8cca7142fa9233b84c807e05e7 P 7e63e9c084d540329c00e5d311bc39ca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251209 08:58:29.599400 26402 ts_tablet_manager.cc:1916] T 09d67a8cca7142fa9233b84c807e05e7 P a019260f6dcf40e5a792d27d93bb86b3: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251209 08:58:29.601383 25897 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.20.49.65:33127 (error 108)
I20251209 08:58:29.602253 26402 ts_tablet_manager.cc:1929] T 09d67a8cca7142fa9233b84c807e05e7 P a019260f6dcf40e5a792d27d93bb86b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251209 08:58:29.602298 20677 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task8XuDZV/build/debug/bin/kudu with pid 26101
I20251209 08:58:29.602378 26402 log.cc:1199] T 09d67a8cca7142fa9233b84c807e05e7 P a019260f6dcf40e5a792d27d93bb86b3: Deleting WAL directory at /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1765270668420570-20677-0/minicluster-data/ts-1/wal/wals/09d67a8cca7142fa9233b84c807e05e7
I20251209 08:58:29.603277 26401 ts_tablet_manager.cc:1929] T 09d67a8cca7142fa9233b84c807e05e7 P 7e63e9c084d540329c00e5d311bc39ca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251209 08:58:29.603387 26401 log.cc:1199] T 09d67a8cca7142fa9233b84c807e05e7 P 7e63e9c084d540329c00e5d311bc39ca: Deleting WAL directory at /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1765270668420570-20677-0/minicluster-data/ts-2/wal/wals/09d67a8cca7142fa9233b84c807e05e7
I20251209 08:58:29.603730 26401 ts_tablet_manager.cc:1950] T 09d67a8cca7142fa9233b84c807e05e7 P 7e63e9c084d540329c00e5d311bc39ca: Deleting consensus metadata
I20251209 08:58:29.604681 26372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:58:29.604358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.49.65:33127 (local address 127.0.0.1:52044)
1209 08:58:29.604503 (+ 145us) negotiation.cc:107] Waiting for socket to connect
1209 08:58:29.604582 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.49.65:33127: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20251209 08:58:29.604872 25897 catalog_manager.cc:4712] TS 845b6c0baf6b4617a765c5894813e7d4 (127.20.49.65:33127): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 09d67a8cca7142fa9233b84c807e05e7: Network error: Client connection negotiation failed: client connection to 127.20.49.65:33127: connect: Connection refused (error 111)
I20251209 08:58:29.605103 25897 catalog_manager.cc:4985] TS 7e63e9c084d540329c00e5d311bc39ca (127.20.49.67:33297): tablet 09d67a8cca7142fa9233b84c807e05e7 (table test-table [id=fbd6f12bff834b929c3d020931b1c46f]) successfully deleted
I20251209 08:58:29.609463 20677 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task8XuDZV/build/debug/bin/kudu with pid 26232
I20251209 08:58:29.610555 26372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:58:29.609654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.49.66:37777 (local address 127.0.0.1:39858)
1209 08:58:29.610375 (+ 721us) negotiation.cc:107] Waiting for socket to connect
1209 08:58:29.610442 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.49.66:37777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":602}
I20251209 08:58:29.616314 20677 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task8XuDZV/build/debug/bin/kudu with pid 25879
2025-12-09T08:58:29Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (981 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20251209 08:58:29.641909 20677 mini_oidc.cc:150] Starting JWKS server
I20251209 08:58:29.643281 20677 webserver.cc:492] Webserver started at https://127.0.0.1:40931/ using document root <none> and password file <none>
I20251209 08:58:29.643404 20677 mini_oidc.cc:186] Starting OIDC Discovery server
I20251209 08:58:29.643597 20677 webserver.cc:492] Webserver started at http://127.0.0.1:35607/ using document root <none> and password file <none>
2025-12-09T08:58:29Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-12-09T08:58:29Z Disabled control of system clock
I20251209 08:58:29.652992 20677 external_mini_cluster.cc:1366] Running /tmp/dist-test-task8XuDZV/build/debug/bin/kudu
/tmp/dist-test-task8XuDZV/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1765270668420570-20677-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1765270668420570-20677-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1765270668420570-20677-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1765270668420570-20677-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.49.126:45567
--webserver_interface=127.20.49.126
--webserver_port=0
--builtin_ntp_servers=127.20.49.84:38935
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.49.126:45567
--enable_jwt_token_auth=true
--jwks_url=https://localhost:40931/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1765270668420570-20677-0/testchainca.cert with env {}
W20251209 08:58:29.764715 26409 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251209 08:58:29.764977 26409 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251209 08:58:29.765031 26409 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251209 08:58:29.768482 26409 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251209 08:58:29.768556 26409 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251209 08:58:29.768573 26409 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251209 08:58:29.768594 26409 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251209 08:58:29.768611 26409 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20251209 08:58:29.768638 26409 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20251209 08:58:29.768671 26409 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:40931/jwks.json
I20251209 08:58:29.773173 26409 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.49.84:38935
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1765270668420570-20677-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1765270668420570-20677-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.49.126:45567
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.49.126:45567
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:40931/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1765270668420570-20677-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.49.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1765270668420570-20677-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26409
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1765270668420570-20677-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 468897755969d686fb08386f42f7835685e7953a
build type DEBUG
built by None at 09 Dec 2025 08:43:15 UTC on 24a791456cd2
build id 9426
I20251209 08:58:29.774338 26409 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251209 08:58:29.775204 26409 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251209 08:58:29.780829 26417 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251209 08:58:29.780848 26415 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251209 08:58:29.780825 26414 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251209 08:58:29.781148 26409 server_base.cc:1047] running on GCE node
I20251209 08:58:29.781741 26409 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251209 08:58:29.782691 26409 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251209 08:58:29.783948 26409 hybrid_clock.cc:648] HybridClock initialized: now 1765270709783928 us; error 37 us; skew 500 ppm
I20251209 08:58:29.785818 26409 webserver.cc:492] Webserver started at http://127.20.49.126:37947/ using document root <none> and password file <none>
I20251209 08:58:29.786440 26409 fs_manager.cc:362] Metadata directory not provided
I20251209 08:58:29.786502 26409 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251209 08:58:29.786691 26409 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251209 08:58:29.788388 26409 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1765270668420570-20677-0/minicluster-data/master-0/data/instance:
uuid: "ed4ba32e7f154e7982b8086045d96d88"
format_stamp: "Formatted at 2025-12-09 08:58:29 on dist-test-slave-8w6c"
server_key: "57f2c42137f751306ce43d7b45eb1d4f"
server_key_iv: "47c0d30b5a01b7f95c9da3df6a691619"
server_key_version: "encryptionkey@0"
I20251209 08:58:29.788877 26409 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1765270668420570-20677-0/minicluster-data/master-0/wal/instance:
uuid: "ed4ba32e7f154e7982b8086045d96d88"
format_stamp: "Formatted at 2025-12-09 08:58:29 on dist-test-slave-8w6c"
server_key: "57f2c42137f751306ce43d7b45eb1d4f"
server_key_iv: "47c0d30b5a01b7f95c9da3df6a691619"
server_key_version: "encryptionkey@0"
I20251209 08:58:29.792361 26409 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.003s
I20251209 08:58:29.794708 26423 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251209 08:58:29.795703 26409 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20251209 08:58:29.795814 26409 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1765270668420570-20677-0/minicluster-data/master-0/data,/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1765270668420570-20677-0/minicluster-data/master-0/wal
uuid: "ed4ba32e7f154e7982b8086045d96d88"
format_stamp: "Formatted at 2025-12-09 08:58:29 on dist-test-slave-8w6c"
server_key: "57f2c42137f751306ce43d7b45eb1d4f"
server_key_iv: "47c0d30b5a01b7f95c9da3df6a691619"
server_key_version: "encryptionkey@0"
I20251209 08:58:29.795921 26409 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1765270668420570-20677-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1765270668420570-20677-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1765270668420570-20677-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251209 08:58:29.808317 26409 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251209 08:58:29.808944 26409 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251209 08:58:29.809131 26409 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251209 08:58:29.816107 26409 rpc_server.cc:307] RPC server started. Bound to: 127.20.49.126:45567
I20251209 08:58:29.816113 26475 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.49.126:45567 every 8 connection(s)
I20251209 08:58:29.817126 26409 server_base.cc:1179] Dumped server information to /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1765270668420570-20677-0/minicluster-data/master-0/data/info.pb
I20251209 08:58:29.818958 20677 external_mini_cluster.cc:1428] Started /tmp/dist-test-task8XuDZV/build/debug/bin/kudu as pid 26409
I20251209 08:58:29.819103 20677 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1765270668420570-20677-0/minicluster-data/master-0/wal/instance
I20251209 08:58:29.819391 20677 external_mini_cluster.cc:1468] Setting key 7dd8ee0b1ddd7b1a46ce17516fc13765
I20251209 08:58:29.820307 26476 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251209 08:58:29.826045 26476 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ed4ba32e7f154e7982b8086045d96d88: Bootstrap starting.
I20251209 08:58:29.827224 26479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:58:29.820648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47106 (local address 127.20.49.126:45567)
1209 08:58:29.821270 (+ 622us) server_negotiation.cc:207] Beginning negotiation
1209 08:58:29.821279 (+ 9us) server_negotiation.cc:400] Waiting for connection header
1209 08:58:29.821314 (+ 35us) server_negotiation.cc:408] Connection header received
1209 08:58:29.822015 (+ 701us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1209 08:58:29.822035 (+ 20us) server_negotiation.cc:462] Received NEGOTIATE request from client
1209 08:58:29.822381 (+ 346us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1209 08:58:29.822694 (+ 313us) server_negotiation.cc:227] Negotiated authn=SASL
1209 08:58:29.823204 (+ 510us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:58:29.824069 (+ 865us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1209 08:58:29.824751 (+ 682us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:58:29.825065 (+ 314us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:58:29.825597 (+ 532us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1209 08:58:29.825639 (+ 42us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1209 08:58:29.825649 (+ 10us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1209 08:58:29.826082 (+ 433us) server_negotiation.cc:923] calling sasl_server_start()
1209 08:58:29.826135 (+ 53us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1209 08:58:29.826156 (+ 21us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1209 08:58:29.826271 (+ 115us) server_negotiation.cc:1036] Waiting for connection context
1209 08:58:29.826438 (+ 167us) server_negotiation.cc:300] Negotiation successful
1209 08:58:29.826654 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":275,"thread_start_us":128,"threads_started":1}
I20251209 08:58:29.828382 26476 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ed4ba32e7f154e7982b8086045d96d88: Neither blocks nor log segments found. Creating new log.
I20251209 08:58:29.829190 26476 log.cc:826] T 00000000000000000000000000000000 P ed4ba32e7f154e7982b8086045d96d88: Log is configured to *not* fsync() on all Append() calls
I20251209 08:58:29.830017 20677 external_mini_cluster.cc:949] 0 TS(s) registered with all masters
I20251209 08:58:29.831182 26476 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ed4ba32e7f154e7982b8086045d96d88: No bootstrap required, opened a new log
I20251209 08:58:29.834022 26476 raft_consensus.cc:359] T 00000000000000000000000000000000 P ed4ba32e7f154e7982b8086045d96d88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed4ba32e7f154e7982b8086045d96d88" member_type: VOTER last_known_addr { host: "127.20.49.126" port: 45567 } }
I20251209 08:58:29.834209 26476 raft_consensus.cc:385] T 00000000000000000000000000000000 P ed4ba32e7f154e7982b8086045d96d88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251209 08:58:29.834270 26476 raft_consensus.cc:740] T 00000000000000000000000000000000 P ed4ba32e7f154e7982b8086045d96d88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed4ba32e7f154e7982b8086045d96d88, State: Initialized, Role: FOLLOWER
I20251209 08:58:29.834617 26476 consensus_queue.cc:260] T 00000000000000000000000000000000 P ed4ba32e7f154e7982b8086045d96d88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed4ba32e7f154e7982b8086045d96d88" member_type: VOTER last_known_addr { host: "127.20.49.126" port: 45567 } }
I20251209 08:58:29.834722 26476 raft_consensus.cc:399] T 00000000000000000000000000000000 P ed4ba32e7f154e7982b8086045d96d88 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251209 08:58:29.834766 26476 raft_consensus.cc:493] T 00000000000000000000000000000000 P ed4ba32e7f154e7982b8086045d96d88 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251209 08:58:29.834829 26476 raft_consensus.cc:3060] T 00000000000000000000000000000000 P ed4ba32e7f154e7982b8086045d96d88 [term 0 FOLLOWER]: Advancing to term 1
I20251209 08:58:29.835644 26476 raft_consensus.cc:515] T 00000000000000000000000000000000 P ed4ba32e7f154e7982b8086045d96d88 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed4ba32e7f154e7982b8086045d96d88" member_type: VOTER last_known_addr { host: "127.20.49.126" port: 45567 } }
I20251209 08:58:29.835948 26476 leader_election.cc:304] T 00000000000000000000000000000000 P ed4ba32e7f154e7982b8086045d96d88 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ed4ba32e7f154e7982b8086045d96d88; no voters:
I20251209 08:58:29.836189 26476 leader_election.cc:290] T 00000000000000000000000000000000 P ed4ba32e7f154e7982b8086045d96d88 [CANDIDATE]: Term 1 election: Requested vote from peers
I20251209 08:58:29.836318 26481 raft_consensus.cc:2804] T 00000000000000000000000000000000 P ed4ba32e7f154e7982b8086045d96d88 [term 1 FOLLOWER]: Leader election won for term 1
I20251209 08:58:29.836520 26481 raft_consensus.cc:697] T 00000000000000000000000000000000 P ed4ba32e7f154e7982b8086045d96d88 [term 1 LEADER]: Becoming Leader. State: Replica: ed4ba32e7f154e7982b8086045d96d88, State: Running, Role: LEADER
I20251209 08:58:29.836797 26481 consensus_queue.cc:237] T 00000000000000000000000000000000 P ed4ba32e7f154e7982b8086045d96d88 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed4ba32e7f154e7982b8086045d96d88" member_type: VOTER last_known_addr { host: "127.20.49.126" port: 45567 } }
I20251209 08:58:29.837280 26476 sys_catalog.cc:565] T 00000000000000000000000000000000 P ed4ba32e7f154e7982b8086045d96d88 [sys.catalog]: configured and running, proceeding with master startup.
I20251209 08:58:29.838256 26482 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed4ba32e7f154e7982b8086045d96d88 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ed4ba32e7f154e7982b8086045d96d88" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed4ba32e7f154e7982b8086045d96d88" member_type: VOTER last_known_addr { host: "127.20.49.126" port: 45567 } } }
I20251209 08:58:29.838397 26482 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed4ba32e7f154e7982b8086045d96d88 [sys.catalog]: This master's current role is: LEADER
I20251209 08:58:29.838850 26490 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251209 08:58:29.839120 26483 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed4ba32e7f154e7982b8086045d96d88 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ed4ba32e7f154e7982b8086045d96d88. Latest consensus state: current_term: 1 leader_uuid: "ed4ba32e7f154e7982b8086045d96d88" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed4ba32e7f154e7982b8086045d96d88" member_type: VOTER last_known_addr { host: "127.20.49.126" port: 45567 } } }
I20251209 08:58:29.839221 26483 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed4ba32e7f154e7982b8086045d96d88 [sys.catalog]: This master's current role is: LEADER
I20251209 08:58:29.842095 26490 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251209 08:58:29.847524 26490 catalog_manager.cc:1357] Generated new cluster ID: b87a2a4643ac496f8046d9018b1ed944
I20251209 08:58:29.847604 26490 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251209 08:58:29.859545 26490 catalog_manager.cc:1380] Generated new certificate authority record
I20251209 08:58:29.860682 26490 catalog_manager.cc:1514] Loading token signing keys...
I20251209 08:58:29.865729 26490 catalog_manager.cc:6027] T 00000000000000000000000000000000 P ed4ba32e7f154e7982b8086045d96d88: Generated new TSK 0
I20251209 08:58:29.866606 26490 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251209 08:58:29.902379 26479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:58:29.878660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47116 (local address 127.20.49.126:45567)
1209 08:58:29.878807 (+ 147us) server_negotiation.cc:207] Beginning negotiation
1209 08:58:29.878811 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1209 08:58:29.878947 (+ 136us) server_negotiation.cc:408] Connection header received
1209 08:58:29.879183 (+ 236us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1209 08:58:29.879186 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1209 08:58:29.879237 (+ 51us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1209 08:58:29.879340 (+ 103us) server_negotiation.cc:227] Negotiated authn=JWT
1209 08:58:29.879885 (+ 545us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:58:29.880436 (+ 551us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1209 08:58:29.881371 (+ 935us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:58:29.881534 (+ 163us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:58:29.881628 (+ 94us) server_negotiation.cc:366] Received JWT_EXCHANGE NegotiatePB request
1209 08:58:29.901844 (+ 20216us) server_negotiation.cc:378] Sending JWT_EXCHANGE NegotiatePB response
1209 08:58:29.901949 (+ 105us) server_negotiation.cc:1036] Waiting for connection context
1209 08:58:29.902164 (+ 215us) server_negotiation.cc:300] Negotiation successful
1209 08:58:29.902225 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60,"thread_start_us":158,"threads_started":1}
I20251209 08:58:29.913936 26479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:58:29.910167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47120 (local address 127.20.49.126:45567)
1209 08:58:29.910305 (+ 138us) server_negotiation.cc:207] Beginning negotiation
1209 08:58:29.910308 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1209 08:58:29.910411 (+ 103us) server_negotiation.cc:408] Connection header received
1209 08:58:29.910595 (+ 184us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1209 08:58:29.910598 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1209 08:58:29.910646 (+ 48us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1209 08:58:29.910725 (+ 79us) server_negotiation.cc:227] Negotiated authn=JWT
1209 08:58:29.911177 (+ 452us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:58:29.911698 (+ 521us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1209 08:58:29.912583 (+ 885us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:58:29.912721 (+ 138us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:58:29.912763 (+ 42us) server_negotiation.cc:366] Received JWT_EXCHANGE NegotiatePB request
1209 08:58:29.913584 (+ 821us) server_negotiation.cc:395] Sending RPC error: FATAL_INVALID_JWT: Not authorized: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:../crypto/rsa/rsa_pk1.c:67 error:04067072:rsa routines:rsa_ossl_public_decrypt:padding check failed:../crypto/rsa/rsa_ossl.c:581)
1209 08:58:29.913837 (+ 253us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:47120: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:../crypto/rsa/rsa_pk1.c:67 error:04067072:rsa routines:rsa_ossl_public_decrypt:padding check failed:../crypto/rsa/rsa_ossl.c:581)
Metrics: {"server-negotiator.queue_time_us":55}
W20251209 08:58:29.914043 26479 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:47120: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:../crypto/rsa/rsa_pk1.c:67 error:04067072:rsa routines:rsa_ossl_public_decrypt:padding check failed:../crypto/rsa/rsa_ossl.c:581)
W20251209 08:58:29.914815 26512 negotiation.cc:337] Failed RPC negotiation. Trace:
1209 08:58:29.910030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.49.126:45567 (local address 127.0.0.1:47120)
1209 08:58:29.910320 (+ 290us) negotiation.cc:107] Waiting for socket to connect
1209 08:58:29.910335 (+ 15us) client_negotiation.cc:174] Beginning negotiation
1209 08:58:29.910452 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1209 08:58:29.910746 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1209 08:58:29.910749 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1209 08:58:29.910757 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
1209 08:58:29.911035 (+ 278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:58:29.911045 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:58:29.911817 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1209 08:58:29.911821 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1209 08:58:29.912475 (+ 654us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:58:29.912485 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:58:29.912606 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:58:29.912630 (+ 24us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
1209 08:58:29.913816 (+ 1186us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:../crypto/rsa/rsa_pk1.c:67 error:04067072:rsa routines:rsa_ossl_public_decrypt:padding check failed:../crypto/rsa/rsa_ossl.c:581)
1209 08:58:29.913984 (+ 168us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.20.49.126:45567: FATAL_INVALID_JWT: Not authorized: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:../crypto/rsa/rsa_pk1.c:67 error:04067072:rsa routines:rsa_ossl_public_decrypt:padding check failed:../crypto/rsa/rsa_ossl.c:581)
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":114,"threads_started":1}
I20251209 08:58:44.923877 26518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:58:44.920407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34756 (local address 127.20.49.126:45567)
1209 08:58:44.920720 (+ 313us) server_negotiation.cc:207] Beginning negotiation
1209 08:58:44.920727 (+ 7us) server_negotiation.cc:400] Waiting for connection header
1209 08:58:44.920746 (+ 19us) server_negotiation.cc:408] Connection header received
1209 08:58:44.920798 (+ 52us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1209 08:58:44.920800 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1209 08:58:44.920844 (+ 44us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1209 08:58:44.920925 (+ 81us) server_negotiation.cc:227] Negotiated authn=JWT
1209 08:58:44.921343 (+ 418us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:58:44.921927 (+ 584us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1209 08:58:44.922626 (+ 699us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:58:44.922818 (+ 192us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:58:44.922869 (+ 51us) server_negotiation.cc:366] Received JWT_EXCHANGE NegotiatePB request
1209 08:58:44.923523 (+ 654us) server_negotiation.cc:395] Sending RPC error: FATAL_INVALID_JWT: Not authorized: JWT verification failed: token expired
1209 08:58:44.923713 (+ 190us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:34756: JWT verification failed: token expired
Metrics: {"server-negotiator.queue_time_us":208,"thread_start_us":83,"threads_started":1}
W20251209 08:58:44.923944 26517 negotiation.cc:337] Failed RPC negotiation. Trace:
1209 08:58:44.920212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.49.126:45567 (local address 127.0.0.1:34756)
1209 08:58:44.920504 (+ 292us) negotiation.cc:107] Waiting for socket to connect
1209 08:58:44.920520 (+ 16us) client_negotiation.cc:174] Beginning negotiation
1209 08:58:44.920650 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1209 08:58:44.920959 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1209 08:58:44.920963 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
1209 08:58:44.920971 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
1209 08:58:44.921200 (+ 229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:58:44.921208 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:58:44.922063 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1209 08:58:44.922067 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1209 08:58:44.922495 (+ 428us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:58:44.922503 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:58:44.922624 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:58:44.922653 (+ 29us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
1209 08:58:44.923660 (+ 1007us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: JWT verification failed: token expired
1209 08:58:44.923803 (+ 143us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.20.49.126:45567: FATAL_INVALID_JWT: Not authorized: JWT verification failed: token expired
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":108,"threads_started":1}
W20251209 08:58:44.923964 26518 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:34756: JWT verification failed: token expired
W20251209 08:58:44.926769 26518 negotiation.cc:337] Failed RPC negotiation. Trace:
1209 08:58:44.925890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34772 (local address 127.20.49.126:45567)
1209 08:58:44.926062 (+ 172us) server_negotiation.cc:207] Beginning negotiation
1209 08:58:44.926067 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1209 08:58:44.926167 (+ 100us) server_negotiation.cc:408] Connection header received
1209 08:58:44.926292 (+ 125us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1209 08:58:44.926295 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1209 08:58:44.926349 (+ 54us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1209 08:58:44.926427 (+ 78us) server_negotiation.cc:227] Negotiated authn=SASL
1209 08:58:44.926657 (+ 230us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34772: BlockingRecv error: recv got EOF from 127.0.0.1:34772 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
W20251209 08:58:44.930248 26528 client_negotiation.cc:375] the client has a JWT but it isn't advertising its JWT-based authn capability since it doesn't trust the certificate of the server at 127.20.49.126:45567
W20251209 08:58:44.930847 26518 negotiation.cc:337] Failed RPC negotiation. Trace:
1209 08:58:44.929972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34776 (local address 127.20.49.126:45567)
1209 08:58:44.930116 (+ 144us) server_negotiation.cc:207] Beginning negotiation
1209 08:58:44.930120 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1209 08:58:44.930203 (+ 83us) server_negotiation.cc:408] Connection header received
1209 08:58:44.930410 (+ 207us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1209 08:58:44.930412 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1209 08:58:44.930457 (+ 45us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1209 08:58:44.930539 (+ 82us) server_negotiation.cc:227] Negotiated authn=SASL
1209 08:58:44.930742 (+ 203us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34776: BlockingRecv error: recv got EOF from 127.0.0.1:34776 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
I20251209 08:58:44.931139 20677 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task8XuDZV/build/debug/bin/kudu with pid 26409
2025-12-09T08:58:44Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniCluster (15331 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20251209 08:58:44.973398 20677 mini_oidc.cc:150] Starting JWKS server
I20251209 08:58:44.974385 20677 webserver.cc:492] Webserver started at https://127.0.0.1:34495/ using document root <none> and password file <none>
I20251209 08:58:44.974473 20677 mini_oidc.cc:186] Starting OIDC Discovery server
I20251209 08:58:44.974707 20677 webserver.cc:492] Webserver started at http://127.0.0.1:34253/ using document root <none> and password file <none>
2025-12-09T08:58:44Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-12-09T08:58:44Z Disabled control of system clock
I20251209 08:58:44.983840 20677 external_mini_cluster.cc:1366] Running /tmp/dist-test-task8XuDZV/build/debug/bin/kudu
/tmp/dist-test-task8XuDZV/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1765270668420570-20677-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1765270668420570-20677-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1765270668420570-20677-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1765270668420570-20677-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.49.126:40909
--webserver_interface=127.20.49.126
--webserver_port=0
--builtin_ntp_servers=127.20.49.84:46321
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.49.126:40909
--enable_jwt_token_auth=true
--jwks_url=https://localhost:34495/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1765270668420570-20677-0/testchainca.cert with env {}
W20251209 08:58:45.095254 26534 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251209 08:58:45.095683 26534 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251209 08:58:45.095808 26534 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251209 08:58:45.099543 26534 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251209 08:58:45.099611 26534 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251209 08:58:45.099628 26534 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251209 08:58:45.099648 26534 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251209 08:58:45.099668 26534 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20251209 08:58:45.099705 26534 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20251209 08:58:45.099733 26534 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:34495/jwks.json
I20251209 08:58:45.104319 26534 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.49.84:46321
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1765270668420570-20677-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1765270668420570-20677-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.49.126:40909
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.49.126:40909
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:34495/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1765270668420570-20677-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.49.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1765270668420570-20677-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26534
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1765270668420570-20677-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 468897755969d686fb08386f42f7835685e7953a
build type DEBUG
built by None at 09 Dec 2025 08:43:15 UTC on 24a791456cd2
build id 9426
I20251209 08:58:45.105481 26534 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251209 08:58:45.106443 26534 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251209 08:58:45.112102 26542 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251209 08:58:45.112108 26539 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251209 08:58:45.112120 26540 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251209 08:58:45.112587 26534 server_base.cc:1047] running on GCE node
I20251209 08:58:45.113044 26534 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251209 08:58:45.113996 26534 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251209 08:58:45.115157 26534 hybrid_clock.cc:648] HybridClock initialized: now 1765270725115136 us; error 45 us; skew 500 ppm
I20251209 08:58:45.117070 26534 webserver.cc:492] Webserver started at http://127.20.49.126:40763/ using document root <none> and password file <none>
I20251209 08:58:45.117646 26534 fs_manager.cc:362] Metadata directory not provided
I20251209 08:58:45.117745 26534 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251209 08:58:45.117950 26534 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251209 08:58:45.119673 26534 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1765270668420570-20677-0/minicluster-data/master-0/data/instance:
uuid: "90994d55e9624fe198ce42e1b92137d8"
format_stamp: "Formatted at 2025-12-09 08:58:45 on dist-test-slave-8w6c"
server_key: "8c5989f0b3f8b7416ebeb091c86781b8"
server_key_iv: "8b093640f804b8ea877533826d9298c8"
server_key_version: "encryptionkey@0"
I20251209 08:58:45.120162 26534 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1765270668420570-20677-0/minicluster-data/master-0/wal/instance:
uuid: "90994d55e9624fe198ce42e1b92137d8"
format_stamp: "Formatted at 2025-12-09 08:58:45 on dist-test-slave-8w6c"
server_key: "8c5989f0b3f8b7416ebeb091c86781b8"
server_key_iv: "8b093640f804b8ea877533826d9298c8"
server_key_version: "encryptionkey@0"
I20251209 08:58:45.123720 26534 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20251209 08:58:45.126020 26548 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251209 08:58:45.127151 26534 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20251209 08:58:45.127287 26534 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1765270668420570-20677-0/minicluster-data/master-0/data,/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1765270668420570-20677-0/minicluster-data/master-0/wal
uuid: "90994d55e9624fe198ce42e1b92137d8"
format_stamp: "Formatted at 2025-12-09 08:58:45 on dist-test-slave-8w6c"
server_key: "8c5989f0b3f8b7416ebeb091c86781b8"
server_key_iv: "8b093640f804b8ea877533826d9298c8"
server_key_version: "encryptionkey@0"
I20251209 08:58:45.127395 26534 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1765270668420570-20677-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1765270668420570-20677-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1765270668420570-20677-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251209 08:58:45.146672 26534 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251209 08:58:45.147413 26534 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251209 08:58:45.147604 26534 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251209 08:58:45.155376 26600 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.49.126:40909 every 8 connection(s)
I20251209 08:58:45.155380 26534 rpc_server.cc:307] RPC server started. Bound to: 127.20.49.126:40909
I20251209 08:58:45.156581 26534 server_base.cc:1179] Dumped server information to /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1765270668420570-20677-0/minicluster-data/master-0/data/info.pb
I20251209 08:58:45.159516 20677 external_mini_cluster.cc:1428] Started /tmp/dist-test-task8XuDZV/build/debug/bin/kudu as pid 26534
I20251209 08:58:45.159662 20677 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1765270668420570-20677-0/minicluster-data/master-0/wal/instance
I20251209 08:58:45.159890 26601 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251209 08:58:45.159941 20677 external_mini_cluster.cc:1468] Setting key a673a3da99d29d6b44949abbe24dab92
I20251209 08:58:45.166595 26601 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 90994d55e9624fe198ce42e1b92137d8: Bootstrap starting.
I20251209 08:58:45.168879 26601 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 90994d55e9624fe198ce42e1b92137d8: Neither blocks nor log segments found. Creating new log.
I20251209 08:58:45.169647 26601 log.cc:826] T 00000000000000000000000000000000 P 90994d55e9624fe198ce42e1b92137d8: Log is configured to *not* fsync() on all Append() calls
I20251209 08:58:45.171157 26604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:58:45.161153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58488 (local address 127.20.49.126:40909)
1209 08:58:45.161889 (+ 736us) server_negotiation.cc:207] Beginning negotiation
1209 08:58:45.161898 (+ 9us) server_negotiation.cc:400] Waiting for connection header
1209 08:58:45.161937 (+ 39us) server_negotiation.cc:408] Connection header received
1209 08:58:45.162765 (+ 828us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1209 08:58:45.162787 (+ 22us) server_negotiation.cc:462] Received NEGOTIATE request from client
1209 08:58:45.163144 (+ 357us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1209 08:58:45.163483 (+ 339us) server_negotiation.cc:227] Negotiated authn=SASL
1209 08:58:45.164012 (+ 529us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:58:45.164792 (+ 780us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1209 08:58:45.165500 (+ 708us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:58:45.165836 (+ 336us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:58:45.169356 (+ 3520us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1209 08:58:45.169374 (+ 18us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1209 08:58:45.169385 (+ 11us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1209 08:58:45.169839 (+ 454us) server_negotiation.cc:923] calling sasl_server_start()
1209 08:58:45.169891 (+ 52us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1209 08:58:45.169908 (+ 17us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1209 08:58:45.170016 (+ 108us) server_negotiation.cc:1036] Waiting for connection context
1209 08:58:45.170196 (+ 180us) server_negotiation.cc:300] Negotiation successful
1209 08:58:45.170451 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":334,"thread_start_us":144,"threads_started":1}
I20251209 08:58:45.171801 26601 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 90994d55e9624fe198ce42e1b92137d8: No bootstrap required, opened a new log
I20251209 08:58:45.173898 20677 external_mini_cluster.cc:949] 0 TS(s) registered with all masters
I20251209 08:58:45.174690 26601 raft_consensus.cc:359] T 00000000000000000000000000000000 P 90994d55e9624fe198ce42e1b92137d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90994d55e9624fe198ce42e1b92137d8" member_type: VOTER last_known_addr { host: "127.20.49.126" port: 40909 } }
I20251209 08:58:45.174923 26601 raft_consensus.cc:385] T 00000000000000000000000000000000 P 90994d55e9624fe198ce42e1b92137d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251209 08:58:45.174996 26601 raft_consensus.cc:740] T 00000000000000000000000000000000 P 90994d55e9624fe198ce42e1b92137d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90994d55e9624fe198ce42e1b92137d8, State: Initialized, Role: FOLLOWER
I20251209 08:58:45.175452 26601 consensus_queue.cc:260] T 00000000000000000000000000000000 P 90994d55e9624fe198ce42e1b92137d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90994d55e9624fe198ce42e1b92137d8" member_type: VOTER last_known_addr { host: "127.20.49.126" port: 40909 } }
I20251209 08:58:45.175592 26601 raft_consensus.cc:399] T 00000000000000000000000000000000 P 90994d55e9624fe198ce42e1b92137d8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251209 08:58:45.175657 26601 raft_consensus.cc:493] T 00000000000000000000000000000000 P 90994d55e9624fe198ce42e1b92137d8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251209 08:58:45.175753 26601 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 90994d55e9624fe198ce42e1b92137d8 [term 0 FOLLOWER]: Advancing to term 1
I20251209 08:58:45.176599 26601 raft_consensus.cc:515] T 00000000000000000000000000000000 P 90994d55e9624fe198ce42e1b92137d8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90994d55e9624fe198ce42e1b92137d8" member_type: VOTER last_known_addr { host: "127.20.49.126" port: 40909 } }
I20251209 08:58:45.176918 26601 leader_election.cc:304] T 00000000000000000000000000000000 P 90994d55e9624fe198ce42e1b92137d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 90994d55e9624fe198ce42e1b92137d8; no voters:
I20251209 08:58:45.177212 26601 leader_election.cc:290] T 00000000000000000000000000000000 P 90994d55e9624fe198ce42e1b92137d8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20251209 08:58:45.177412 26606 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 90994d55e9624fe198ce42e1b92137d8 [term 1 FOLLOWER]: Leader election won for term 1
I20251209 08:58:45.177611 26606 raft_consensus.cc:697] T 00000000000000000000000000000000 P 90994d55e9624fe198ce42e1b92137d8 [term 1 LEADER]: Becoming Leader. State: Replica: 90994d55e9624fe198ce42e1b92137d8, State: Running, Role: LEADER
I20251209 08:58:45.177927 26606 consensus_queue.cc:237] T 00000000000000000000000000000000 P 90994d55e9624fe198ce42e1b92137d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90994d55e9624fe198ce42e1b92137d8" member_type: VOTER last_known_addr { host: "127.20.49.126" port: 40909 } }
I20251209 08:58:45.178406 26601 sys_catalog.cc:565] T 00000000000000000000000000000000 P 90994d55e9624fe198ce42e1b92137d8 [sys.catalog]: configured and running, proceeding with master startup.
I20251209 08:58:45.179570 26607 sys_catalog.cc:455] T 00000000000000000000000000000000 P 90994d55e9624fe198ce42e1b92137d8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "90994d55e9624fe198ce42e1b92137d8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90994d55e9624fe198ce42e1b92137d8" member_type: VOTER last_known_addr { host: "127.20.49.126" port: 40909 } } }
I20251209 08:58:45.179570 26608 sys_catalog.cc:455] T 00000000000000000000000000000000 P 90994d55e9624fe198ce42e1b92137d8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 90994d55e9624fe198ce42e1b92137d8. Latest consensus state: current_term: 1 leader_uuid: "90994d55e9624fe198ce42e1b92137d8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90994d55e9624fe198ce42e1b92137d8" member_type: VOTER last_known_addr { host: "127.20.49.126" port: 40909 } } }
I20251209 08:58:45.179816 26608 sys_catalog.cc:458] T 00000000000000000000000000000000 P 90994d55e9624fe198ce42e1b92137d8 [sys.catalog]: This master's current role is: LEADER
I20251209 08:58:45.179817 26607 sys_catalog.cc:458] T 00000000000000000000000000000000 P 90994d55e9624fe198ce42e1b92137d8 [sys.catalog]: This master's current role is: LEADER
I20251209 08:58:45.180224 26615 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251209 08:58:45.183066 26615 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251209 08:58:45.188570 26615 catalog_manager.cc:1357] Generated new cluster ID: 64c4140781ee4350bb6a0bfc9c93cdcf
I20251209 08:58:45.188656 26615 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251209 08:58:45.204368 26615 catalog_manager.cc:1380] Generated new certificate authority record
I20251209 08:58:45.205667 26615 catalog_manager.cc:1514] Loading token signing keys...
I20251209 08:58:45.213737 26615 catalog_manager.cc:6027] T 00000000000000000000000000000000 P 90994d55e9624fe198ce42e1b92137d8: Generated new TSK 0
I20251209 08:58:45.214720 26615 catalog_manager.cc:1524] Initializing in-progress tserver states...
E20251209 08:58:45.243969 26631 webserver.cc:578] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20251209 08:58:45.244068 26631 webserver.cc:578] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20251209 08:58:45.244735 26604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:58:45.227994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58498 (local address 127.20.49.126:40909)
1209 08:58:45.228130 (+ 136us) server_negotiation.cc:207] Beginning negotiation
1209 08:58:45.228134 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1209 08:58:45.228229 (+ 95us) server_negotiation.cc:408] Connection header received
1209 08:58:45.228419 (+ 190us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1209 08:58:45.228422 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1209 08:58:45.228493 (+ 71us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1209 08:58:45.228583 (+ 90us) server_negotiation.cc:227] Negotiated authn=JWT
1209 08:58:45.229102 (+ 519us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:58:45.230226 (+ 1124us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1209 08:58:45.231159 (+ 933us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:58:45.231345 (+ 186us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:58:45.231445 (+ 100us) server_negotiation.cc:366] Received JWT_EXCHANGE NegotiatePB request
1209 08:58:45.244364 (+ 12919us) server_negotiation.cc:395] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34495/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
1209 08:58:45.244623 (+ 259us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:58498: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34495/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":60}
W20251209 08:58:45.244810 26604 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:58498: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34495/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20251209 08:58:45.244980 26630 negotiation.cc:337] Failed RPC negotiation. Trace:
1209 08:58:45.227839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.49.126:40909 (local address 127.0.0.1:58498)
1209 08:58:45.228132 (+ 293us) negotiation.cc:107] Waiting for socket to connect
1209 08:58:45.228150 (+ 18us) client_negotiation.cc:174] Beginning negotiation
1209 08:58:45.228288 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1209 08:58:45.228607 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1209 08:58:45.228611 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
1209 08:58:45.228620 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
1209 08:58:45.228934 (+ 314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:58:45.228944 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:58:45.230375 (+ 1431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1209 08:58:45.230380 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1209 08:58:45.231047 (+ 667us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:58:45.231059 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:58:45.231188 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:58:45.231214 (+ 26us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
1209 08:58:45.244596 (+ 13382us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34495/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
1209 08:58:45.244787 (+ 191us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.20.49.126:40909: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34495/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":109,"threads_started":1}
I20251209 08:58:45.245801 20677 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task8XuDZV/build/debug/bin/kudu with pid 26534
2025-12-09T08:58:45Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (312 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20251209 08:58:45.285477 20677 mini_oidc.cc:150] Starting JWKS server
I20251209 08:58:45.286545 20677 webserver.cc:492] Webserver started at https://127.0.0.1:44697/ using document root <none> and password file <none>
I20251209 08:58:45.286638 20677 mini_oidc.cc:186] Starting OIDC Discovery server
I20251209 08:58:45.286875 20677 webserver.cc:492] Webserver started at http://127.0.0.1:42859/ using document root <none> and password file <none>
2025-12-09T08:58:45Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-12-09T08:58:45Z Disabled control of system clock
I20251209 08:58:45.295603 20677 external_mini_cluster.cc:1366] Running /tmp/dist-test-task8XuDZV/build/debug/bin/kudu
/tmp/dist-test-task8XuDZV/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1765270668420570-20677-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1765270668420570-20677-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1765270668420570-20677-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1765270668420570-20677-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.49.126:44263
--webserver_interface=127.20.49.126
--webserver_port=0
--builtin_ntp_servers=127.20.49.84:45557
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.49.126:44263
--enable_jwt_token_auth=true
--jwks_url=https://localhost:44697/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20251209 08:58:45.408105 26637 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251209 08:58:45.408362 26637 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251209 08:58:45.408414 26637 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251209 08:58:45.411823 26637 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251209 08:58:45.411895 26637 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251209 08:58:45.411912 26637 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251209 08:58:45.411935 26637 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251209 08:58:45.411953 26637 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20251209 08:58:45.411971 26637 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20251209 08:58:45.411985 26637 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:44697/jwks.json
I20251209 08:58:45.416016 26637 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.49.84:45557
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1765270668420570-20677-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1765270668420570-20677-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.49.126:44263
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.49.126:44263
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:44697/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1765270668420570-20677-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.49.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26637
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1765270668420570-20677-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 468897755969d686fb08386f42f7835685e7953a
build type DEBUG
built by None at 09 Dec 2025 08:43:15 UTC on 24a791456cd2
build id 9426
I20251209 08:58:45.417119 26637 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251209 08:58:45.418313 26637 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251209 08:58:45.423910 26643 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251209 08:58:45.423916 26642 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251209 08:58:45.424137 26637 server_base.cc:1047] running on GCE node
W20251209 08:58:45.423916 26645 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251209 08:58:45.424642 26637 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251209 08:58:45.425529 26637 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251209 08:58:45.426682 26637 hybrid_clock.cc:648] HybridClock initialized: now 1765270725426684 us; error 38 us; skew 500 ppm
I20251209 08:58:45.428491 26637 webserver.cc:492] Webserver started at http://127.20.49.126:43585/ using document root <none> and password file <none>
I20251209 08:58:45.429049 26637 fs_manager.cc:362] Metadata directory not provided
I20251209 08:58:45.429111 26637 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251209 08:58:45.429322 26637 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251209 08:58:45.431054 26637 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1765270668420570-20677-0/minicluster-data/master-0/data/instance:
uuid: "e1282d4177bf4858ba4c63982c6b664d"
format_stamp: "Formatted at 2025-12-09 08:58:45 on dist-test-slave-8w6c"
server_key: "46ca1fb588c61c104713ae2be0fb73aa"
server_key_iv: "8c0a858834ea00d09d6c6390c7a50d0e"
server_key_version: "encryptionkey@0"
I20251209 08:58:45.431537 26637 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1765270668420570-20677-0/minicluster-data/master-0/wal/instance:
uuid: "e1282d4177bf4858ba4c63982c6b664d"
format_stamp: "Formatted at 2025-12-09 08:58:45 on dist-test-slave-8w6c"
server_key: "46ca1fb588c61c104713ae2be0fb73aa"
server_key_iv: "8c0a858834ea00d09d6c6390c7a50d0e"
server_key_version: "encryptionkey@0"
I20251209 08:58:45.434839 26637 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.002s
I20251209 08:58:45.437041 26651 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251209 08:58:45.438047 26637 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20251209 08:58:45.438156 26637 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1765270668420570-20677-0/minicluster-data/master-0/data,/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1765270668420570-20677-0/minicluster-data/master-0/wal
uuid: "e1282d4177bf4858ba4c63982c6b664d"
format_stamp: "Formatted at 2025-12-09 08:58:45 on dist-test-slave-8w6c"
server_key: "46ca1fb588c61c104713ae2be0fb73aa"
server_key_iv: "8c0a858834ea00d09d6c6390c7a50d0e"
server_key_version: "encryptionkey@0"
I20251209 08:58:45.438263 26637 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1765270668420570-20677-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1765270668420570-20677-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1765270668420570-20677-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251209 08:58:45.479534 26637 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251209 08:58:45.480232 26637 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251209 08:58:45.480430 26637 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251209 08:58:45.487756 26637 rpc_server.cc:307] RPC server started. Bound to: 127.20.49.126:44263
I20251209 08:58:45.487767 26703 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.49.126:44263 every 8 connection(s)
I20251209 08:58:45.488792 26637 server_base.cc:1179] Dumped server information to /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1765270668420570-20677-0/minicluster-data/master-0/data/info.pb
I20251209 08:58:45.491429 20677 external_mini_cluster.cc:1428] Started /tmp/dist-test-task8XuDZV/build/debug/bin/kudu as pid 26637
I20251209 08:58:45.491560 20677 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1765270668420570-20677-0/minicluster-data/master-0/wal/instance
I20251209 08:58:45.491829 20677 external_mini_cluster.cc:1468] Setting key 6ce0359fa2ec363a6d398401cad15980
I20251209 08:58:45.491945 26704 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251209 08:58:45.498281 26704 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e1282d4177bf4858ba4c63982c6b664d: Bootstrap starting.
I20251209 08:58:45.499651 26707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:58:45.493111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55390 (local address 127.20.49.126:44263)
1209 08:58:45.493782 (+ 671us) server_negotiation.cc:207] Beginning negotiation
1209 08:58:45.493790 (+ 8us) server_negotiation.cc:400] Waiting for connection header
1209 08:58:45.493824 (+ 34us) server_negotiation.cc:408] Connection header received
1209 08:58:45.494472 (+ 648us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1209 08:58:45.494490 (+ 18us) server_negotiation.cc:462] Received NEGOTIATE request from client
1209 08:58:45.494825 (+ 335us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1209 08:58:45.495139 (+ 314us) server_negotiation.cc:227] Negotiated authn=SASL
1209 08:58:45.495679 (+ 540us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:58:45.496410 (+ 731us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1209 08:58:45.497096 (+ 686us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:58:45.497402 (+ 306us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:58:45.498014 (+ 612us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1209 08:58:45.498059 (+ 45us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1209 08:58:45.498069 (+ 10us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1209 08:58:45.498554 (+ 485us) server_negotiation.cc:923] calling sasl_server_start()
1209 08:58:45.498596 (+ 42us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1209 08:58:45.498611 (+ 15us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1209 08:58:45.498712 (+ 101us) server_negotiation.cc:1036] Waiting for connection context
1209 08:58:45.498870 (+ 158us) server_negotiation.cc:300] Negotiation successful
1209 08:58:45.499086 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":299,"thread_start_us":150,"threads_started":1}
I20251209 08:58:45.500528 26704 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e1282d4177bf4858ba4c63982c6b664d: Neither blocks nor log segments found. Creating new log.
I20251209 08:58:45.501368 26704 log.cc:826] T 00000000000000000000000000000000 P e1282d4177bf4858ba4c63982c6b664d: Log is configured to *not* fsync() on all Append() calls
I20251209 08:58:45.502570 20677 external_mini_cluster.cc:949] 0 TS(s) registered with all masters
I20251209 08:58:45.503492 26704 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e1282d4177bf4858ba4c63982c6b664d: No bootstrap required, opened a new log
I20251209 08:58:45.506089 26704 raft_consensus.cc:359] T 00000000000000000000000000000000 P e1282d4177bf4858ba4c63982c6b664d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1282d4177bf4858ba4c63982c6b664d" member_type: VOTER last_known_addr { host: "127.20.49.126" port: 44263 } }
I20251209 08:58:45.506276 26704 raft_consensus.cc:385] T 00000000000000000000000000000000 P e1282d4177bf4858ba4c63982c6b664d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251209 08:58:45.506318 26704 raft_consensus.cc:740] T 00000000000000000000000000000000 P e1282d4177bf4858ba4c63982c6b664d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1282d4177bf4858ba4c63982c6b664d, State: Initialized, Role: FOLLOWER
I20251209 08:58:45.506666 26704 consensus_queue.cc:260] T 00000000000000000000000000000000 P e1282d4177bf4858ba4c63982c6b664d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1282d4177bf4858ba4c63982c6b664d" member_type: VOTER last_known_addr { host: "127.20.49.126" port: 44263 } }
I20251209 08:58:45.506771 26704 raft_consensus.cc:399] T 00000000000000000000000000000000 P e1282d4177bf4858ba4c63982c6b664d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251209 08:58:45.506814 26704 raft_consensus.cc:493] T 00000000000000000000000000000000 P e1282d4177bf4858ba4c63982c6b664d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251209 08:58:45.506880 26704 raft_consensus.cc:3060] T 00000000000000000000000000000000 P e1282d4177bf4858ba4c63982c6b664d [term 0 FOLLOWER]: Advancing to term 1
I20251209 08:58:45.507658 26704 raft_consensus.cc:515] T 00000000000000000000000000000000 P e1282d4177bf4858ba4c63982c6b664d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1282d4177bf4858ba4c63982c6b664d" member_type: VOTER last_known_addr { host: "127.20.49.126" port: 44263 } }
I20251209 08:58:45.507954 26704 leader_election.cc:304] T 00000000000000000000000000000000 P e1282d4177bf4858ba4c63982c6b664d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e1282d4177bf4858ba4c63982c6b664d; no voters:
I20251209 08:58:45.508193 26704 leader_election.cc:290] T 00000000000000000000000000000000 P e1282d4177bf4858ba4c63982c6b664d [CANDIDATE]: Term 1 election: Requested vote from peers
I20251209 08:58:45.508317 26709 raft_consensus.cc:2804] T 00000000000000000000000000000000 P e1282d4177bf4858ba4c63982c6b664d [term 1 FOLLOWER]: Leader election won for term 1
I20251209 08:58:45.508517 26709 raft_consensus.cc:697] T 00000000000000000000000000000000 P e1282d4177bf4858ba4c63982c6b664d [term 1 LEADER]: Becoming Leader. State: Replica: e1282d4177bf4858ba4c63982c6b664d, State: Running, Role: LEADER
I20251209 08:58:45.508880 26709 consensus_queue.cc:237] T 00000000000000000000000000000000 P e1282d4177bf4858ba4c63982c6b664d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1282d4177bf4858ba4c63982c6b664d" member_type: VOTER last_known_addr { host: "127.20.49.126" port: 44263 } }
I20251209 08:58:45.509263 26704 sys_catalog.cc:565] T 00000000000000000000000000000000 P e1282d4177bf4858ba4c63982c6b664d [sys.catalog]: configured and running, proceeding with master startup.
I20251209 08:58:45.510445 26711 sys_catalog.cc:455] T 00000000000000000000000000000000 P e1282d4177bf4858ba4c63982c6b664d [sys.catalog]: SysCatalogTable state changed. Reason: New leader e1282d4177bf4858ba4c63982c6b664d. Latest consensus state: current_term: 1 leader_uuid: "e1282d4177bf4858ba4c63982c6b664d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1282d4177bf4858ba4c63982c6b664d" member_type: VOTER last_known_addr { host: "127.20.49.126" port: 44263 } } }
I20251209 08:58:45.510564 26711 sys_catalog.cc:458] T 00000000000000000000000000000000 P e1282d4177bf4858ba4c63982c6b664d [sys.catalog]: This master's current role is: LEADER
I20251209 08:58:45.510967 26718 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251209 08:58:45.511598 26710 sys_catalog.cc:455] T 00000000000000000000000000000000 P e1282d4177bf4858ba4c63982c6b664d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e1282d4177bf4858ba4c63982c6b664d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1282d4177bf4858ba4c63982c6b664d" member_type: VOTER last_known_addr { host: "127.20.49.126" port: 44263 } } }
I20251209 08:58:45.511688 26710 sys_catalog.cc:458] T 00000000000000000000000000000000 P e1282d4177bf4858ba4c63982c6b664d [sys.catalog]: This master's current role is: LEADER
I20251209 08:58:45.514065 26718 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251209 08:58:45.519953 26718 catalog_manager.cc:1357] Generated new cluster ID: c666e532f145476c8bd5254063b50503
I20251209 08:58:45.520032 26718 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251209 08:58:45.527096 26718 catalog_manager.cc:1380] Generated new certificate authority record
I20251209 08:58:45.528025 26718 catalog_manager.cc:1514] Loading token signing keys...
I20251209 08:58:45.539305 26718 catalog_manager.cc:6027] T 00000000000000000000000000000000 P e1282d4177bf4858ba4c63982c6b664d: Generated new TSK 0
I20251209 08:58:45.540113 26718 catalog_manager.cc:1524] Initializing in-progress tserver states...
E20251209 08:58:45.573797 26734 webserver.cc:578] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20251209 08:58:45.573887 26734 webserver.cc:578] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20251209 08:58:45.575347 26707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:58:45.549967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55402 (local address 127.20.49.126:44263)
1209 08:58:45.550113 (+ 146us) server_negotiation.cc:207] Beginning negotiation
1209 08:58:45.550118 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1209 08:58:45.550275 (+ 157us) server_negotiation.cc:408] Connection header received
1209 08:58:45.550457 (+ 182us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1209 08:58:45.550459 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1209 08:58:45.550523 (+ 64us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1209 08:58:45.550611 (+ 88us) server_negotiation.cc:227] Negotiated authn=JWT
1209 08:58:45.551094 (+ 483us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:58:45.551681 (+ 587us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1209 08:58:45.552630 (+ 949us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:58:45.552787 (+ 157us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:58:45.552880 (+ 93us) server_negotiation.cc:366] Received JWT_EXCHANGE NegotiatePB request
1209 08:58:45.574910 (+ 22030us) server_negotiation.cc:395] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44697/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
1209 08:58:45.575233 (+ 323us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:55402: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44697/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":48}
W20251209 08:58:45.575410 26707 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:55402: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44697/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20251209 08:58:45.575496 26733 negotiation.cc:337] Failed RPC negotiation. Trace:
1209 08:58:45.549873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.49.126:44263 (local address 127.0.0.1:55402)
1209 08:58:45.550186 (+ 313us) negotiation.cc:107] Waiting for socket to connect
1209 08:58:45.550202 (+ 16us) client_negotiation.cc:174] Beginning negotiation
1209 08:58:45.550328 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1209 08:58:45.550632 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1209 08:58:45.550636 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
1209 08:58:45.550645 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
1209 08:58:45.550943 (+ 298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:58:45.550952 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:58:45.551819 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1209 08:58:45.551824 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1209 08:58:45.552498 (+ 674us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:58:45.552508 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:58:45.552625 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:58:45.552650 (+ 25us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
1209 08:58:45.575117 (+ 22467us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44697/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
1209 08:58:45.575306 (+ 189us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.20.49.126:44263: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44697/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":102,"threads_started":1}
I20251209 08:58:45.576195 20677 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task8XuDZV/build/debug/bin/kudu with pid 26637
2025-12-09T08:58:45Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (331 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2025-12-09T08:58:45Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-12-09T08:58:45Z Disabled control of system clock
I20251209 08:58:45.626654 20677 external_mini_cluster.cc:1366] Running /tmp/dist-test-task8XuDZV/build/debug/bin/kudu
/tmp/dist-test-task8XuDZV/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1765270668420570-20677-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1765270668420570-20677-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1765270668420570-20677-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1765270668420570-20677-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.49.126:34537
--webserver_interface=127.20.49.126
--webserver_port=0
--builtin_ntp_servers=127.20.49.84:40121
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.49.126:34537
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1765270668420570-20677-0/testchainca.cert with env {}
W20251209 08:58:45.738822 26738 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251209 08:58:45.739099 26738 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251209 08:58:45.739192 26738 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251209 08:58:45.742517 26738 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251209 08:58:45.742583 26738 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251209 08:58:45.742614 26738 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251209 08:58:45.742645 26738 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251209 08:58:45.742688 26738 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20251209 08:58:45.742724 26738 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20251209 08:58:45.742759 26738 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20251209 08:58:45.747221 26738 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.49.84:40121
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1765270668420570-20677-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1765270668420570-20677-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.49.126:34537
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.49.126:34537
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1765270668420570-20677-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.49.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1765270668420570-20677-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26738
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1765270668420570-20677-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 468897755969d686fb08386f42f7835685e7953a
build type DEBUG
built by None at 09 Dec 2025 08:43:15 UTC on 24a791456cd2
build id 9426
I20251209 08:58:45.748322 26738 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251209 08:58:45.749166 26738 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251209 08:58:45.754863 26744 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251209 08:58:45.754858 26746 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251209 08:58:45.754858 26743 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251209 08:58:45.755116 26738 server_base.cc:1047] running on GCE node
I20251209 08:58:45.755677 26738 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251209 08:58:45.756595 26738 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251209 08:58:45.757745 26738 hybrid_clock.cc:648] HybridClock initialized: now 1765270725757721 us; error 40 us; skew 500 ppm
I20251209 08:58:45.759639 26738 webserver.cc:492] Webserver started at http://127.20.49.126:40231/ using document root <none> and password file <none>
I20251209 08:58:45.760195 26738 fs_manager.cc:362] Metadata directory not provided
I20251209 08:58:45.760249 26738 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251209 08:58:45.760418 26738 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251209 08:58:45.762193 26738 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1765270668420570-20677-0/minicluster-data/master-0/data/instance:
uuid: "1a1d59a66faf4f4db3ca531569cd98ca"
format_stamp: "Formatted at 2025-12-09 08:58:45 on dist-test-slave-8w6c"
server_key: "6b969fe77238ceaf38a41e00338cb875"
server_key_iv: "082d390236fb4e68b68f55b637c7d3ca"
server_key_version: "encryptionkey@0"
I20251209 08:58:45.762662 26738 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1765270668420570-20677-0/minicluster-data/master-0/wal/instance:
uuid: "1a1d59a66faf4f4db3ca531569cd98ca"
format_stamp: "Formatted at 2025-12-09 08:58:45 on dist-test-slave-8w6c"
server_key: "6b969fe77238ceaf38a41e00338cb875"
server_key_iv: "082d390236fb4e68b68f55b637c7d3ca"
server_key_version: "encryptionkey@0"
I20251209 08:58:45.766033 26738 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20251209 08:58:45.768381 26752 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251209 08:58:45.769423 26738 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20251209 08:58:45.769551 26738 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1765270668420570-20677-0/minicluster-data/master-0/data,/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1765270668420570-20677-0/minicluster-data/master-0/wal
uuid: "1a1d59a66faf4f4db3ca531569cd98ca"
format_stamp: "Formatted at 2025-12-09 08:58:45 on dist-test-slave-8w6c"
server_key: "6b969fe77238ceaf38a41e00338cb875"
server_key_iv: "082d390236fb4e68b68f55b637c7d3ca"
server_key_version: "encryptionkey@0"
I20251209 08:58:45.769644 26738 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1765270668420570-20677-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1765270668420570-20677-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1765270668420570-20677-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251209 08:58:45.785156 26738 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251209 08:58:45.785854 26738 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251209 08:58:45.786024 26738 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251209 08:58:45.793350 26738 rpc_server.cc:307] RPC server started. Bound to: 127.20.49.126:34537
I20251209 08:58:45.793416 26804 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.49.126:34537 every 8 connection(s)
I20251209 08:58:45.794449 26738 server_base.cc:1179] Dumped server information to /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1765270668420570-20677-0/minicluster-data/master-0/data/info.pb
I20251209 08:58:45.797271 26805 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251209 08:58:45.802296 20677 external_mini_cluster.cc:1428] Started /tmp/dist-test-task8XuDZV/build/debug/bin/kudu as pid 26738
I20251209 08:58:45.802443 20677 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1765270668420570-20677-0/minicluster-data/master-0/wal/instance
I20251209 08:58:45.802654 20677 external_mini_cluster.cc:1468] Setting key 41bcb5cd5812e485128e342a19a6925f
I20251209 08:58:45.802910 26805 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1a1d59a66faf4f4db3ca531569cd98ca: Bootstrap starting.
I20251209 08:58:45.805419 26805 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1a1d59a66faf4f4db3ca531569cd98ca: Neither blocks nor log segments found. Creating new log.
I20251209 08:58:45.806298 26805 log.cc:826] T 00000000000000000000000000000000 P 1a1d59a66faf4f4db3ca531569cd98ca: Log is configured to *not* fsync() on all Append() calls
I20251209 08:58:45.808192 26805 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1a1d59a66faf4f4db3ca531569cd98ca: No bootstrap required, opened a new log
I20251209 08:58:45.810716 26808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:58:45.803848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59062 (local address 127.20.49.126:34537)
1209 08:58:45.804532 (+ 684us) server_negotiation.cc:207] Beginning negotiation
1209 08:58:45.804544 (+ 12us) server_negotiation.cc:400] Waiting for connection header
1209 08:58:45.804585 (+ 41us) server_negotiation.cc:408] Connection header received
1209 08:58:45.805285 (+ 700us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1209 08:58:45.805313 (+ 28us) server_negotiation.cc:462] Received NEGOTIATE request from client
1209 08:58:45.805797 (+ 484us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1209 08:58:45.806157 (+ 360us) server_negotiation.cc:227] Negotiated authn=SASL
1209 08:58:45.806635 (+ 478us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:58:45.807635 (+ 1000us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1209 08:58:45.808279 (+ 644us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:58:45.808566 (+ 287us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:58:45.808950 (+ 384us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1209 08:58:45.808969 (+ 19us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1209 08:58:45.808980 (+ 11us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1209 08:58:45.809438 (+ 458us) server_negotiation.cc:923] calling sasl_server_start()
1209 08:58:45.809496 (+ 58us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1209 08:58:45.809515 (+ 19us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1209 08:58:45.809626 (+ 111us) server_negotiation.cc:1036] Waiting for connection context
1209 08:58:45.809764 (+ 138us) server_negotiation.cc:300] Negotiation successful
1209 08:58:45.809991 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":287,"thread_start_us":121,"threads_started":1}
I20251209 08:58:45.811324 26805 raft_consensus.cc:359] T 00000000000000000000000000000000 P 1a1d59a66faf4f4db3ca531569cd98ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a1d59a66faf4f4db3ca531569cd98ca" member_type: VOTER last_known_addr { host: "127.20.49.126" port: 34537 } }
I20251209 08:58:45.811499 26805 raft_consensus.cc:385] T 00000000000000000000000000000000 P 1a1d59a66faf4f4db3ca531569cd98ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251209 08:58:45.811565 26805 raft_consensus.cc:740] T 00000000000000000000000000000000 P 1a1d59a66faf4f4db3ca531569cd98ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a1d59a66faf4f4db3ca531569cd98ca, State: Initialized, Role: FOLLOWER
I20251209 08:58:45.811970 26805 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1a1d59a66faf4f4db3ca531569cd98ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a1d59a66faf4f4db3ca531569cd98ca" member_type: VOTER last_known_addr { host: "127.20.49.126" port: 34537 } }
I20251209 08:58:45.812105 26805 raft_consensus.cc:399] T 00000000000000000000000000000000 P 1a1d59a66faf4f4db3ca531569cd98ca [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251209 08:58:45.812167 26805 raft_consensus.cc:493] T 00000000000000000000000000000000 P 1a1d59a66faf4f4db3ca531569cd98ca [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251209 08:58:45.812247 26805 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 1a1d59a66faf4f4db3ca531569cd98ca [term 0 FOLLOWER]: Advancing to term 1
I20251209 08:58:45.813138 26805 raft_consensus.cc:515] T 00000000000000000000000000000000 P 1a1d59a66faf4f4db3ca531569cd98ca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a1d59a66faf4f4db3ca531569cd98ca" member_type: VOTER last_known_addr { host: "127.20.49.126" port: 34537 } }
I20251209 08:58:45.813478 26805 leader_election.cc:304] T 00000000000000000000000000000000 P 1a1d59a66faf4f4db3ca531569cd98ca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1a1d59a66faf4f4db3ca531569cd98ca; no voters:
I20251209 08:58:45.813827 26805 leader_election.cc:290] T 00000000000000000000000000000000 P 1a1d59a66faf4f4db3ca531569cd98ca [CANDIDATE]: Term 1 election: Requested vote from peers
I20251209 08:58:45.813927 26810 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 1a1d59a66faf4f4db3ca531569cd98ca [term 1 FOLLOWER]: Leader election won for term 1
I20251209 08:58:45.814146 26810 raft_consensus.cc:697] T 00000000000000000000000000000000 P 1a1d59a66faf4f4db3ca531569cd98ca [term 1 LEADER]: Becoming Leader. State: Replica: 1a1d59a66faf4f4db3ca531569cd98ca, State: Running, Role: LEADER
I20251209 08:58:45.814162 20677 external_mini_cluster.cc:949] 0 TS(s) registered with all masters
I20251209 08:58:45.814451 26810 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1a1d59a66faf4f4db3ca531569cd98ca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a1d59a66faf4f4db3ca531569cd98ca" member_type: VOTER last_known_addr { host: "127.20.49.126" port: 34537 } }
I20251209 08:58:45.814957 26805 sys_catalog.cc:565] T 00000000000000000000000000000000 P 1a1d59a66faf4f4db3ca531569cd98ca [sys.catalog]: configured and running, proceeding with master startup.
I20251209 08:58:45.816514 26811 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a1d59a66faf4f4db3ca531569cd98ca [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1a1d59a66faf4f4db3ca531569cd98ca. Latest consensus state: current_term: 1 leader_uuid: "1a1d59a66faf4f4db3ca531569cd98ca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a1d59a66faf4f4db3ca531569cd98ca" member_type: VOTER last_known_addr { host: "127.20.49.126" port: 34537 } } }
I20251209 08:58:45.816612 26811 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a1d59a66faf4f4db3ca531569cd98ca [sys.catalog]: This master's current role is: LEADER
I20251209 08:58:45.816933 26812 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a1d59a66faf4f4db3ca531569cd98ca [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1a1d59a66faf4f4db3ca531569cd98ca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a1d59a66faf4f4db3ca531569cd98ca" member_type: VOTER last_known_addr { host: "127.20.49.126" port: 34537 } } }
I20251209 08:58:45.816988 26819 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251209 08:58:45.817047 26812 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a1d59a66faf4f4db3ca531569cd98ca [sys.catalog]: This master's current role is: LEADER
I20251209 08:58:45.820111 26819 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251209 08:58:45.825292 26819 catalog_manager.cc:1357] Generated new cluster ID: 10ce99f8f9a64f5f94751099e8c7ea04
I20251209 08:58:45.825428 26819 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251209 08:58:45.849792 26819 catalog_manager.cc:1380] Generated new certificate authority record
I20251209 08:58:45.850783 26819 catalog_manager.cc:1514] Loading token signing keys...
I20251209 08:58:45.857018 26819 catalog_manager.cc:6027] T 00000000000000000000000000000000 P 1a1d59a66faf4f4db3ca531569cd98ca: Generated new TSK 0
I20251209 08:58:45.857785 26819 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251209 08:58:45.911952 26808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:58:45.873635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59078 (local address 127.20.49.126:34537)
1209 08:58:45.873894 (+ 259us) server_negotiation.cc:207] Beginning negotiation
1209 08:58:45.873901 (+ 7us) server_negotiation.cc:400] Waiting for connection header
1209 08:58:45.873916 (+ 15us) server_negotiation.cc:408] Connection header received
1209 08:58:45.874099 (+ 183us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1209 08:58:45.874102 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1209 08:58:45.874156 (+ 54us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1209 08:58:45.874252 (+ 96us) server_negotiation.cc:227] Negotiated authn=JWT
1209 08:58:45.874613 (+ 361us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:58:45.875196 (+ 583us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1209 08:58:45.875869 (+ 673us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:58:45.876065 (+ 196us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:58:45.876162 (+ 97us) server_negotiation.cc:366] Received JWT_EXCHANGE NegotiatePB request
1209 08:58:45.911486 (+ 35324us) server_negotiation.cc:395] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Could not resolve hostname: Could not resolve host: default.url
1209 08:58:45.911793 (+ 307us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:59078: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Could not resolve hostname: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":155}
W20251209 08:58:45.912045 26808 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:59078: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Could not resolve hostname: Could not resolve host: default.url
W20251209 08:58:45.912156 26834 negotiation.cc:337] Failed RPC negotiation. Trace:
1209 08:58:45.873521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.49.126:34537 (local address 127.0.0.1:59078)
1209 08:58:45.873829 (+ 308us) negotiation.cc:107] Waiting for socket to connect
1209 08:58:45.873845 (+ 16us) client_negotiation.cc:174] Beginning negotiation
1209 08:58:45.873956 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1209 08:58:45.874266 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1209 08:58:45.874269 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1209 08:58:45.874279 (+ 10us) client_negotiation.cc:189] Negotiated authn=JWT
1209 08:58:45.874495 (+ 216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:58:45.874502 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:58:45.875326 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1209 08:58:45.875329 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1209 08:58:45.875754 (+ 425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:58:45.875762 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:58:45.875881 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:58:45.875906 (+ 25us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
1209 08:58:45.911765 (+ 35859us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Could not resolve hostname: Could not resolve host: default.url
1209 08:58:45.911956 (+ 191us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.20.49.126:34537: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Could not resolve hostname: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":106,"threads_started":1}
I20251209 08:58:45.912806 20677 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task8XuDZV/build/debug/bin/kudu with pid 26738
2025-12-09T08:58:45Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (315 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (123 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (123 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765270668420570-20677-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Dec 09 08:58:46 dist-test-slave-8w6c krb5kdc[26846](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Dec 09 08:58:46 dist-test-slave-8w6c krb5kdc[26846](info): set up 2 sockets
Dec 09 08:58:46 dist-test-slave-8w6c krb5kdc[26846](info): commencing operation
krb5kdc: starting...
W20251209 08:58:48.170621 20677 mini_kdc.cc:121] Time spent starting KDC: real 1.990s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765270668420570-20677-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765270668420570-20677-0/krb5kdc/test-user.keytab.
Dec 09 08:58:48 dist-test-slave-8w6c krb5kdc[26846](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765270728, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-12-09T08:58:48Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-12-09T08:58:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.49.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.49.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.49.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765270668420570-20677-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.20.49.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765270668420570-20677-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.20.49.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.20.49.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.20.49.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765270668420570-20677-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.20.49.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765270668420570-20677-0/krb5kdc/kudu.keytab.
I20251209 08:58:48.312817 20677 external_mini_cluster.cc:1366] Running /tmp/dist-test-task8XuDZV/build/debug/bin/kudu
/tmp/dist-test-task8XuDZV/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765270668420570-20677-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765270668420570-20677-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765270668420570-20677-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765270668420570-20677-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.49.126:36295
--webserver_interface=127.20.49.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765270668420570-20677-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.49.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.20.49.84:44977
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.49.126:36295
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765270668420570-20677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765270668420570-20677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765270668420570-20677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251209 08:58:48.427188 26862 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251209 08:58:48.427467 26862 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251209 08:58:48.427558 26862 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251209 08:58:48.430905 26862 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251209 08:58:48.430989 26862 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251209 08:58:48.431028 26862 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251209 08:58:48.431074 26862 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251209 08:58:48.431111 26862 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251209 08:58:48.435544 26862 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.49.84:44977
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765270668420570-20677-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765270668420570-20677-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.49.126:36295
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765270668420570-20677-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.49.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.49.126:36295
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765270668420570-20677-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.49.126
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26862
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765270668420570-20677-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 468897755969d686fb08386f42f7835685e7953a
build type DEBUG
built by None at 09 Dec 2025 08:43:15 UTC on 24a791456cd2
build id 9426
I20251209 08:58:48.436720 26862 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251209 08:58:48.437598 26862 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251209 08:58:48.443122 26867 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251209 08:58:48.443274 26862 server_base.cc:1047] running on GCE node
W20251209 08:58:48.443128 26870 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251209 08:58:48.443122 26868 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251209 08:58:48.443907 26862 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251209 08:58:48.444768 26862 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251209 08:58:48.445923 26862 hybrid_clock.cc:648] HybridClock initialized: now 1765270728445904 us; error 34 us; skew 500 ppm
Dec 09 08:58:48 dist-test-slave-8w6c krb5kdc[26846](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765270728, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.49.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251209 08:58:48.448554 26862 init.cc:377] Logged in from keytab as kudu/127.20.49.126@KRBTEST.COM (short username kudu)
I20251209 08:58:48.449609 26862 webserver.cc:492] Webserver started at http://127.20.49.126:44457/ using document root <none> and password file <none>
I20251209 08:58:48.450198 26862 fs_manager.cc:362] Metadata directory not provided
I20251209 08:58:48.450253 26862 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251209 08:58:48.450459 26862 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251209 08:58:48.452173 26862 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765270668420570-20677-0/minicluster-data/master-0/data/instance:
uuid: "2d71e7d24e54472f944796cdd62edc22"
format_stamp: "Formatted at 2025-12-09 08:58:48 on dist-test-slave-8w6c"
server_key: "865cd6556f1bc85f6e1a9e54e4d3a72b"
server_key_iv: "02231ce4e2bbe76ce55bb2dcd5604b6e"
server_key_version: "encryptionkey@0"
I20251209 08:58:48.452647 26862 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765270668420570-20677-0/minicluster-data/master-0/wal/instance:
uuid: "2d71e7d24e54472f944796cdd62edc22"
format_stamp: "Formatted at 2025-12-09 08:58:48 on dist-test-slave-8w6c"
server_key: "865cd6556f1bc85f6e1a9e54e4d3a72b"
server_key_iv: "02231ce4e2bbe76ce55bb2dcd5604b6e"
server_key_version: "encryptionkey@0"
I20251209 08:58:48.456065 26862 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20251209 08:58:48.458283 26877 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251209 08:58:48.459311 26862 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20251209 08:58:48.459424 26862 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765270668420570-20677-0/minicluster-data/master-0/data,/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765270668420570-20677-0/minicluster-data/master-0/wal
uuid: "2d71e7d24e54472f944796cdd62edc22"
format_stamp: "Formatted at 2025-12-09 08:58:48 on dist-test-slave-8w6c"
server_key: "865cd6556f1bc85f6e1a9e54e4d3a72b"
server_key_iv: "02231ce4e2bbe76ce55bb2dcd5604b6e"
server_key_version: "encryptionkey@0"
I20251209 08:58:48.459527 26862 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765270668420570-20677-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765270668420570-20677-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765270668420570-20677-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251209 08:58:48.473790 26862 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251209 08:58:48.476662 26862 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251209 08:58:48.476851 26862 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251209 08:58:48.484378 26862 rpc_server.cc:307] RPC server started. Bound to: 127.20.49.126:36295
I20251209 08:58:48.484382 26929 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.49.126:36295 every 8 connection(s)
I20251209 08:58:48.485373 26862 server_base.cc:1179] Dumped server information to /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765270668420570-20677-0/minicluster-data/master-0/data/info.pb
I20251209 08:58:48.488138 26930 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251209 08:58:48.489025 20677 external_mini_cluster.cc:1428] Started /tmp/dist-test-task8XuDZV/build/debug/bin/kudu as pid 26862
I20251209 08:58:48.489164 20677 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765270668420570-20677-0/minicluster-data/master-0/wal/instance
I20251209 08:58:48.489449 20677 external_mini_cluster.cc:1468] Setting key ac76fc7f4531e2754430b47ecef98d01
I20251209 08:58:48.494249 26930 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2d71e7d24e54472f944796cdd62edc22: Bootstrap starting.
I20251209 08:58:48.496920 26930 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2d71e7d24e54472f944796cdd62edc22: Neither blocks nor log segments found. Creating new log.
Dec 09 08:58:48 dist-test-slave-8w6c krb5kdc[26846](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765270728, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.49.126@KRBTEST.COM
I20251209 08:58:48.497658 26930 log.cc:826] T 00000000000000000000000000000000 P 2d71e7d24e54472f944796cdd62edc22: Log is configured to *not* fsync() on all Append() calls
I20251209 08:58:48.499462 26930 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2d71e7d24e54472f944796cdd62edc22: No bootstrap required, opened a new log
I20251209 08:58:48.501941 26930 raft_consensus.cc:359] T 00000000000000000000000000000000 P 2d71e7d24e54472f944796cdd62edc22 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d71e7d24e54472f944796cdd62edc22" member_type: VOTER last_known_addr { host: "127.20.49.126" port: 36295 } }
I20251209 08:58:48.502139 26930 raft_consensus.cc:385] T 00000000000000000000000000000000 P 2d71e7d24e54472f944796cdd62edc22 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251209 08:58:48.502207 26930 raft_consensus.cc:740] T 00000000000000000000000000000000 P 2d71e7d24e54472f944796cdd62edc22 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d71e7d24e54472f944796cdd62edc22, State: Initialized, Role: FOLLOWER
I20251209 08:58:48.502283 26933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:58:48.490875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35826 (local address 127.20.49.126:36295)
1209 08:58:48.491330 (+ 455us) server_negotiation.cc:207] Beginning negotiation
1209 08:58:48.491339 (+ 9us) server_negotiation.cc:400] Waiting for connection header
1209 08:58:48.491375 (+ 36us) server_negotiation.cc:408] Connection header received
1209 08:58:48.491910 (+ 535us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1209 08:58:48.491926 (+ 16us) server_negotiation.cc:462] Received NEGOTIATE request from client
1209 08:58:48.492212 (+ 286us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1209 08:58:48.492523 (+ 311us) server_negotiation.cc:227] Negotiated authn=SASL
1209 08:58:48.493445 (+ 922us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:58:48.494239 (+ 794us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1209 08:58:48.495120 (+ 881us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:58:48.495388 (+ 268us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:58:48.497894 (+ 2506us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1209 08:58:48.497910 (+ 16us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1209 08:58:48.497921 (+ 11us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1209 08:58:48.497952 (+ 31us) server_negotiation.cc:923] calling sasl_server_start()
1209 08:58:48.499861 (+ 1909us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1209 08:58:48.500268 (+ 407us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1209 08:58:48.500273 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1209 08:58:48.500277 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
1209 08:58:48.500346 (+ 69us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1209 08:58:48.500570 (+ 224us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1209 08:58:48.500573 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1209 08:58:48.500575 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1209 08:58:48.500887 (+ 312us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1209 08:58:48.501105 (+ 218us) server_negotiation.cc:1036] Waiting for connection context
1209 08:58:48.501375 (+ 270us) server_negotiation.cc:300] Negotiation successful
1209 08:58:48.501625 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":299,"thread_start_us":136,"threads_started":1}
I20251209 08:58:48.502698 26930 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2d71e7d24e54472f944796cdd62edc22 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d71e7d24e54472f944796cdd62edc22" member_type: VOTER last_known_addr { host: "127.20.49.126" port: 36295 } }
I20251209 08:58:48.502856 26930 raft_consensus.cc:399] T 00000000000000000000000000000000 P 2d71e7d24e54472f944796cdd62edc22 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251209 08:58:48.502916 26930 raft_consensus.cc:493] T 00000000000000000000000000000000 P 2d71e7d24e54472f944796cdd62edc22 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251209 08:58:48.503003 26930 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 2d71e7d24e54472f944796cdd62edc22 [term 0 FOLLOWER]: Advancing to term 1
I20251209 08:58:48.504140 26930 raft_consensus.cc:515] T 00000000000000000000000000000000 P 2d71e7d24e54472f944796cdd62edc22 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d71e7d24e54472f944796cdd62edc22" member_type: VOTER last_known_addr { host: "127.20.49.126" port: 36295 } }
I20251209 08:58:48.504516 26930 leader_election.cc:304] T 00000000000000000000000000000000 P 2d71e7d24e54472f944796cdd62edc22 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2d71e7d24e54472f944796cdd62edc22; no voters:
I20251209 08:58:48.504799 26930 leader_election.cc:290] T 00000000000000000000000000000000 P 2d71e7d24e54472f944796cdd62edc22 [CANDIDATE]: Term 1 election: Requested vote from peers
I20251209 08:58:48.504959 26935 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 2d71e7d24e54472f944796cdd62edc22 [term 1 FOLLOWER]: Leader election won for term 1
I20251209 08:58:48.505285 26935 raft_consensus.cc:697] T 00000000000000000000000000000000 P 2d71e7d24e54472f944796cdd62edc22 [term 1 LEADER]: Becoming Leader. State: Replica: 2d71e7d24e54472f944796cdd62edc22, State: Running, Role: LEADER
I20251209 08:58:48.505580 26935 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2d71e7d24e54472f944796cdd62edc22 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d71e7d24e54472f944796cdd62edc22" member_type: VOTER last_known_addr { host: "127.20.49.126" port: 36295 } }
I20251209 08:58:48.505888 26930 sys_catalog.cc:565] T 00000000000000000000000000000000 P 2d71e7d24e54472f944796cdd62edc22 [sys.catalog]: configured and running, proceeding with master startup.
I20251209 08:58:48.506999 26937 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2d71e7d24e54472f944796cdd62edc22 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2d71e7d24e54472f944796cdd62edc22. Latest consensus state: current_term: 1 leader_uuid: "2d71e7d24e54472f944796cdd62edc22" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d71e7d24e54472f944796cdd62edc22" member_type: VOTER last_known_addr { host: "127.20.49.126" port: 36295 } } }
I20251209 08:58:48.507102 26937 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2d71e7d24e54472f944796cdd62edc22 [sys.catalog]: This master's current role is: LEADER
I20251209 08:58:48.507277 26936 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2d71e7d24e54472f944796cdd62edc22 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2d71e7d24e54472f944796cdd62edc22" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d71e7d24e54472f944796cdd62edc22" member_type: VOTER last_known_addr { host: "127.20.49.126" port: 36295 } } }
I20251209 08:58:48.507424 26936 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2d71e7d24e54472f944796cdd62edc22 [sys.catalog]: This master's current role is: LEADER
I20251209 08:58:48.507725 26944 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251209 08:58:48.510620 26944 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251209 08:58:48.515659 26944 catalog_manager.cc:1357] Generated new cluster ID: 077b52ba7e174545b73256f47717f46f
I20251209 08:58:48.515738 26944 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251209 08:58:48.532753 26944 catalog_manager.cc:1380] Generated new certificate authority record
I20251209 08:58:48.533771 26944 catalog_manager.cc:1514] Loading token signing keys...
I20251209 08:58:48.546005 26944 catalog_manager.cc:6027] T 00000000000000000000000000000000 P 2d71e7d24e54472f944796cdd62edc22: Generated new TSK 0
I20251209 08:58:48.546962 26944 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.20.49.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.49.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.49.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765270668420570-20677-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.20.49.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765270668420570-20677-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.20.49.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.20.49.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.20.49.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765270668420570-20677-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.20.49.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765270668420570-20677-0/krb5kdc/kudu.keytab.
I20251209 08:58:48.608889 20677 external_mini_cluster.cc:1366] Running /tmp/dist-test-task8XuDZV/build/debug/bin/kudu
/tmp/dist-test-task8XuDZV/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765270668420570-20677-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765270668420570-20677-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765270668420570-20677-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765270668420570-20677-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.49.65:0
--local_ip_for_outbound_sockets=127.20.49.65
--webserver_interface=127.20.49.65
--webserver_port=0
--tserver_master_addrs=127.20.49.126:36295
--keytab_file=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765270668420570-20677-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.49.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.20.49.84:44977
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765270668420570-20677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765270668420570-20677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765270668420570-20677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251209 08:58:48.722378 26958 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251209 08:58:48.722610 26958 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251209 08:58:48.722681 26958 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251209 08:58:48.726080 26958 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251209 08:58:48.726150 26958 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251209 08:58:48.726249 26958 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.49.65
I20251209 08:58:48.730520 26958 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.49.84:44977
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765270668420570-20677-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765270668420570-20677-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765270668420570-20677-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.49.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.49.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765270668420570-20677-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.49.65
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.20.49.126:36295
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26958
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.49.65
--log_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765270668420570-20677-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 468897755969d686fb08386f42f7835685e7953a
build type DEBUG
built by None at 09 Dec 2025 08:43:15 UTC on 24a791456cd2
build id 9426
I20251209 08:58:48.731564 26958 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251209 08:58:48.732438 26958 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251209 08:58:48.739202 26966 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251209 08:58:48.739197 26963 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251209 08:58:48.739192 26964 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251209 08:58:48.739496 26958 server_base.cc:1047] running on GCE node
I20251209 08:58:48.739914 26958 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251209 08:58:48.740525 26958 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251209 08:58:48.741717 26958 hybrid_clock.cc:648] HybridClock initialized: now 1765270728741697 us; error 50 us; skew 500 ppm
Dec 09 08:58:48 dist-test-slave-8w6c krb5kdc[26846](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765270728, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.49.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251209 08:58:48.744475 26958 init.cc:377] Logged in from keytab as kudu/127.20.49.65@KRBTEST.COM (short username kudu)
I20251209 08:58:48.745579 26958 webserver.cc:492] Webserver started at http://127.20.49.65:42887/ using document root <none> and password file <none>
I20251209 08:58:48.746186 26958 fs_manager.cc:362] Metadata directory not provided
I20251209 08:58:48.746246 26958 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251209 08:58:48.746443 26958 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251209 08:58:48.748211 26958 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765270668420570-20677-0/minicluster-data/ts-0/data/instance:
uuid: "0793ca68653745dbb1312fece128958f"
format_stamp: "Formatted at 2025-12-09 08:58:48 on dist-test-slave-8w6c"
server_key: "c73b9158d7182f817ba1a09be9c6fbaf"
server_key_iv: "ba63a35df1f06c6ebacf6a109f313579"
server_key_version: "encryptionkey@0"
I20251209 08:58:48.748713 26958 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765270668420570-20677-0/minicluster-data/ts-0/wal/instance:
uuid: "0793ca68653745dbb1312fece128958f"
format_stamp: "Formatted at 2025-12-09 08:58:48 on dist-test-slave-8w6c"
server_key: "c73b9158d7182f817ba1a09be9c6fbaf"
server_key_iv: "ba63a35df1f06c6ebacf6a109f313579"
server_key_version: "encryptionkey@0"
I20251209 08:58:48.752208 26958 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20251209 08:58:48.754626 26973 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251209 08:58:48.755755 26958 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20251209 08:58:48.755869 26958 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765270668420570-20677-0/minicluster-data/ts-0/data,/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765270668420570-20677-0/minicluster-data/ts-0/wal
uuid: "0793ca68653745dbb1312fece128958f"
format_stamp: "Formatted at 2025-12-09 08:58:48 on dist-test-slave-8w6c"
server_key: "c73b9158d7182f817ba1a09be9c6fbaf"
server_key_iv: "ba63a35df1f06c6ebacf6a109f313579"
server_key_version: "encryptionkey@0"
I20251209 08:58:48.755963 26958 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765270668420570-20677-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765270668420570-20677-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765270668420570-20677-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251209 08:58:48.772181 26958 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251209 08:58:48.775259 26958 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251209 08:58:48.775462 26958 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251209 08:58:48.776137 26958 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251209 08:58:48.777205 26958 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251209 08:58:48.777256 26958 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251209 08:58:48.777297 26958 ts_tablet_manager.cc:616] Registered 0 tablets
I20251209 08:58:48.777314 26958 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251209 08:58:48.786688 26958 rpc_server.cc:307] RPC server started. Bound to: 127.20.49.65:33061
I20251209 08:58:48.786731 27086 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.49.65:33061 every 8 connection(s)
I20251209 08:58:48.787665 26958 server_base.cc:1179] Dumped server information to /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765270668420570-20677-0/minicluster-data/ts-0/data/info.pb
I20251209 08:58:48.795538 20677 external_mini_cluster.cc:1428] Started /tmp/dist-test-task8XuDZV/build/debug/bin/kudu as pid 26958
I20251209 08:58:48.795673 20677 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765270668420570-20677-0/minicluster-data/ts-0/wal/instance
I20251209 08:58:48.795914 20677 external_mini_cluster.cc:1468] Setting key ed11bb72fd3205ab518b8ab1c3ecd185
Dec 09 08:58:48 dist-test-slave-8w6c krb5kdc[26846](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765270728, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.49.65@KRBTEST.COM for kudu/127.20.49.126@KRBTEST.COM
I20251209 08:58:48.802435 26933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:58:48.789502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.49.65:55765 (local address 127.20.49.126:36295)
1209 08:58:48.789640 (+ 138us) server_negotiation.cc:207] Beginning negotiation
1209 08:58:48.789644 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1209 08:58:48.790340 (+ 696us) server_negotiation.cc:408] Connection header received
1209 08:58:48.791216 (+ 876us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1209 08:58:48.791220 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1209 08:58:48.791282 (+ 62us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1209 08:58:48.791368 (+ 86us) server_negotiation.cc:227] Negotiated authn=SASL
1209 08:58:48.793055 (+ 1687us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:58:48.793557 (+ 502us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1209 08:58:48.794227 (+ 670us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:58:48.794416 (+ 189us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:58:48.796758 (+ 2342us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1209 08:58:48.796777 (+ 19us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1209 08:58:48.796780 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1209 08:58:48.796805 (+ 25us) server_negotiation.cc:923] calling sasl_server_start()
1209 08:58:48.800173 (+ 3368us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1209 08:58:48.800702 (+ 529us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1209 08:58:48.800709 (+ 7us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1209 08:58:48.800714 (+ 5us) server_negotiation.cc:968] calling sasl_server_step()
1209 08:58:48.800776 (+ 62us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1209 08:58:48.801044 (+ 268us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1209 08:58:48.801051 (+ 7us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1209 08:58:48.801055 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
1209 08:58:48.801307 (+ 252us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1209 08:58:48.801424 (+ 117us) server_negotiation.cc:1036] Waiting for connection context
1209 08:58:48.802027 (+ 603us) server_negotiation.cc:300] Negotiation successful
1209 08:58:48.802185 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20251209 08:58:48.802942 27089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:58:48.789775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.49.126:36295 (local address 127.20.49.65:55765)
1209 08:58:48.790193 (+ 418us) negotiation.cc:107] Waiting for socket to connect
1209 08:58:48.790229 (+ 36us) client_negotiation.cc:174] Beginning negotiation
1209 08:58:48.791008 (+ 779us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1209 08:58:48.791551 (+ 543us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1209 08:58:48.791559 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
1209 08:58:48.791953 (+ 394us) client_negotiation.cc:189] Negotiated authn=SASL
1209 08:58:48.792895 (+ 942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:58:48.792912 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:58:48.793702 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1209 08:58:48.793705 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1209 08:58:48.794110 (+ 405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:58:48.794116 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:58:48.794349 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:58:48.795007 (+ 658us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1209 08:58:48.795030 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
1209 08:58:48.796601 (+ 1571us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1209 08:58:48.800301 (+ 3700us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1209 08:58:48.800308 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1209 08:58:48.800319 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
1209 08:58:48.800590 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1209 08:58:48.800866 (+ 276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1209 08:58:48.800868 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1209 08:58:48.800870 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1209 08:58:48.800954 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1209 08:58:48.801439 (+ 485us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1209 08:58:48.801449 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1209 08:58:48.801766 (+ 317us) client_negotiation.cc:769] Sending connection context
1209 08:58:48.801959 (+ 193us) client_negotiation.cc:240] Negotiation successful
1209 08:58:48.802194 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":105,"threads_started":1}
I20251209 08:58:48.804149 27087 heartbeater.cc:344] Connected to a master server at 127.20.49.126:36295
I20251209 08:58:48.804474 27087 heartbeater.cc:461] Registering TS with master...
I20251209 08:58:48.805158 27087 heartbeater.cc:507] Master 127.20.49.126:36295 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.20.49.66@KRBTEST.COM; defaulting to no policy
I20251209 08:58:48.806777 26894 ts_manager.cc:194] Registered new tserver with Master: 0793ca68653745dbb1312fece128958f (127.20.49.65:33061)
I20251209 08:58:48.808135 26894 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.49.65@KRBTEST.COM'} at 127.20.49.65:55765
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.49.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.49.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765270668420570-20677-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.20.49.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765270668420570-20677-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.20.49.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.20.49.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.20.49.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765270668420570-20677-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.20.49.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765270668420570-20677-0/krb5kdc/kudu.keytab.
I20251209 08:58:48.848874 20677 external_mini_cluster.cc:1366] Running /tmp/dist-test-task8XuDZV/build/debug/bin/kudu
/tmp/dist-test-task8XuDZV/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765270668420570-20677-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765270668420570-20677-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765270668420570-20677-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765270668420570-20677-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.49.66:0
--local_ip_for_outbound_sockets=127.20.49.66
--webserver_interface=127.20.49.66
--webserver_port=0
--tserver_master_addrs=127.20.49.126:36295
--keytab_file=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765270668420570-20677-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.49.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.20.49.84:44977
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765270668420570-20677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765270668420570-20677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765270668420570-20677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251209 08:58:48.959612 27094 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251209 08:58:48.959867 27094 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251209 08:58:48.960018 27094 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251209 08:58:48.963397 27094 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251209 08:58:48.963466 27094 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251209 08:58:48.963549 27094 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.49.66
I20251209 08:58:48.967913 27094 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.49.84:44977
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765270668420570-20677-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765270668420570-20677-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765270668420570-20677-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.49.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.49.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765270668420570-20677-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.49.66
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.20.49.126:36295
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27094
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.49.66
--log_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765270668420570-20677-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 468897755969d686fb08386f42f7835685e7953a
build type DEBUG
built by None at 09 Dec 2025 08:43:15 UTC on 24a791456cd2
build id 9426
I20251209 08:58:48.969033 27094 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251209 08:58:48.969947 27094 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251209 08:58:48.976656 27100 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251209 08:58:48.976656 27099 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251209 08:58:48.976657 27102 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251209 08:58:48.976929 27094 server_base.cc:1047] running on GCE node
I20251209 08:58:48.977304 27094 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251209 08:58:48.977936 27094 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251209 08:58:48.979095 27094 hybrid_clock.cc:648] HybridClock initialized: now 1765270728979085 us; error 31 us; skew 500 ppm
Dec 09 08:58:48 dist-test-slave-8w6c krb5kdc[26846](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765270728, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.49.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251209 08:58:48.981864 27094 init.cc:377] Logged in from keytab as kudu/127.20.49.66@KRBTEST.COM (short username kudu)
I20251209 08:58:48.982928 27094 webserver.cc:492] Webserver started at http://127.20.49.66:39347/ using document root <none> and password file <none>
I20251209 08:58:48.983505 27094 fs_manager.cc:362] Metadata directory not provided
I20251209 08:58:48.983561 27094 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251209 08:58:48.983743 27094 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251209 08:58:48.985481 27094 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765270668420570-20677-0/minicluster-data/ts-1/data/instance:
uuid: "5dc358f48890471e92309defebd02308"
format_stamp: "Formatted at 2025-12-09 08:58:48 on dist-test-slave-8w6c"
server_key: "2eec4b0a877a490e9ff387431b343c4b"
server_key_iv: "bbd67ab922076bfc785b09e9bb63379d"
server_key_version: "encryptionkey@0"
I20251209 08:58:48.985981 27094 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765270668420570-20677-0/minicluster-data/ts-1/wal/instance:
uuid: "5dc358f48890471e92309defebd02308"
format_stamp: "Formatted at 2025-12-09 08:58:48 on dist-test-slave-8w6c"
server_key: "2eec4b0a877a490e9ff387431b343c4b"
server_key_iv: "bbd67ab922076bfc785b09e9bb63379d"
server_key_version: "encryptionkey@0"
I20251209 08:58:48.989377 27094 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.000s sys 0.002s
I20251209 08:58:48.991744 27109 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251209 08:58:48.992806 27094 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20251209 08:58:48.992918 27094 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765270668420570-20677-0/minicluster-data/ts-1/data,/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765270668420570-20677-0/minicluster-data/ts-1/wal
uuid: "5dc358f48890471e92309defebd02308"
format_stamp: "Formatted at 2025-12-09 08:58:48 on dist-test-slave-8w6c"
server_key: "2eec4b0a877a490e9ff387431b343c4b"
server_key_iv: "bbd67ab922076bfc785b09e9bb63379d"
server_key_version: "encryptionkey@0"
I20251209 08:58:48.993010 27094 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765270668420570-20677-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765270668420570-20677-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765270668420570-20677-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251209 08:58:49.015782 27094 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251209 08:58:49.019843 27094 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251209 08:58:49.020047 27094 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251209 08:58:49.020642 27094 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251209 08:58:49.021551 27094 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251209 08:58:49.021605 27094 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251209 08:58:49.021665 27094 ts_tablet_manager.cc:616] Registered 0 tablets
I20251209 08:58:49.021731 27094 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251209 08:58:49.031078 27094 rpc_server.cc:307] RPC server started. Bound to: 127.20.49.66:39307
I20251209 08:58:49.031107 27222 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.49.66:39307 every 8 connection(s)
I20251209 08:58:49.032056 27094 server_base.cc:1179] Dumped server information to /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765270668420570-20677-0/minicluster-data/ts-1/data/info.pb
I20251209 08:58:49.035843 20677 external_mini_cluster.cc:1428] Started /tmp/dist-test-task8XuDZV/build/debug/bin/kudu as pid 27094
I20251209 08:58:49.035952 20677 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765270668420570-20677-0/minicluster-data/ts-1/wal/instance
I20251209 08:58:49.036197 20677 external_mini_cluster.cc:1468] Setting key 04c66120ad506324b5d9ad69311e1661
Dec 09 08:58:49 dist-test-slave-8w6c krb5kdc[26846](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765270728, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.49.66@KRBTEST.COM for kudu/127.20.49.126@KRBTEST.COM
I20251209 08:58:49.045663 26933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:58:49.033883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.49.66:58339 (local address 127.20.49.126:36295)
1209 08:58:49.034025 (+ 142us) server_negotiation.cc:207] Beginning negotiation
1209 08:58:49.034029 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1209 08:58:49.034782 (+ 753us) server_negotiation.cc:408] Connection header received
1209 08:58:49.035677 (+ 895us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1209 08:58:49.035681 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1209 08:58:49.035748 (+ 67us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1209 08:58:49.035826 (+ 78us) server_negotiation.cc:227] Negotiated authn=SASL
1209 08:58:49.037581 (+ 1755us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:58:49.038122 (+ 541us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1209 08:58:49.038815 (+ 693us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:58:49.039000 (+ 185us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:58:49.041669 (+ 2669us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1209 08:58:49.041720 (+ 51us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1209 08:58:49.041724 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1209 08:58:49.041755 (+ 31us) server_negotiation.cc:923] calling sasl_server_start()
1209 08:58:49.043425 (+ 1670us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1209 08:58:49.044086 (+ 661us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1209 08:58:49.044089 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1209 08:58:49.044090 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1209 08:58:49.044128 (+ 38us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1209 08:58:49.044452 (+ 324us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1209 08:58:49.044454 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1209 08:58:49.044455 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1209 08:58:49.044625 (+ 170us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1209 08:58:49.044703 (+ 78us) server_negotiation.cc:1036] Waiting for connection context
1209 08:58:49.045394 (+ 691us) server_negotiation.cc:300] Negotiation successful
1209 08:58:49.045518 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
WARNING: no policy specified for kudu/127.20.49.67@KRBTEST.COM; defaulting to no policy
I20251209 08:58:49.046749 27225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:58:49.034175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.49.126:36295 (local address 127.20.49.66:58339)
1209 08:58:49.034628 (+ 453us) negotiation.cc:107] Waiting for socket to connect
1209 08:58:49.034665 (+ 37us) client_negotiation.cc:174] Beginning negotiation
1209 08:58:49.035450 (+ 785us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1209 08:58:49.036013 (+ 563us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1209 08:58:49.036024 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
1209 08:58:49.036479 (+ 455us) client_negotiation.cc:189] Negotiated authn=SASL
1209 08:58:49.037427 (+ 948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:58:49.037441 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:58:49.038266 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1209 08:58:49.038270 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1209 08:58:49.038691 (+ 421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:58:49.038698 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:58:49.038925 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:58:49.039599 (+ 674us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1209 08:58:49.039625 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
1209 08:58:49.041490 (+ 1865us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1209 08:58:49.043562 (+ 2072us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1209 08:58:49.043572 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1209 08:58:49.043589 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
1209 08:58:49.043983 (+ 394us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1209 08:58:49.044204 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1209 08:58:49.044211 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1209 08:58:49.044215 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
1209 08:58:49.044368 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1209 08:58:49.044728 (+ 360us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1209 08:58:49.044737 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1209 08:58:49.045090 (+ 353us) client_negotiation.cc:769] Sending connection context
1209 08:58:49.045368 (+ 278us) client_negotiation.cc:240] Negotiation successful
1209 08:58:49.045720 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":119,"threads_started":1}
I20251209 08:58:49.048075 27223 heartbeater.cc:344] Connected to a master server at 127.20.49.126:36295
I20251209 08:58:49.048341 27223 heartbeater.cc:461] Registering TS with master...
I20251209 08:58:49.048990 27223 heartbeater.cc:507] Master 127.20.49.126:36295 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.49.67@KRBTEST.COM" created.
I20251209 08:58:49.050138 26894 ts_manager.cc:194] Registered new tserver with Master: 5dc358f48890471e92309defebd02308 (127.20.49.66:39307)
I20251209 08:58:49.050798 26894 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.49.66@KRBTEST.COM'} at 127.20.49.66:58339
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.49.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765270668420570-20677-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.20.49.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765270668420570-20677-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.20.49.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.20.49.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.20.49.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765270668420570-20677-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.20.49.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765270668420570-20677-0/krb5kdc/kudu.keytab.
I20251209 08:58:49.088480 20677 external_mini_cluster.cc:1366] Running /tmp/dist-test-task8XuDZV/build/debug/bin/kudu
/tmp/dist-test-task8XuDZV/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765270668420570-20677-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765270668420570-20677-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765270668420570-20677-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765270668420570-20677-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.49.67:0
--local_ip_for_outbound_sockets=127.20.49.67
--webserver_interface=127.20.49.67
--webserver_port=0
--tserver_master_addrs=127.20.49.126:36295
--keytab_file=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765270668420570-20677-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.49.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.20.49.84:44977
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765270668420570-20677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765270668420570-20677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765270668420570-20677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251209 08:58:49.196882 27230 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251209 08:58:49.197120 27230 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251209 08:58:49.197208 27230 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251209 08:58:49.200703 27230 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251209 08:58:49.200784 27230 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251209 08:58:49.200893 27230 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.49.67
I20251209 08:58:49.205097 27230 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.49.84:44977
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765270668420570-20677-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765270668420570-20677-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765270668420570-20677-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.49.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.49.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765270668420570-20677-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.49.67
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.20.49.126:36295
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27230
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.49.67
--log_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765270668420570-20677-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 468897755969d686fb08386f42f7835685e7953a
build type DEBUG
built by None at 09 Dec 2025 08:43:15 UTC on 24a791456cd2
build id 9426
I20251209 08:58:49.206269 27230 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251209 08:58:49.207226 27230 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251209 08:58:49.213801 27235 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251209 08:58:49.213780 27238 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251209 08:58:49.213773 27236 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251209 08:58:49.214214 27230 server_base.cc:1047] running on GCE node
I20251209 08:58:49.214573 27230 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251209 08:58:49.215111 27230 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251209 08:58:49.216266 27230 hybrid_clock.cc:648] HybridClock initialized: now 1765270729216248 us; error 34 us; skew 500 ppm
Dec 09 08:58:49 dist-test-slave-8w6c krb5kdc[26846](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765270729, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.49.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251209 08:58:49.219205 27230 init.cc:377] Logged in from keytab as kudu/127.20.49.67@KRBTEST.COM (short username kudu)
I20251209 08:58:49.220247 27230 webserver.cc:492] Webserver started at http://127.20.49.67:39497/ using document root <none> and password file <none>
I20251209 08:58:49.220772 27230 fs_manager.cc:362] Metadata directory not provided
I20251209 08:58:49.220819 27230 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251209 08:58:49.220971 27230 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251209 08:58:49.222661 27230 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765270668420570-20677-0/minicluster-data/ts-2/data/instance:
uuid: "a2736b7ffaa84cd59556013f2a02a73b"
format_stamp: "Formatted at 2025-12-09 08:58:49 on dist-test-slave-8w6c"
server_key: "79012c6729186600d06bf74668b5c010"
server_key_iv: "9b1efad6efab7e31dfecdbf719c3c26e"
server_key_version: "encryptionkey@0"
I20251209 08:58:49.223088 27230 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765270668420570-20677-0/minicluster-data/ts-2/wal/instance:
uuid: "a2736b7ffaa84cd59556013f2a02a73b"
format_stamp: "Formatted at 2025-12-09 08:58:49 on dist-test-slave-8w6c"
server_key: "79012c6729186600d06bf74668b5c010"
server_key_iv: "9b1efad6efab7e31dfecdbf719c3c26e"
server_key_version: "encryptionkey@0"
I20251209 08:58:49.226445 27230 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20251209 08:58:49.228653 27245 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251209 08:58:49.229787 27230 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20251209 08:58:49.229888 27230 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765270668420570-20677-0/minicluster-data/ts-2/data,/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765270668420570-20677-0/minicluster-data/ts-2/wal
uuid: "a2736b7ffaa84cd59556013f2a02a73b"
format_stamp: "Formatted at 2025-12-09 08:58:49 on dist-test-slave-8w6c"
server_key: "79012c6729186600d06bf74668b5c010"
server_key_iv: "9b1efad6efab7e31dfecdbf719c3c26e"
server_key_version: "encryptionkey@0"
I20251209 08:58:49.229969 27230 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765270668420570-20677-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765270668420570-20677-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765270668420570-20677-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251209 08:58:49.253485 27230 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251209 08:58:49.256897 27230 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251209 08:58:49.257066 27230 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251209 08:58:49.257607 27230 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251209 08:58:49.258626 27230 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251209 08:58:49.258675 27230 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251209 08:58:49.258715 27230 ts_tablet_manager.cc:616] Registered 0 tablets
I20251209 08:58:49.258737 27230 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251209 08:58:49.268095 27230 rpc_server.cc:307] RPC server started. Bound to: 127.20.49.67:39365
I20251209 08:58:49.268114 27358 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.49.67:39365 every 8 connection(s)
I20251209 08:58:49.269063 27230 server_base.cc:1179] Dumped server information to /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765270668420570-20677-0/minicluster-data/ts-2/data/info.pb
I20251209 08:58:49.274993 20677 external_mini_cluster.cc:1428] Started /tmp/dist-test-task8XuDZV/build/debug/bin/kudu as pid 27230
I20251209 08:58:49.275138 20677 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765270668420570-20677-0/minicluster-data/ts-2/wal/instance
I20251209 08:58:49.275399 20677 external_mini_cluster.cc:1468] Setting key 532b064d03324c2afa41dd6c429fea3a
Dec 09 08:58:49 dist-test-slave-8w6c krb5kdc[26846](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765270729, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.49.67@KRBTEST.COM for kudu/127.20.49.126@KRBTEST.COM
I20251209 08:58:49.281617 26933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:58:49.270832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.49.67:49869 (local address 127.20.49.126:36295)
1209 08:58:49.270961 (+ 129us) server_negotiation.cc:207] Beginning negotiation
1209 08:58:49.270965 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1209 08:58:49.271626 (+ 661us) server_negotiation.cc:408] Connection header received
1209 08:58:49.272488 (+ 862us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1209 08:58:49.272493 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1209 08:58:49.272557 (+ 64us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1209 08:58:49.272644 (+ 87us) server_negotiation.cc:227] Negotiated authn=SASL
1209 08:58:49.274369 (+ 1725us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:58:49.274967 (+ 598us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1209 08:58:49.275797 (+ 830us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:58:49.275992 (+ 195us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:58:49.278551 (+ 2559us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1209 08:58:49.278567 (+ 16us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1209 08:58:49.278569 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1209 08:58:49.278593 (+ 24us) server_negotiation.cc:923] calling sasl_server_start()
1209 08:58:49.279862 (+ 1269us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1209 08:58:49.280323 (+ 461us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1209 08:58:49.280326 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1209 08:58:49.280327 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1209 08:58:49.280370 (+ 43us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1209 08:58:49.280609 (+ 239us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1209 08:58:49.280611 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1209 08:58:49.280612 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1209 08:58:49.280747 (+ 135us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1209 08:58:49.280820 (+ 73us) server_negotiation.cc:1036] Waiting for connection context
1209 08:58:49.281381 (+ 561us) server_negotiation.cc:300] Negotiation successful
1209 08:58:49.281493 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20251209 08:58:49.282298 27361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:58:49.271051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.49.126:36295 (local address 127.20.49.67:49869)
1209 08:58:49.271475 (+ 424us) negotiation.cc:107] Waiting for socket to connect
1209 08:58:49.271517 (+ 42us) client_negotiation.cc:174] Beginning negotiation
1209 08:58:49.272278 (+ 761us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1209 08:58:49.272815 (+ 537us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1209 08:58:49.272823 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
1209 08:58:49.273213 (+ 390us) client_negotiation.cc:189] Negotiated authn=SASL
1209 08:58:49.274214 (+ 1001us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:58:49.274229 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:58:49.275149 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1209 08:58:49.275153 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1209 08:58:49.275648 (+ 495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:58:49.275658 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:58:49.275908 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:58:49.276610 (+ 702us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1209 08:58:49.276645 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
1209 08:58:49.278412 (+ 1767us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1209 08:58:49.279967 (+ 1555us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1209 08:58:49.279971 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1209 08:58:49.279980 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
1209 08:58:49.280203 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1209 08:58:49.280445 (+ 242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1209 08:58:49.280447 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1209 08:58:49.280448 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1209 08:58:49.280541 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1209 08:58:49.280828 (+ 287us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1209 08:58:49.280833 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1209 08:58:49.281096 (+ 263us) client_negotiation.cc:769] Sending connection context
1209 08:58:49.281320 (+ 224us) client_negotiation.cc:240] Negotiation successful
1209 08:58:49.281562 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":99,"threads_started":1}
I20251209 08:58:49.283365 27359 heartbeater.cc:344] Connected to a master server at 127.20.49.126:36295
I20251209 08:58:49.283603 27359 heartbeater.cc:461] Registering TS with master...
I20251209 08:58:49.284106 27359 heartbeater.cc:507] Master 127.20.49.126:36295 requested a full tablet report, sending...
I20251209 08:58:49.285046 26894 ts_manager.cc:194] Registered new tserver with Master: a2736b7ffaa84cd59556013f2a02a73b (127.20.49.67:39365)
I20251209 08:58:49.285585 26894 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.49.67@KRBTEST.COM'} at 127.20.49.67:49869
I20251209 08:58:49.289408 20677 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765270668420570-20677-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765270668420570-20677-0/krb5kdc/test-admin.keytab.
Dec 09 08:58:49 dist-test-slave-8w6c krb5kdc[26846](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765270729, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251209 08:58:49.306432 20677 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20251209 08:58:49.309082 27366 negotiation.cc:337] Failed RPC negotiation. Trace:
1209 08:58:49.307842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56472 (local address 127.20.49.65:33061)
1209 08:58:49.308108 (+ 266us) server_negotiation.cc:207] Beginning negotiation
1209 08:58:49.308114 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1209 08:58:49.308139 (+ 25us) server_negotiation.cc:408] Connection header received
1209 08:58:49.308196 (+ 57us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1209 08:58:49.308201 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1209 08:58:49.308394 (+ 193us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1209 08:58:49.308517 (+ 123us) server_negotiation.cc:227] Negotiated authn=SASL
1209 08:58:49.308917 (+ 400us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56472: BlockingRecv error: recv got EOF from 127.0.0.1:56472 (error 108)
Metrics: {"server-negotiator.queue_time_us":169,"thread_start_us":73,"threads_started":1}
W20251209 08:58:49.309213 20677 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765270668420570-20677-0/krb5kdc/krb5cc' of type FILE
Dec 09 08:58:49 dist-test-slave-8w6c krb5kdc[26846](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765270729, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251209 08:58:49.310334 20677 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Dec 09 08:58:49 dist-test-slave-8w6c krb5kdc[26846](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765270729, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251209 08:58:49.311244 20677 init.cc:295] Successfully reacquired a new kerberos TGT
Dec 09 08:58:49 dist-test-slave-8w6c krb5kdc[26846](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765270729, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.49.65@KRBTEST.COM
I20251209 08:58:49.320307 27366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:58:49.312118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56478 (local address 127.20.49.65:33061)
1209 08:58:49.312271 (+ 153us) server_negotiation.cc:207] Beginning negotiation
1209 08:58:49.312276 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1209 08:58:49.312290 (+ 14us) server_negotiation.cc:408] Connection header received
1209 08:58:49.312422 (+ 132us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1209 08:58:49.312425 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1209 08:58:49.312470 (+ 45us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1209 08:58:49.312540 (+ 70us) server_negotiation.cc:227] Negotiated authn=SASL
1209 08:58:49.313213 (+ 673us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:58:49.313969 (+ 756us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1209 08:58:49.314640 (+ 671us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:58:49.314899 (+ 259us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:58:49.316859 (+ 1960us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1209 08:58:49.316897 (+ 38us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1209 08:58:49.316909 (+ 12us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1209 08:58:49.316935 (+ 26us) server_negotiation.cc:923] calling sasl_server_start()
1209 08:58:49.318726 (+ 1791us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1209 08:58:49.319137 (+ 411us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1209 08:58:49.319143 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1209 08:58:49.319144 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1209 08:58:49.319201 (+ 57us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1209 08:58:49.319424 (+ 223us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1209 08:58:49.319428 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1209 08:58:49.319430 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1209 08:58:49.319629 (+ 199us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1209 08:58:49.319833 (+ 204us) server_negotiation.cc:1036] Waiting for connection context
1209 08:58:49.320020 (+ 187us) server_negotiation.cc:300] Negotiation successful
1209 08:58:49.320172 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
W20251209 08:58:49.323800 27366 negotiation.cc:337] Failed RPC negotiation. Trace:
1209 08:58:49.322964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56482 (local address 127.20.49.65:33061)
1209 08:58:49.323102 (+ 138us) server_negotiation.cc:207] Beginning negotiation
1209 08:58:49.323106 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1209 08:58:49.323151 (+ 45us) server_negotiation.cc:408] Connection header received
1209 08:58:49.323240 (+ 89us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1209 08:58:49.323243 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1209 08:58:49.323282 (+ 39us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1209 08:58:49.323353 (+ 71us) server_negotiation.cc:227] Negotiated authn=SASL
1209 08:58:49.323701 (+ 348us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56482: BlockingRecv error: recv got EOF from 127.0.0.1:56482 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
W20251209 08:58:49.323942 20677 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765270668420570-20677-0/krb5kdc/krb5cc' of type FILE
Dec 09 08:58:49 dist-test-slave-8w6c krb5kdc[26846](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765270729, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251209 08:58:49.324975 20677 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Dec 09 08:58:49 dist-test-slave-8w6c krb5kdc[26846](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765270729, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251209 08:58:49.325898 20677 init.cc:295] Successfully reacquired a new kerberos TGT
Dec 09 08:58:49 dist-test-slave-8w6c krb5kdc[26846](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765270729, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.49.65@KRBTEST.COM
I20251209 08:58:49.333942 27366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:58:49.326711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56488 (local address 127.20.49.65:33061)
1209 08:58:49.326831 (+ 120us) server_negotiation.cc:207] Beginning negotiation
1209 08:58:49.326834 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1209 08:58:49.326848 (+ 14us) server_negotiation.cc:408] Connection header received
1209 08:58:49.326943 (+ 95us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1209 08:58:49.326945 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1209 08:58:49.326983 (+ 38us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1209 08:58:49.327049 (+ 66us) server_negotiation.cc:227] Negotiated authn=SASL
1209 08:58:49.327597 (+ 548us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:58:49.328071 (+ 474us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1209 08:58:49.328625 (+ 554us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:58:49.328769 (+ 144us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:58:49.330858 (+ 2089us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1209 08:58:49.330876 (+ 18us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1209 08:58:49.330878 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1209 08:58:49.330903 (+ 25us) server_negotiation.cc:923] calling sasl_server_start()
1209 08:58:49.332293 (+ 1390us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1209 08:58:49.332746 (+ 453us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1209 08:58:49.332750 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1209 08:58:49.332751 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1209 08:58:49.332801 (+ 50us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1209 08:58:49.333102 (+ 301us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1209 08:58:49.333106 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1209 08:58:49.333107 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1209 08:58:49.333261 (+ 154us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1209 08:58:49.333379 (+ 118us) server_negotiation.cc:1036] Waiting for connection context
1209 08:58:49.333649 (+ 270us) server_negotiation.cc:300] Negotiation successful
1209 08:58:49.333814 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
W20251209 08:58:49.336585 27366 negotiation.cc:337] Failed RPC negotiation. Trace:
1209 08:58:49.335788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56500 (local address 127.20.49.65:33061)
1209 08:58:49.335926 (+ 138us) server_negotiation.cc:207] Beginning negotiation
1209 08:58:49.335930 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1209 08:58:49.335958 (+ 28us) server_negotiation.cc:408] Connection header received
1209 08:58:49.336045 (+ 87us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1209 08:58:49.336047 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1209 08:58:49.336090 (+ 43us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1209 08:58:49.336150 (+ 60us) server_negotiation.cc:227] Negotiated authn=SASL
1209 08:58:49.336492 (+ 342us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56500: BlockingRecv error: recv got EOF from 127.0.0.1:56500 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
W20251209 08:58:49.336782 20677 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1765270668420570-20677-0/krb5kdc/krb5cc' of type FILE
Dec 09 08:58:49 dist-test-slave-8w6c krb5kdc[26846](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765270729, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251209 08:58:49.337826 20677 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Dec 09 08:58:49 dist-test-slave-8w6c krb5kdc[26846](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765270729, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251209 08:58:49.338775 20677 init.cc:295] Successfully reacquired a new kerberos TGT
Dec 09 08:58:49 dist-test-slave-8w6c krb5kdc[26846](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765270729, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.49.65@KRBTEST.COM
I20251209 08:58:49.347054 27366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:58:49.339588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56514 (local address 127.20.49.65:33061)
1209 08:58:49.339702 (+ 114us) server_negotiation.cc:207] Beginning negotiation
1209 08:58:49.339708 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1209 08:58:49.339774 (+ 66us) server_negotiation.cc:408] Connection header received
1209 08:58:49.339896 (+ 122us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1209 08:58:49.339901 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1209 08:58:49.339937 (+ 36us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1209 08:58:49.340005 (+ 68us) server_negotiation.cc:227] Negotiated authn=SASL
1209 08:58:49.340618 (+ 613us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:58:49.341066 (+ 448us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1209 08:58:49.341667 (+ 601us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:58:49.341909 (+ 242us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:58:49.344048 (+ 2139us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1209 08:58:49.344065 (+ 17us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1209 08:58:49.344067 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1209 08:58:49.344093 (+ 26us) server_negotiation.cc:923] calling sasl_server_start()
1209 08:58:49.345550 (+ 1457us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1209 08:58:49.346017 (+ 467us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1209 08:58:49.346020 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1209 08:58:49.346021 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1209 08:58:49.346065 (+ 44us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1209 08:58:49.346277 (+ 212us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1209 08:58:49.346281 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1209 08:58:49.346283 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1209 08:58:49.346433 (+ 150us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1209 08:58:49.346592 (+ 159us) server_negotiation.cc:1036] Waiting for connection context
1209 08:58:49.346783 (+ 191us) server_negotiation.cc:300] Negotiation successful
1209 08:58:49.346896 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20251209 08:58:49.348145 20677 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task8XuDZV/build/debug/bin/kudu with pid 26958
I20251209 08:58:49.355146 20677 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task8XuDZV/build/debug/bin/kudu with pid 27094
I20251209 08:58:49.361310 20677 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task8XuDZV/build/debug/bin/kudu with pid 27230
I20251209 08:58:49.367062 20677 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task8XuDZV/build/debug/bin/kudu with pid 26862
2025-12-09T08:58:49Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (3209 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765270668420570-20677-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Dec 09 08:58:49 dist-test-slave-8w6c krb5kdc[27379](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Dec 09 08:58:49 dist-test-slave-8w6c krb5kdc[27379](info): set up 2 sockets
Dec 09 08:58:49 dist-test-slave-8w6c krb5kdc[27379](info): commencing operation
krb5kdc: starting...
W20251209 08:58:51.378561 20677 mini_kdc.cc:121] Time spent starting KDC: real 1.988s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765270668420570-20677-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765270668420570-20677-0/krb5kdc/test-user.keytab.
Dec 09 08:58:51 dist-test-slave-8w6c krb5kdc[27379](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765270731, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-12-09T08:58:51Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-12-09T08:58:51Z Disabled control of system clock
WARNING: no policy specified for oryx/127.20.49.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.20.49.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.20.49.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765270668420570-20677-0/krb5kdc/oryx.keytab.
Entry for principal oryx/127.20.49.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765270668420570-20677-0/krb5kdc/oryx.keytab.
WARNING: no policy specified for HTTP/127.20.49.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.20.49.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.20.49.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765270668420570-20677-0/krb5kdc/oryx.keytab.
Entry for principal HTTP/127.20.49.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765270668420570-20677-0/krb5kdc/oryx.keytab.
I20251209 08:58:51.527268 20677 external_mini_cluster.cc:1366] Running /tmp/dist-test-task8XuDZV/build/debug/bin/kudu
/tmp/dist-test-task8XuDZV/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765270668420570-20677-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765270668420570-20677-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765270668420570-20677-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765270668420570-20677-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.49.126:42449
--webserver_interface=127.20.49.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765270668420570-20677-0/krb5kdc/oryx.keytab
--principal=oryx/127.20.49.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.20.49.84:40253
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.49.126:42449
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765270668420570-20677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765270668420570-20677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765270668420570-20677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251209 08:58:51.641032 27395 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251209 08:58:51.641336 27395 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251209 08:58:51.641422 27395 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251209 08:58:51.644930 27395 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251209 08:58:51.645021 27395 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251209 08:58:51.645061 27395 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251209 08:58:51.645115 27395 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251209 08:58:51.645159 27395 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251209 08:58:51.649761 27395 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.49.84:40253
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765270668420570-20677-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765270668420570-20677-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.49.126:42449
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765270668420570-20677-0/krb5kdc/oryx.keytab
--principal=oryx/127.20.49.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.49.126:42449
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765270668420570-20677-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.49.126
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27395
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765270668420570-20677-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 468897755969d686fb08386f42f7835685e7953a
build type DEBUG
built by None at 09 Dec 2025 08:43:15 UTC on 24a791456cd2
build id 9426
I20251209 08:58:51.650924 27395 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251209 08:58:51.651837 27395 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251209 08:58:51.657655 27403 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251209 08:58:51.657647 27401 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251209 08:58:51.657733 27400 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251209 08:58:51.657902 27395 server_base.cc:1047] running on GCE node
I20251209 08:58:51.658531 27395 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251209 08:58:51.659457 27395 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251209 08:58:51.660640 27395 hybrid_clock.cc:648] HybridClock initialized: now 1765270731660622 us; error 47 us; skew 500 ppm
Dec 09 08:58:51 dist-test-slave-8w6c krb5kdc[27379](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765270731, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.49.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251209 08:58:51.663398 27395 init.cc:377] Logged in from keytab as oryx/127.20.49.126@KRBTEST.COM (short username oryx)
I20251209 08:58:51.664507 27395 webserver.cc:492] Webserver started at http://127.20.49.126:40689/ using document root <none> and password file <none>
I20251209 08:58:51.665038 27395 fs_manager.cc:362] Metadata directory not provided
I20251209 08:58:51.665093 27395 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251209 08:58:51.665258 27395 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251209 08:58:51.667040 27395 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765270668420570-20677-0/minicluster-data/master-0/data/instance:
uuid: "c0a0d121c6bb474dac7c98dc7ec52d14"
format_stamp: "Formatted at 2025-12-09 08:58:51 on dist-test-slave-8w6c"
server_key: "52628977308dd3a1ffabdeae11c1e899"
server_key_iv: "b7e664a00ba56087de8892698740b1ef"
server_key_version: "encryptionkey@0"
I20251209 08:58:51.667521 27395 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765270668420570-20677-0/minicluster-data/master-0/wal/instance:
uuid: "c0a0d121c6bb474dac7c98dc7ec52d14"
format_stamp: "Formatted at 2025-12-09 08:58:51 on dist-test-slave-8w6c"
server_key: "52628977308dd3a1ffabdeae11c1e899"
server_key_iv: "b7e664a00ba56087de8892698740b1ef"
server_key_version: "encryptionkey@0"
I20251209 08:58:51.670992 27395 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20251209 08:58:51.673715 27410 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251209 08:58:51.674830 27395 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20251209 08:58:51.674942 27395 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765270668420570-20677-0/minicluster-data/master-0/data,/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765270668420570-20677-0/minicluster-data/master-0/wal
uuid: "c0a0d121c6bb474dac7c98dc7ec52d14"
format_stamp: "Formatted at 2025-12-09 08:58:51 on dist-test-slave-8w6c"
server_key: "52628977308dd3a1ffabdeae11c1e899"
server_key_iv: "b7e664a00ba56087de8892698740b1ef"
server_key_version: "encryptionkey@0"
I20251209 08:58:51.675026 27395 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765270668420570-20677-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765270668420570-20677-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765270668420570-20677-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251209 08:58:51.687441 27395 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251209 08:58:51.695149 27395 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251209 08:58:51.695343 27395 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251209 08:58:51.702942 27462 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.49.126:42449 every 8 connection(s)
I20251209 08:58:51.702955 27395 rpc_server.cc:307] RPC server started. Bound to: 127.20.49.126:42449
I20251209 08:58:51.704095 27395 server_base.cc:1179] Dumped server information to /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765270668420570-20677-0/minicluster-data/master-0/data/info.pb
I20251209 08:58:51.707027 27463 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251209 08:58:51.712798 27463 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c0a0d121c6bb474dac7c98dc7ec52d14: Bootstrap starting.
I20251209 08:58:51.714260 20677 external_mini_cluster.cc:1428] Started /tmp/dist-test-task8XuDZV/build/debug/bin/kudu as pid 27395
I20251209 08:58:51.714419 20677 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765270668420570-20677-0/minicluster-data/master-0/wal/instance
I20251209 08:58:51.714668 20677 external_mini_cluster.cc:1468] Setting key 7848a35d1aa7f98bd581f4843bebc2b3
I20251209 08:58:51.715574 27463 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c0a0d121c6bb474dac7c98dc7ec52d14: Neither blocks nor log segments found. Creating new log.
I20251209 08:58:51.716413 27463 log.cc:826] T 00000000000000000000000000000000 P c0a0d121c6bb474dac7c98dc7ec52d14: Log is configured to *not* fsync() on all Append() calls
I20251209 08:58:51.718767 27463 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c0a0d121c6bb474dac7c98dc7ec52d14: No bootstrap required, opened a new log
I20251209 08:58:51.722018 27463 raft_consensus.cc:359] T 00000000000000000000000000000000 P c0a0d121c6bb474dac7c98dc7ec52d14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0a0d121c6bb474dac7c98dc7ec52d14" member_type: VOTER last_known_addr { host: "127.20.49.126" port: 42449 } }
I20251209 08:58:51.722277 27463 raft_consensus.cc:385] T 00000000000000000000000000000000 P c0a0d121c6bb474dac7c98dc7ec52d14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251209 08:58:51.722337 27463 raft_consensus.cc:740] T 00000000000000000000000000000000 P c0a0d121c6bb474dac7c98dc7ec52d14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0a0d121c6bb474dac7c98dc7ec52d14, State: Initialized, Role: FOLLOWER
Dec 09 08:58:51 dist-test-slave-8w6c krb5kdc[27379](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765270731, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.20.49.126@KRBTEST.COM
I20251209 08:58:51.722757 27463 consensus_queue.cc:260] T 00000000000000000000000000000000 P c0a0d121c6bb474dac7c98dc7ec52d14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0a0d121c6bb474dac7c98dc7ec52d14" member_type: VOTER last_known_addr { host: "127.20.49.126" port: 42449 } }
I20251209 08:58:51.722891 27463 raft_consensus.cc:399] T 00000000000000000000000000000000 P c0a0d121c6bb474dac7c98dc7ec52d14 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251209 08:58:51.722958 27463 raft_consensus.cc:493] T 00000000000000000000000000000000 P c0a0d121c6bb474dac7c98dc7ec52d14 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251209 08:58:51.723054 27463 raft_consensus.cc:3060] T 00000000000000000000000000000000 P c0a0d121c6bb474dac7c98dc7ec52d14 [term 0 FOLLOWER]: Advancing to term 1
I20251209 08:58:51.724159 27463 raft_consensus.cc:515] T 00000000000000000000000000000000 P c0a0d121c6bb474dac7c98dc7ec52d14 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0a0d121c6bb474dac7c98dc7ec52d14" member_type: VOTER last_known_addr { host: "127.20.49.126" port: 42449 } }
I20251209 08:58:51.724570 27463 leader_election.cc:304] T 00000000000000000000000000000000 P c0a0d121c6bb474dac7c98dc7ec52d14 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c0a0d121c6bb474dac7c98dc7ec52d14; no voters:
I20251209 08:58:51.724924 27463 leader_election.cc:290] T 00000000000000000000000000000000 P c0a0d121c6bb474dac7c98dc7ec52d14 [CANDIDATE]: Term 1 election: Requested vote from peers
I20251209 08:58:51.725312 27468 raft_consensus.cc:2804] T 00000000000000000000000000000000 P c0a0d121c6bb474dac7c98dc7ec52d14 [term 1 FOLLOWER]: Leader election won for term 1
I20251209 08:58:51.725561 27468 raft_consensus.cc:697] T 00000000000000000000000000000000 P c0a0d121c6bb474dac7c98dc7ec52d14 [term 1 LEADER]: Becoming Leader. State: Replica: c0a0d121c6bb474dac7c98dc7ec52d14, State: Running, Role: LEADER
I20251209 08:58:51.725929 27468 consensus_queue.cc:237] T 00000000000000000000000000000000 P c0a0d121c6bb474dac7c98dc7ec52d14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0a0d121c6bb474dac7c98dc7ec52d14" member_type: VOTER last_known_addr { host: "127.20.49.126" port: 42449 } }
I20251209 08:58:51.726290 27463 sys_catalog.cc:565] T 00000000000000000000000000000000 P c0a0d121c6bb474dac7c98dc7ec52d14 [sys.catalog]: configured and running, proceeding with master startup.
I20251209 08:58:51.727510 27470 sys_catalog.cc:455] T 00000000000000000000000000000000 P c0a0d121c6bb474dac7c98dc7ec52d14 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c0a0d121c6bb474dac7c98dc7ec52d14. Latest consensus state: current_term: 1 leader_uuid: "c0a0d121c6bb474dac7c98dc7ec52d14" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0a0d121c6bb474dac7c98dc7ec52d14" member_type: VOTER last_known_addr { host: "127.20.49.126" port: 42449 } } }
I20251209 08:58:51.727625 27470 sys_catalog.cc:458] T 00000000000000000000000000000000 P c0a0d121c6bb474dac7c98dc7ec52d14 [sys.catalog]: This master's current role is: LEADER
I20251209 08:58:51.727936 27469 sys_catalog.cc:455] T 00000000000000000000000000000000 P c0a0d121c6bb474dac7c98dc7ec52d14 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c0a0d121c6bb474dac7c98dc7ec52d14" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0a0d121c6bb474dac7c98dc7ec52d14" member_type: VOTER last_known_addr { host: "127.20.49.126" port: 42449 } } }
I20251209 08:58:51.727993 27477 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251209 08:58:51.728055 27469 sys_catalog.cc:458] T 00000000000000000000000000000000 P c0a0d121c6bb474dac7c98dc7ec52d14 [sys.catalog]: This master's current role is: LEADER
I20251209 08:58:51.729439 27466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:58:51.715787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58456 (local address 127.20.49.126:42449)
1209 08:58:51.716204 (+ 417us) server_negotiation.cc:207] Beginning negotiation
1209 08:58:51.716215 (+ 11us) server_negotiation.cc:400] Waiting for connection header
1209 08:58:51.716251 (+ 36us) server_negotiation.cc:408] Connection header received
1209 08:58:51.716917 (+ 666us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1209 08:58:51.716946 (+ 29us) server_negotiation.cc:462] Received NEGOTIATE request from client
1209 08:58:51.717335 (+ 389us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1209 08:58:51.717639 (+ 304us) server_negotiation.cc:227] Negotiated authn=SASL
1209 08:58:51.718777 (+ 1138us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:58:51.719565 (+ 788us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1209 08:58:51.720462 (+ 897us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:58:51.720720 (+ 258us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:58:51.723450 (+ 2730us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1209 08:58:51.723472 (+ 22us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1209 08:58:51.723486 (+ 14us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1209 08:58:51.723530 (+ 44us) server_negotiation.cc:923] calling sasl_server_start()
1209 08:58:51.725953 (+ 2423us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1209 08:58:51.726502 (+ 549us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1209 08:58:51.726509 (+ 7us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1209 08:58:51.726515 (+ 6us) server_negotiation.cc:968] calling sasl_server_step()
1209 08:58:51.726615 (+ 100us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1209 08:58:51.726902 (+ 287us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1209 08:58:51.726905 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1209 08:58:51.726907 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1209 08:58:51.727401 (+ 494us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1209 08:58:51.727674 (+ 273us) server_negotiation.cc:1036] Waiting for connection context
1209 08:58:51.728317 (+ 643us) server_negotiation.cc:300] Negotiation successful
1209 08:58:51.728607 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":284,"thread_start_us":134,"threads_started":1}
I20251209 08:58:51.731673 27477 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251209 08:58:51.737520 27477 catalog_manager.cc:1357] Generated new cluster ID: 51c444202dc5408cb61915bc1a2c504c
I20251209 08:58:51.737643 27477 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251209 08:58:51.776800 27477 catalog_manager.cc:1380] Generated new certificate authority record
I20251209 08:58:51.778188 27477 catalog_manager.cc:1514] Loading token signing keys...
I20251209 08:58:51.785418 27477 catalog_manager.cc:6027] T 00000000000000000000000000000000 P c0a0d121c6bb474dac7c98dc7ec52d14: Generated new TSK 0
I20251209 08:58:51.786399 27477 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.20.49.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.20.49.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.20.49.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765270668420570-20677-0/krb5kdc/oryx.keytab.
Entry for principal oryx/127.20.49.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765270668420570-20677-0/krb5kdc/oryx.keytab.
WARNING: no policy specified for HTTP/127.20.49.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.20.49.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.20.49.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765270668420570-20677-0/krb5kdc/oryx.keytab.
Entry for principal HTTP/127.20.49.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765270668420570-20677-0/krb5kdc/oryx.keytab.
I20251209 08:58:51.840797 20677 external_mini_cluster.cc:1366] Running /tmp/dist-test-task8XuDZV/build/debug/bin/kudu
/tmp/dist-test-task8XuDZV/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765270668420570-20677-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765270668420570-20677-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765270668420570-20677-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765270668420570-20677-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.49.65:0
--local_ip_for_outbound_sockets=127.20.49.65
--webserver_interface=127.20.49.65
--webserver_port=0
--tserver_master_addrs=127.20.49.126:42449
--keytab_file=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765270668420570-20677-0/krb5kdc/oryx.keytab
--principal=oryx/127.20.49.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.20.49.84:40253
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765270668420570-20677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765270668420570-20677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765270668420570-20677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251209 08:58:51.953078 27491 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251209 08:58:51.953334 27491 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251209 08:58:51.953397 27491 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251209 08:58:51.956795 27491 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251209 08:58:51.956861 27491 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251209 08:58:51.956941 27491 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.49.65
I20251209 08:58:51.961237 27491 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.49.84:40253
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765270668420570-20677-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765270668420570-20677-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765270668420570-20677-0/krb5kdc/oryx.keytab
--principal=oryx/127.20.49.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.49.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765270668420570-20677-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.49.65
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.20.49.126:42449
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27491
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.49.65
--log_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765270668420570-20677-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 468897755969d686fb08386f42f7835685e7953a
build type DEBUG
built by None at 09 Dec 2025 08:43:15 UTC on 24a791456cd2
build id 9426
I20251209 08:58:51.962297 27491 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251209 08:58:51.963124 27491 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251209 08:58:51.969916 27496 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251209 08:58:51.969925 27499 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251209 08:58:51.969936 27497 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251209 08:58:51.970120 27491 server_base.cc:1047] running on GCE node
I20251209 08:58:51.970568 27491 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251209 08:58:51.971119 27491 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251209 08:58:51.972337 27491 hybrid_clock.cc:648] HybridClock initialized: now 1765270731972310 us; error 42 us; skew 500 ppm
Dec 09 08:58:51 dist-test-slave-8w6c krb5kdc[27379](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765270731, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.49.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251209 08:58:51.975112 27491 init.cc:377] Logged in from keytab as oryx/127.20.49.65@KRBTEST.COM (short username oryx)
I20251209 08:58:51.976182 27491 webserver.cc:492] Webserver started at http://127.20.49.65:37597/ using document root <none> and password file <none>
I20251209 08:58:51.976751 27491 fs_manager.cc:362] Metadata directory not provided
I20251209 08:58:51.976807 27491 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251209 08:58:51.976995 27491 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251209 08:58:51.978793 27491 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765270668420570-20677-0/minicluster-data/ts-0/data/instance:
uuid: "c7b2b642c9144d568877db72b8553ac5"
format_stamp: "Formatted at 2025-12-09 08:58:51 on dist-test-slave-8w6c"
server_key: "204ae2da64bef4c03034d89940ac1943"
server_key_iv: "d69e132ff5074ea344155fba327b27e7"
server_key_version: "encryptionkey@0"
I20251209 08:58:51.979297 27491 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765270668420570-20677-0/minicluster-data/ts-0/wal/instance:
uuid: "c7b2b642c9144d568877db72b8553ac5"
format_stamp: "Formatted at 2025-12-09 08:58:51 on dist-test-slave-8w6c"
server_key: "204ae2da64bef4c03034d89940ac1943"
server_key_iv: "d69e132ff5074ea344155fba327b27e7"
server_key_version: "encryptionkey@0"
I20251209 08:58:51.982738 27491 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.000s sys 0.004s
I20251209 08:58:51.984998 27506 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251209 08:58:51.986069 27491 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20251209 08:58:51.986176 27491 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765270668420570-20677-0/minicluster-data/ts-0/data,/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765270668420570-20677-0/minicluster-data/ts-0/wal
uuid: "c7b2b642c9144d568877db72b8553ac5"
format_stamp: "Formatted at 2025-12-09 08:58:51 on dist-test-slave-8w6c"
server_key: "204ae2da64bef4c03034d89940ac1943"
server_key_iv: "d69e132ff5074ea344155fba327b27e7"
server_key_version: "encryptionkey@0"
I20251209 08:58:51.986264 27491 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765270668420570-20677-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765270668420570-20677-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765270668420570-20677-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251209 08:58:52.035531 27491 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251209 08:58:52.038825 27491 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251209 08:58:52.039024 27491 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251209 08:58:52.039623 27491 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251209 08:58:52.040546 27491 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251209 08:58:52.040598 27491 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251209 08:58:52.040653 27491 ts_tablet_manager.cc:616] Registered 0 tablets
I20251209 08:58:52.040689 27491 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251209 08:58:52.050050 27491 rpc_server.cc:307] RPC server started. Bound to: 127.20.49.65:38709
I20251209 08:58:52.050108 27619 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.49.65:38709 every 8 connection(s)
I20251209 08:58:52.051076 27491 server_base.cc:1179] Dumped server information to /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765270668420570-20677-0/minicluster-data/ts-0/data/info.pb
I20251209 08:58:52.058323 20677 external_mini_cluster.cc:1428] Started /tmp/dist-test-task8XuDZV/build/debug/bin/kudu as pid 27491
I20251209 08:58:52.058444 20677 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765270668420570-20677-0/minicluster-data/ts-0/wal/instance
I20251209 08:58:52.058694 20677 external_mini_cluster.cc:1468] Setting key 0a60c8f04e94deea1a1ef2b36a863369
Dec 09 08:58:52 dist-test-slave-8w6c krb5kdc[27379](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765270731, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.49.65@KRBTEST.COM for oryx/127.20.49.126@KRBTEST.COM
I20251209 08:58:52.065652 27466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:58:52.053211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.49.65:49341 (local address 127.20.49.126:42449)
1209 08:58:52.053397 (+ 186us) server_negotiation.cc:207] Beginning negotiation
1209 08:58:52.053401 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1209 08:58:52.053957 (+ 556us) server_negotiation.cc:408] Connection header received
1209 08:58:52.054907 (+ 950us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1209 08:58:52.054911 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1209 08:58:52.054964 (+ 53us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1209 08:58:52.055053 (+ 89us) server_negotiation.cc:227] Negotiated authn=SASL
1209 08:58:52.056919 (+ 1866us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:58:52.057522 (+ 603us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1209 08:58:52.058312 (+ 790us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:58:52.058502 (+ 190us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:58:52.061160 (+ 2658us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1209 08:58:52.061185 (+ 25us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1209 08:58:52.061192 (+ 7us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1209 08:58:52.061229 (+ 37us) server_negotiation.cc:923] calling sasl_server_start()
1209 08:58:52.063179 (+ 1950us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1209 08:58:52.063868 (+ 689us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1209 08:58:52.063871 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1209 08:58:52.063872 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1209 08:58:52.063922 (+ 50us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1209 08:58:52.064352 (+ 430us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1209 08:58:52.064354 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1209 08:58:52.064356 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1209 08:58:52.064558 (+ 202us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1209 08:58:52.064648 (+ 90us) server_negotiation.cc:1036] Waiting for connection context
1209 08:58:52.065382 (+ 734us) server_negotiation.cc:300] Negotiation successful
1209 08:58:52.065513 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20251209 08:58:52.066728 27622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:58:52.053302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.49.126:42449 (local address 127.20.49.65:49341)
1209 08:58:52.053801 (+ 499us) negotiation.cc:107] Waiting for socket to connect
1209 08:58:52.053841 (+ 40us) client_negotiation.cc:174] Beginning negotiation
1209 08:58:52.054679 (+ 838us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1209 08:58:52.055258 (+ 579us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1209 08:58:52.055268 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
1209 08:58:52.055739 (+ 471us) client_negotiation.cc:189] Negotiated authn=SASL
1209 08:58:52.056728 (+ 989us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:58:52.056744 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:58:52.057668 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1209 08:58:52.057671 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1209 08:58:52.058169 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:58:52.058178 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:58:52.058444 (+ 266us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:58:52.059159 (+ 715us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1209 08:58:52.059187 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
1209 08:58:52.060999 (+ 1812us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1209 08:58:52.063316 (+ 2317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1209 08:58:52.063322 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1209 08:58:52.063336 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
1209 08:58:52.063735 (+ 399us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1209 08:58:52.064022 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1209 08:58:52.064029 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1209 08:58:52.064034 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
1209 08:58:52.064207 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1209 08:58:52.064676 (+ 469us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1209 08:58:52.064687 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1209 08:58:52.065037 (+ 350us) client_negotiation.cc:769] Sending connection context
1209 08:58:52.065315 (+ 278us) client_negotiation.cc:240] Negotiation successful
1209 08:58:52.065707 (+ 392us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":291,"thread_start_us":99,"threads_started":1}
I20251209 08:58:52.068058 27620 heartbeater.cc:344] Connected to a master server at 127.20.49.126:42449
I20251209 08:58:52.068359 27620 heartbeater.cc:461] Registering TS with master...
WARNING: no policy specified for oryx/127.20.49.66@KRBTEST.COM; defaulting to no policy
I20251209 08:58:52.068996 27620 heartbeater.cc:507] Master 127.20.49.126:42449 requested a full tablet report, sending...
I20251209 08:58:52.070607 27427 ts_manager.cc:194] Registered new tserver with Master: c7b2b642c9144d568877db72b8553ac5 (127.20.49.65:38709)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.20.49.66@KRBTEST.COM" created.
I20251209 08:58:52.072000 27427 master_service.cc:502] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.20.49.65@KRBTEST.COM'} at 127.20.49.65:49341
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.20.49.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765270668420570-20677-0/krb5kdc/oryx.keytab.
Entry for principal oryx/127.20.49.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765270668420570-20677-0/krb5kdc/oryx.keytab.
WARNING: no policy specified for HTTP/127.20.49.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.20.49.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.20.49.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765270668420570-20677-0/krb5kdc/oryx.keytab.
Entry for principal HTTP/127.20.49.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765270668420570-20677-0/krb5kdc/oryx.keytab.
I20251209 08:58:52.121028 20677 external_mini_cluster.cc:1366] Running /tmp/dist-test-task8XuDZV/build/debug/bin/kudu
/tmp/dist-test-task8XuDZV/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765270668420570-20677-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765270668420570-20677-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765270668420570-20677-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765270668420570-20677-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.49.66:0
--local_ip_for_outbound_sockets=127.20.49.66
--webserver_interface=127.20.49.66
--webserver_port=0
--tserver_master_addrs=127.20.49.126:42449
--keytab_file=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765270668420570-20677-0/krb5kdc/oryx.keytab
--principal=oryx/127.20.49.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.20.49.84:40253
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765270668420570-20677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765270668420570-20677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765270668420570-20677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251209 08:58:52.237344 27627 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251209 08:58:52.237587 27627 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251209 08:58:52.237648 27627 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251209 08:58:52.241367 27627 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251209 08:58:52.241431 27627 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251209 08:58:52.241514 27627 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.49.66
I20251209 08:58:52.245821 27627 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.49.84:40253
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765270668420570-20677-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765270668420570-20677-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765270668420570-20677-0/krb5kdc/oryx.keytab
--principal=oryx/127.20.49.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.49.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765270668420570-20677-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.49.66
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.20.49.126:42449
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27627
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.49.66
--log_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765270668420570-20677-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 468897755969d686fb08386f42f7835685e7953a
build type DEBUG
built by None at 09 Dec 2025 08:43:15 UTC on 24a791456cd2
build id 9426
I20251209 08:58:52.246922 27627 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251209 08:58:52.247731 27627 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251209 08:58:52.254534 27632 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251209 08:58:52.254539 27635 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251209 08:58:52.254702 27627 server_base.cc:1047] running on GCE node
W20251209 08:58:52.254562 27633 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251209 08:58:52.255214 27627 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251209 08:58:52.255800 27627 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251209 08:58:52.256961 27627 hybrid_clock.cc:648] HybridClock initialized: now 1765270732256949 us; error 26 us; skew 500 ppm
Dec 09 08:58:52 dist-test-slave-8w6c krb5kdc[27379](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765270732, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.49.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251209 08:58:52.259878 27627 init.cc:377] Logged in from keytab as oryx/127.20.49.66@KRBTEST.COM (short username oryx)
I20251209 08:58:52.260916 27627 webserver.cc:492] Webserver started at http://127.20.49.66:33885/ using document root <none> and password file <none>
I20251209 08:58:52.261453 27627 fs_manager.cc:362] Metadata directory not provided
I20251209 08:58:52.261507 27627 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251209 08:58:52.261713 27627 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251209 08:58:52.263401 27627 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765270668420570-20677-0/minicluster-data/ts-1/data/instance:
uuid: "7182d3e0faba4ceaa1acc9b8b3674cbb"
format_stamp: "Formatted at 2025-12-09 08:58:52 on dist-test-slave-8w6c"
server_key: "271312f522119c7b1a805989f64aed85"
server_key_iv: "397121a43a64c9c935d465b9a1923d49"
server_key_version: "encryptionkey@0"
I20251209 08:58:52.263864 27627 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765270668420570-20677-0/minicluster-data/ts-1/wal/instance:
uuid: "7182d3e0faba4ceaa1acc9b8b3674cbb"
format_stamp: "Formatted at 2025-12-09 08:58:52 on dist-test-slave-8w6c"
server_key: "271312f522119c7b1a805989f64aed85"
server_key_iv: "397121a43a64c9c935d465b9a1923d49"
server_key_version: "encryptionkey@0"
I20251209 08:58:52.267309 27627 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.001s sys 0.003s
I20251209 08:58:52.269486 27642 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251209 08:58:52.270529 27627 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20251209 08:58:52.270659 27627 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765270668420570-20677-0/minicluster-data/ts-1/data,/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765270668420570-20677-0/minicluster-data/ts-1/wal
uuid: "7182d3e0faba4ceaa1acc9b8b3674cbb"
format_stamp: "Formatted at 2025-12-09 08:58:52 on dist-test-slave-8w6c"
server_key: "271312f522119c7b1a805989f64aed85"
server_key_iv: "397121a43a64c9c935d465b9a1923d49"
server_key_version: "encryptionkey@0"
I20251209 08:58:52.270751 27627 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765270668420570-20677-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765270668420570-20677-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765270668420570-20677-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251209 08:58:52.293412 27627 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251209 08:58:52.296326 27627 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251209 08:58:52.296523 27627 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251209 08:58:52.297075 27627 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251209 08:58:52.297986 27627 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251209 08:58:52.298036 27627 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251209 08:58:52.298098 27627 ts_tablet_manager.cc:616] Registered 0 tablets
I20251209 08:58:52.298130 27627 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251209 08:58:52.307410 27627 rpc_server.cc:307] RPC server started. Bound to: 127.20.49.66:35715
I20251209 08:58:52.307447 27755 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.49.66:35715 every 8 connection(s)
I20251209 08:58:52.308483 27627 server_base.cc:1179] Dumped server information to /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765270668420570-20677-0/minicluster-data/ts-1/data/info.pb
Dec 09 08:58:52 dist-test-slave-8w6c krb5kdc[27379](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765270732, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.49.66@KRBTEST.COM for oryx/127.20.49.126@KRBTEST.COM
I20251209 08:58:52.317917 20677 external_mini_cluster.cc:1428] Started /tmp/dist-test-task8XuDZV/build/debug/bin/kudu as pid 27627
I20251209 08:58:52.318051 20677 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765270668420570-20677-0/minicluster-data/ts-1/wal/instance
I20251209 08:58:52.318311 20677 external_mini_cluster.cc:1468] Setting key 0d3938df083bb65130aa73a3dc60c7af
I20251209 08:58:52.322238 27466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:58:52.310474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.49.66:46725 (local address 127.20.49.126:42449)
1209 08:58:52.310627 (+ 153us) server_negotiation.cc:207] Beginning negotiation
1209 08:58:52.310631 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1209 08:58:52.311324 (+ 693us) server_negotiation.cc:408] Connection header received
1209 08:58:52.312221 (+ 897us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1209 08:58:52.312225 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1209 08:58:52.312292 (+ 67us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1209 08:58:52.312381 (+ 89us) server_negotiation.cc:227] Negotiated authn=SASL
1209 08:58:52.314202 (+ 1821us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:58:52.314729 (+ 527us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1209 08:58:52.315370 (+ 641us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:58:52.315555 (+ 185us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:58:52.318041 (+ 2486us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1209 08:58:52.318071 (+ 30us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1209 08:58:52.318073 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1209 08:58:52.318109 (+ 36us) server_negotiation.cc:923] calling sasl_server_start()
1209 08:58:52.319719 (+ 1610us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1209 08:58:52.320368 (+ 649us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1209 08:58:52.320371 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1209 08:58:52.320373 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1209 08:58:52.320421 (+ 48us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1209 08:58:52.320810 (+ 389us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1209 08:58:52.320813 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1209 08:58:52.320815 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1209 08:58:52.320980 (+ 165us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1209 08:58:52.321077 (+ 97us) server_negotiation.cc:1036] Waiting for connection context
1209 08:58:52.321833 (+ 756us) server_negotiation.cc:300] Negotiation successful
1209 08:58:52.322010 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20251209 08:58:52.322832 27758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:58:52.310749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.49.126:42449 (local address 127.20.49.66:46725)
1209 08:58:52.311172 (+ 423us) negotiation.cc:107] Waiting for socket to connect
1209 08:58:52.311210 (+ 38us) client_negotiation.cc:174] Beginning negotiation
1209 08:58:52.312010 (+ 800us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1209 08:58:52.312545 (+ 535us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1209 08:58:52.312554 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
1209 08:58:52.312966 (+ 412us) client_negotiation.cc:189] Negotiated authn=SASL
1209 08:58:52.314036 (+ 1070us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:58:52.314052 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:58:52.314857 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1209 08:58:52.314860 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1209 08:58:52.315247 (+ 387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:58:52.315254 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:58:52.315487 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:58:52.316158 (+ 671us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1209 08:58:52.316183 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
1209 08:58:52.317858 (+ 1675us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1209 08:58:52.319835 (+ 1977us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1209 08:58:52.319841 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1209 08:58:52.319856 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
1209 08:58:52.320231 (+ 375us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1209 08:58:52.320514 (+ 283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1209 08:58:52.320520 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1209 08:58:52.320522 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1209 08:58:52.320681 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1209 08:58:52.321096 (+ 415us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1209 08:58:52.321106 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1209 08:58:52.321449 (+ 343us) client_negotiation.cc:769] Sending connection context
1209 08:58:52.321774 (+ 325us) client_negotiation.cc:240] Negotiation successful
1209 08:58:52.322046 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":100,"threads_started":1}
I20251209 08:58:52.324122 27756 heartbeater.cc:344] Connected to a master server at 127.20.49.126:42449
I20251209 08:58:52.324445 27756 heartbeater.cc:461] Registering TS with master...
I20251209 08:58:52.325193 27756 heartbeater.cc:507] Master 127.20.49.126:42449 requested a full tablet report, sending...
I20251209 08:58:52.326534 27427 ts_manager.cc:194] Registered new tserver with Master: 7182d3e0faba4ceaa1acc9b8b3674cbb (127.20.49.66:35715)
I20251209 08:58:52.327168 27427 master_service.cc:502] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.20.49.66@KRBTEST.COM'} at 127.20.49.66:46725
WARNING: no policy specified for oryx/127.20.49.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.20.49.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.20.49.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765270668420570-20677-0/krb5kdc/oryx.keytab.
Entry for principal oryx/127.20.49.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765270668420570-20677-0/krb5kdc/oryx.keytab.
WARNING: no policy specified for HTTP/127.20.49.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.20.49.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.20.49.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765270668420570-20677-0/krb5kdc/oryx.keytab.
Entry for principal HTTP/127.20.49.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765270668420570-20677-0/krb5kdc/oryx.keytab.
I20251209 08:58:52.370882 20677 external_mini_cluster.cc:1366] Running /tmp/dist-test-task8XuDZV/build/debug/bin/kudu
/tmp/dist-test-task8XuDZV/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765270668420570-20677-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765270668420570-20677-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765270668420570-20677-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765270668420570-20677-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.49.67:0
--local_ip_for_outbound_sockets=127.20.49.67
--webserver_interface=127.20.49.67
--webserver_port=0
--tserver_master_addrs=127.20.49.126:42449
--keytab_file=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765270668420570-20677-0/krb5kdc/oryx.keytab
--principal=oryx/127.20.49.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.20.49.84:40253
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765270668420570-20677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765270668420570-20677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765270668420570-20677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251209 08:58:52.480793 27763 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251209 08:58:52.481040 27763 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251209 08:58:52.481112 27763 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251209 08:58:52.484447 27763 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251209 08:58:52.484531 27763 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251209 08:58:52.484663 27763 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.49.67
I20251209 08:58:52.488898 27763 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.49.84:40253
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765270668420570-20677-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765270668420570-20677-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765270668420570-20677-0/krb5kdc/oryx.keytab
--principal=oryx/127.20.49.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.49.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765270668420570-20677-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.49.67
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.20.49.126:42449
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27763
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.49.67
--log_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765270668420570-20677-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 468897755969d686fb08386f42f7835685e7953a
build type DEBUG
built by None at 09 Dec 2025 08:43:15 UTC on 24a791456cd2
build id 9426
I20251209 08:58:52.489975 27763 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251209 08:58:52.490845 27763 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251209 08:58:52.497332 27771 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251209 08:58:52.497347 27769 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251209 08:58:52.497568 27768 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251209 08:58:52.497860 27763 server_base.cc:1047] running on GCE node
I20251209 08:58:52.498216 27763 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251209 08:58:52.498792 27763 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251209 08:58:52.499940 27763 hybrid_clock.cc:648] HybridClock initialized: now 1765270732499931 us; error 28 us; skew 500 ppm
Dec 09 08:58:52 dist-test-slave-8w6c krb5kdc[27379](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765270732, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.49.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251209 08:58:52.502667 27763 init.cc:377] Logged in from keytab as oryx/127.20.49.67@KRBTEST.COM (short username oryx)
I20251209 08:58:52.503695 27763 webserver.cc:492] Webserver started at http://127.20.49.67:44899/ using document root <none> and password file <none>
I20251209 08:58:52.504220 27763 fs_manager.cc:362] Metadata directory not provided
I20251209 08:58:52.504266 27763 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251209 08:58:52.504429 27763 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251209 08:58:52.506151 27763 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765270668420570-20677-0/minicluster-data/ts-2/data/instance:
uuid: "9787869327f04a04a23ea007848ae36b"
format_stamp: "Formatted at 2025-12-09 08:58:52 on dist-test-slave-8w6c"
server_key: "041248036dac2a232e81804fbb047a0c"
server_key_iv: "760473bb4c6402d90e6e2f5ce047ae1b"
server_key_version: "encryptionkey@0"
I20251209 08:58:52.506654 27763 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765270668420570-20677-0/minicluster-data/ts-2/wal/instance:
uuid: "9787869327f04a04a23ea007848ae36b"
format_stamp: "Formatted at 2025-12-09 08:58:52 on dist-test-slave-8w6c"
server_key: "041248036dac2a232e81804fbb047a0c"
server_key_iv: "760473bb4c6402d90e6e2f5ce047ae1b"
server_key_version: "encryptionkey@0"
I20251209 08:58:52.510174 27763 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20251209 08:58:52.512375 27778 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251209 08:58:52.513378 27763 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20251209 08:58:52.513477 27763 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765270668420570-20677-0/minicluster-data/ts-2/data,/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765270668420570-20677-0/minicluster-data/ts-2/wal
uuid: "9787869327f04a04a23ea007848ae36b"
format_stamp: "Formatted at 2025-12-09 08:58:52 on dist-test-slave-8w6c"
server_key: "041248036dac2a232e81804fbb047a0c"
server_key_iv: "760473bb4c6402d90e6e2f5ce047ae1b"
server_key_version: "encryptionkey@0"
I20251209 08:58:52.513573 27763 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765270668420570-20677-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765270668420570-20677-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765270668420570-20677-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251209 08:58:52.531354 27763 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251209 08:58:52.534479 27763 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251209 08:58:52.534670 27763 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251209 08:58:52.535254 27763 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251209 08:58:52.536164 27763 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251209 08:58:52.536216 27763 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251209 08:58:52.536275 27763 ts_tablet_manager.cc:616] Registered 0 tablets
I20251209 08:58:52.536319 27763 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251209 08:58:52.545787 27763 rpc_server.cc:307] RPC server started. Bound to: 127.20.49.67:33177
I20251209 08:58:52.545840 27891 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.49.67:33177 every 8 connection(s)
I20251209 08:58:52.546758 27763 server_base.cc:1179] Dumped server information to /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765270668420570-20677-0/minicluster-data/ts-2/data/info.pb
I20251209 08:58:52.547262 20677 external_mini_cluster.cc:1428] Started /tmp/dist-test-task8XuDZV/build/debug/bin/kudu as pid 27763
I20251209 08:58:52.547384 20677 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765270668420570-20677-0/minicluster-data/ts-2/wal/instance
I20251209 08:58:52.547645 20677 external_mini_cluster.cc:1468] Setting key 2e3862294786000904abaa65912e5026
Dec 09 08:58:52 dist-test-slave-8w6c krb5kdc[27379](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765270732, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.49.67@KRBTEST.COM for oryx/127.20.49.126@KRBTEST.COM
I20251209 08:58:52.559582 27466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:58:52.548521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.49.67:40085 (local address 127.20.49.126:42449)
1209 08:58:52.548689 (+ 168us) server_negotiation.cc:207] Beginning negotiation
1209 08:58:52.548694 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1209 08:58:52.549404 (+ 710us) server_negotiation.cc:408] Connection header received
1209 08:58:52.550291 (+ 887us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1209 08:58:52.550294 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1209 08:58:52.550349 (+ 55us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1209 08:58:52.550422 (+ 73us) server_negotiation.cc:227] Negotiated authn=SASL
1209 08:58:52.552258 (+ 1836us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:58:52.552800 (+ 542us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1209 08:58:52.553473 (+ 673us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:58:52.553632 (+ 159us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:58:52.555977 (+ 2345us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1209 08:58:52.555996 (+ 19us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1209 08:58:52.555998 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1209 08:58:52.556025 (+ 27us) server_negotiation.cc:923] calling sasl_server_start()
1209 08:58:52.557615 (+ 1590us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1209 08:58:52.558125 (+ 510us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1209 08:58:52.558127 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1209 08:58:52.558129 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1209 08:58:52.558170 (+ 41us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1209 08:58:52.558510 (+ 340us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1209 08:58:52.558512 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1209 08:58:52.558513 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1209 08:58:52.558670 (+ 157us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1209 08:58:52.558745 (+ 75us) server_negotiation.cc:1036] Waiting for connection context
1209 08:58:52.559324 (+ 579us) server_negotiation.cc:300] Negotiation successful
1209 08:58:52.559440 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20251209 08:58:52.560322 27894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:58:52.548841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.49.126:42449 (local address 127.20.49.67:40085)
1209 08:58:52.549268 (+ 427us) negotiation.cc:107] Waiting for socket to connect
1209 08:58:52.549312 (+ 44us) client_negotiation.cc:174] Beginning negotiation
1209 08:58:52.550105 (+ 793us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1209 08:58:52.550625 (+ 520us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1209 08:58:52.550635 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
1209 08:58:52.551055 (+ 420us) client_negotiation.cc:189] Negotiated authn=SASL
1209 08:58:52.552080 (+ 1025us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:58:52.552100 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:58:52.552926 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1209 08:58:52.552930 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1209 08:58:52.553341 (+ 411us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:58:52.553349 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:58:52.553555 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:58:52.554227 (+ 672us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1209 08:58:52.554254 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
1209 08:58:52.555816 (+ 1562us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1209 08:58:52.557779 (+ 1963us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1209 08:58:52.557784 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1209 08:58:52.557793 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
1209 08:58:52.558037 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1209 08:58:52.558291 (+ 254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1209 08:58:52.558295 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1209 08:58:52.558297 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1209 08:58:52.558422 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1209 08:58:52.558786 (+ 364us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1209 08:58:52.558792 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1209 08:58:52.559063 (+ 271us) client_negotiation.cc:769] Sending connection context
1209 08:58:52.559257 (+ 194us) client_negotiation.cc:240] Negotiation successful
1209 08:58:52.559603 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":97,"threads_started":1}
I20251209 08:58:52.561414 27892 heartbeater.cc:344] Connected to a master server at 127.20.49.126:42449
I20251209 08:58:52.561712 27892 heartbeater.cc:461] Registering TS with master...
I20251209 08:58:52.562244 27892 heartbeater.cc:507] Master 127.20.49.126:42449 requested a full tablet report, sending...
I20251209 08:58:52.563220 27427 ts_manager.cc:194] Registered new tserver with Master: 9787869327f04a04a23ea007848ae36b (127.20.49.67:33177)
I20251209 08:58:52.563795 27427 master_service.cc:502] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.20.49.67@KRBTEST.COM'} at 127.20.49.67:40085
I20251209 08:58:52.573326 20677 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.20.49.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.20.49.125@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.20.49.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765270668420570-20677-0/krb5kdc/oryx.keytab.
Entry for principal oryx/127.20.49.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765270668420570-20677-0/krb5kdc/oryx.keytab.
WARNING: no policy specified for HTTP/127.20.49.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.20.49.125@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.20.49.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765270668420570-20677-0/krb5kdc/oryx.keytab.
Entry for principal HTTP/127.20.49.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765270668420570-20677-0/krb5kdc/oryx.keytab.
I20251209 08:58:52.622951 20677 external_mini_cluster.cc:1366] Running /tmp/dist-test-task8XuDZV/build/debug/bin/kudu
/tmp/dist-test-task8XuDZV/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765270668420570-20677-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765270668420570-20677-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765270668420570-20677-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765270668420570-20677-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.49.125:44905
--webserver_interface=127.20.49.125
--webserver_port=0
--keytab_file=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765270668420570-20677-0/krb5kdc/oryx.keytab
--principal=oryx/127.20.49.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.20.49.84:40253
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.49.126:42449,127.20.49.125:44905
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765270668420570-20677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765270668420570-20677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765270668420570-20677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251209 08:58:52.736567 27899 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251209 08:58:52.736814 27899 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251209 08:58:52.736869 27899 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251209 08:58:52.740418 27899 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251209 08:58:52.740499 27899 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251209 08:58:52.740540 27899 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251209 08:58:52.740574 27899 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251209 08:58:52.740612 27899 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251209 08:58:52.745105 27899 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.49.84:40253
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765270668420570-20677-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765270668420570-20677-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.20.49.126:42449,127.20.49.125:44905
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765270668420570-20677-0/krb5kdc/oryx.keytab
--principal=oryx/127.20.49.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.49.125:44905
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765270668420570-20677-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.20.49.125
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27899
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765270668420570-20677-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 468897755969d686fb08386f42f7835685e7953a
build type DEBUG
built by None at 09 Dec 2025 08:43:15 UTC on 24a791456cd2
build id 9426
W20251209 08:58:52.745376 27899 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.20.49.126:42449,127.20.49.125:44905'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20251209 08:58:52.746372 27899 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251209 08:58:52.747226 27899 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251209 08:58:52.753176 27907 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251209 08:58:52.753180 27905 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251209 08:58:52.753257 27899 server_base.cc:1047] running on GCE node
W20251209 08:58:52.753176 27904 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251209 08:58:52.753926 27899 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251209 08:58:52.754855 27899 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251209 08:58:52.756022 27899 hybrid_clock.cc:648] HybridClock initialized: now 1765270732755994 us; error 44 us; skew 500 ppm
Dec 09 08:58:52 dist-test-slave-8w6c krb5kdc[27379](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765270732, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.49.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251209 08:58:52.758898 27899 init.cc:377] Logged in from keytab as oryx/127.20.49.125@KRBTEST.COM (short username oryx)
I20251209 08:58:52.759999 27899 webserver.cc:492] Webserver started at http://127.20.49.125:40699/ using document root <none> and password file <none>
I20251209 08:58:52.760550 27899 fs_manager.cc:362] Metadata directory not provided
I20251209 08:58:52.760612 27899 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251209 08:58:52.760802 27899 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251209 08:58:52.762797 27899 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765270668420570-20677-0/minicluster-data/master-1/data/instance:
uuid: "798ad87d8a5a4392815120c18c40bedf"
format_stamp: "Formatted at 2025-12-09 08:58:52 on dist-test-slave-8w6c"
server_key: "dccade62f07344634c92e79adb235c8b"
server_key_iv: "b96eee0335605981f09e7722932e896e"
server_key_version: "encryptionkey@0"
I20251209 08:58:52.763306 27899 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765270668420570-20677-0/minicluster-data/master-1/wal/instance:
uuid: "798ad87d8a5a4392815120c18c40bedf"
format_stamp: "Formatted at 2025-12-09 08:58:52 on dist-test-slave-8w6c"
server_key: "dccade62f07344634c92e79adb235c8b"
server_key_iv: "b96eee0335605981f09e7722932e896e"
server_key_version: "encryptionkey@0"
I20251209 08:58:52.767174 27899 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20251209 08:58:52.769549 27914 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251209 08:58:52.770761 27899 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20251209 08:58:52.770895 27899 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765270668420570-20677-0/minicluster-data/master-1/data,/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765270668420570-20677-0/minicluster-data/master-1/wal
uuid: "798ad87d8a5a4392815120c18c40bedf"
format_stamp: "Formatted at 2025-12-09 08:58:52 on dist-test-slave-8w6c"
server_key: "dccade62f07344634c92e79adb235c8b"
server_key_iv: "b96eee0335605981f09e7722932e896e"
server_key_version: "encryptionkey@0"
I20251209 08:58:52.771000 27899 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765270668420570-20677-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765270668420570-20677-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765270668420570-20677-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251209 08:58:52.791690 27899 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251209 08:58:52.794901 27899 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251209 08:58:52.795113 27899 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251209 08:58:52.802826 27966 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.49.125:44905 every 8 connection(s)
I20251209 08:58:52.802829 27899 rpc_server.cc:307] RPC server started. Bound to: 127.20.49.125:44905
I20251209 08:58:52.803956 27899 server_base.cc:1179] Dumped server information to /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765270668420570-20677-0/minicluster-data/master-1/data/info.pb
I20251209 08:58:52.806852 27967 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251209 08:58:52.809480 20677 external_mini_cluster.cc:1428] Started /tmp/dist-test-task8XuDZV/build/debug/bin/kudu as pid 27899
I20251209 08:58:52.811430 27967 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.49.126" port: 42449 } has no permanent_uuid. Determining permanent_uuid...
Dec 09 08:58:52 dist-test-slave-8w6c krb5kdc[27379](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765270731, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.20.49.125@KRBTEST.COM
Dec 09 08:58:52 dist-test-slave-8w6c krb5kdc[27379](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765270732, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.49.125@KRBTEST.COM for oryx/127.20.49.126@KRBTEST.COM
I20251209 08:58:52.822705 27466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:58:52.813049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58458 (local address 127.20.49.126:42449)
1209 08:58:52.813191 (+ 142us) server_negotiation.cc:207] Beginning negotiation
1209 08:58:52.813195 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1209 08:58:52.813554 (+ 359us) server_negotiation.cc:408] Connection header received
1209 08:58:52.813675 (+ 121us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1209 08:58:52.813702 (+ 27us) server_negotiation.cc:462] Received NEGOTIATE request from client
1209 08:58:52.813768 (+ 66us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1209 08:58:52.813858 (+ 90us) server_negotiation.cc:227] Negotiated authn=SASL
1209 08:58:52.814949 (+ 1091us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:58:52.815691 (+ 742us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1209 08:58:52.816458 (+ 767us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:58:52.816703 (+ 245us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:58:52.819128 (+ 2425us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1209 08:58:52.819147 (+ 19us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1209 08:58:52.819149 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1209 08:58:52.819177 (+ 28us) server_negotiation.cc:923] calling sasl_server_start()
1209 08:58:52.820894 (+ 1717us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1209 08:58:52.821373 (+ 479us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1209 08:58:52.821377 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1209 08:58:52.821378 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1209 08:58:52.821426 (+ 48us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1209 08:58:52.821721 (+ 295us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1209 08:58:52.821725 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1209 08:58:52.821727 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1209 08:58:52.821913 (+ 186us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1209 08:58:52.822010 (+ 97us) server_negotiation.cc:1036] Waiting for connection context
1209 08:58:52.822392 (+ 382us) server_negotiation.cc:300] Negotiation successful
1209 08:58:52.822521 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20251209 08:58:52.823374 27970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:58:52.811001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36856 (local address 127.20.49.125:44905)
1209 08:58:52.811487 (+ 486us) server_negotiation.cc:207] Beginning negotiation
1209 08:58:52.811505 (+ 18us) server_negotiation.cc:400] Waiting for connection header
1209 08:58:52.811545 (+ 40us) server_negotiation.cc:408] Connection header received
1209 08:58:52.812141 (+ 596us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1209 08:58:52.812160 (+ 19us) server_negotiation.cc:462] Received NEGOTIATE request from client
1209 08:58:52.812474 (+ 314us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1209 08:58:52.812817 (+ 343us) server_negotiation.cc:227] Negotiated authn=SASL
1209 08:58:52.813851 (+ 1034us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:58:52.815003 (+ 1152us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1209 08:58:52.815816 (+ 813us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:58:52.816093 (+ 277us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:58:52.818766 (+ 2673us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1209 08:58:52.818786 (+ 20us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1209 08:58:52.818796 (+ 10us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1209 08:58:52.818819 (+ 23us) server_negotiation.cc:923] calling sasl_server_start()
1209 08:58:52.820967 (+ 2148us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1209 08:58:52.821367 (+ 400us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1209 08:58:52.821374 (+ 7us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1209 08:58:52.821376 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1209 08:58:52.821440 (+ 64us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1209 08:58:52.821667 (+ 227us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1209 08:58:52.821671 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1209 08:58:52.821673 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1209 08:58:52.822026 (+ 353us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1209 08:58:52.822212 (+ 186us) server_negotiation.cc:1036] Waiting for connection context
1209 08:58:52.822424 (+ 212us) server_negotiation.cc:300] Negotiation successful
1209 08:58:52.822612 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":327,"thread_start_us":147,"threads_started":1}
I20251209 08:58:52.823837 27971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:58:52.812977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.49.126:42449 (local address 127.0.0.1:58458)
1209 08:58:52.813406 (+ 429us) negotiation.cc:107] Waiting for socket to connect
1209 08:58:52.813447 (+ 41us) client_negotiation.cc:174] Beginning negotiation
1209 08:58:52.813576 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1209 08:58:52.813910 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1209 08:58:52.813916 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
1209 08:58:52.814460 (+ 544us) client_negotiation.cc:189] Negotiated authn=SASL
1209 08:58:52.814780 (+ 320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:58:52.814792 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:58:52.815816 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1209 08:58:52.815820 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1209 08:58:52.816331 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:58:52.816339 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:58:52.816470 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:58:52.816970 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1209 08:58:52.816994 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
1209 08:58:52.818996 (+ 2002us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1209 08:58:52.821029 (+ 2033us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1209 08:58:52.821036 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1209 08:58:52.821044 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
1209 08:58:52.821245 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1209 08:58:52.821518 (+ 273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1209 08:58:52.821524 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1209 08:58:52.821526 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1209 08:58:52.821598 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1209 08:58:52.822025 (+ 427us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1209 08:58:52.822034 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1209 08:58:52.822159 (+ 125us) client_negotiation.cc:769] Sending connection context
1209 08:58:52.822338 (+ 179us) client_negotiation.cc:240] Negotiation successful
1209 08:58:52.823724 (+ 1386us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":281,"mutex_wait_us":1131,"thread_start_us":140,"threads_started":1}
I20251209 08:58:52.825201 27967 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.49.125" port: 44905 } has no permanent_uuid. Determining permanent_uuid...
Dec 09 08:58:52 dist-test-slave-8w6c krb5kdc[27379](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765270732, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.49.125@KRBTEST.COM for oryx/127.20.49.125@KRBTEST.COM
I20251209 08:58:52.832525 27970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:58:52.825786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36866 (local address 127.20.49.125:44905)
1209 08:58:52.825875 (+ 89us) server_negotiation.cc:207] Beginning negotiation
1209 08:58:52.825881 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1209 08:58:52.825951 (+ 70us) server_negotiation.cc:408] Connection header received
1209 08:58:52.826062 (+ 111us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1209 08:58:52.826066 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1209 08:58:52.826103 (+ 37us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1209 08:58:52.826163 (+ 60us) server_negotiation.cc:227] Negotiated authn=SASL
1209 08:58:52.826672 (+ 509us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:58:52.827155 (+ 483us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1209 08:58:52.827683 (+ 528us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:58:52.827888 (+ 205us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:58:52.829707 (+ 1819us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1209 08:58:52.829726 (+ 19us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1209 08:58:52.829731 (+ 5us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1209 08:58:52.829760 (+ 29us) server_negotiation.cc:923] calling sasl_server_start()
1209 08:58:52.831134 (+ 1374us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1209 08:58:52.831506 (+ 372us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1209 08:58:52.831509 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1209 08:58:52.831513 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
1209 08:58:52.831554 (+ 41us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1209 08:58:52.831745 (+ 191us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1209 08:58:52.831748 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1209 08:58:52.831750 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1209 08:58:52.831889 (+ 139us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1209 08:58:52.831976 (+ 87us) server_negotiation.cc:1036] Waiting for connection context
1209 08:58:52.832197 (+ 221us) server_negotiation.cc:300] Negotiation successful
1209 08:58:52.832409 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":109,"server-negotiator.queue_time_us":34}
I20251209 08:58:52.832523 27971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:58:52.825754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.49.125:44905 (local address 127.0.0.1:36866)
1209 08:58:52.825875 (+ 121us) negotiation.cc:107] Waiting for socket to connect
1209 08:58:52.825891 (+ 16us) client_negotiation.cc:174] Beginning negotiation
1209 08:58:52.825986 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1209 08:58:52.826193 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1209 08:58:52.826196 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1209 08:58:52.826384 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
1209 08:58:52.826565 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:58:52.826571 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:58:52.827250 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1209 08:58:52.827254 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1209 08:58:52.827589 (+ 335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:58:52.827596 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:58:52.827697 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:58:52.828275 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1209 08:58:52.828286 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
1209 08:58:52.829568 (+ 1282us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1209 08:58:52.831251 (+ 1683us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1209 08:58:52.831254 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1209 08:58:52.831257 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
1209 08:58:52.831421 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1209 08:58:52.831630 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1209 08:58:52.831633 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1209 08:58:52.831635 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1209 08:58:52.831674 (+ 39us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1209 08:58:52.831973 (+ 299us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1209 08:58:52.831976 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1209 08:58:52.832047 (+ 71us) client_negotiation.cc:769] Sending connection context
1209 08:58:52.832147 (+ 100us) client_negotiation.cc:240] Negotiation successful
1209 08:58:52.832335 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20251209 08:58:52.835436 27967 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 798ad87d8a5a4392815120c18c40bedf: Bootstrap starting.
I20251209 08:58:52.837917 27967 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 798ad87d8a5a4392815120c18c40bedf: Neither blocks nor log segments found. Creating new log.
I20251209 08:58:52.838654 27967 log.cc:826] T 00000000000000000000000000000000 P 798ad87d8a5a4392815120c18c40bedf: Log is configured to *not* fsync() on all Append() calls
I20251209 08:58:52.840415 27967 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 798ad87d8a5a4392815120c18c40bedf: No bootstrap required, opened a new log
I20251209 08:58:52.842948 27967 raft_consensus.cc:359] T 00000000000000000000000000000000 P 798ad87d8a5a4392815120c18c40bedf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0a0d121c6bb474dac7c98dc7ec52d14" member_type: VOTER last_known_addr { host: "127.20.49.126" port: 42449 } } peers { permanent_uuid: "798ad87d8a5a4392815120c18c40bedf" member_type: VOTER last_known_addr { host: "127.20.49.125" port: 44905 } }
I20251209 08:58:52.843120 27967 raft_consensus.cc:385] T 00000000000000000000000000000000 P 798ad87d8a5a4392815120c18c40bedf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251209 08:58:52.843156 27967 raft_consensus.cc:740] T 00000000000000000000000000000000 P 798ad87d8a5a4392815120c18c40bedf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 798ad87d8a5a4392815120c18c40bedf, State: Initialized, Role: FOLLOWER
I20251209 08:58:52.843541 27967 consensus_queue.cc:260] T 00000000000000000000000000000000 P 798ad87d8a5a4392815120c18c40bedf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0a0d121c6bb474dac7c98dc7ec52d14" member_type: VOTER last_known_addr { host: "127.20.49.126" port: 42449 } } peers { permanent_uuid: "798ad87d8a5a4392815120c18c40bedf" member_type: VOTER last_known_addr { host: "127.20.49.125" port: 44905 } }
I20251209 08:58:52.844013 27973 sys_catalog.cc:455] T 00000000000000000000000000000000 P 798ad87d8a5a4392815120c18c40bedf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0a0d121c6bb474dac7c98dc7ec52d14" member_type: VOTER last_known_addr { host: "127.20.49.126" port: 42449 } } peers { permanent_uuid: "798ad87d8a5a4392815120c18c40bedf" member_type: VOTER last_known_addr { host: "127.20.49.125" port: 44905 } } }
I20251209 08:58:52.844133 27973 sys_catalog.cc:458] T 00000000000000000000000000000000 P 798ad87d8a5a4392815120c18c40bedf [sys.catalog]: This master's current role is: FOLLOWER
I20251209 08:58:52.844614 27967 sys_catalog.cc:565] T 00000000000000000000000000000000 P 798ad87d8a5a4392815120c18c40bedf [sys.catalog]: configured and running, proceeding with master startup.
I20251209 08:58:52.846727 27973 raft_consensus.cc:493] T 00000000000000000000000000000000 P 798ad87d8a5a4392815120c18c40bedf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251209 08:58:52.846993 27973 raft_consensus.cc:515] T 00000000000000000000000000000000 P 798ad87d8a5a4392815120c18c40bedf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0a0d121c6bb474dac7c98dc7ec52d14" member_type: VOTER last_known_addr { host: "127.20.49.126" port: 42449 } } peers { permanent_uuid: "798ad87d8a5a4392815120c18c40bedf" member_type: VOTER last_known_addr { host: "127.20.49.125" port: 44905 } }
I20251209 08:58:52.847913 27973 leader_election.cc:290] T 00000000000000000000000000000000 P 798ad87d8a5a4392815120c18c40bedf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c0a0d121c6bb474dac7c98dc7ec52d14 (127.20.49.126:42449)
I20251209 08:58:52.848472 27437 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "798ad87d8a5a4392815120c18c40bedf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0a0d121c6bb474dac7c98dc7ec52d14" is_pre_election: true
I20251209 08:58:52.849231 27917 leader_election.cc:304] T 00000000000000000000000000000000 P 798ad87d8a5a4392815120c18c40bedf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 798ad87d8a5a4392815120c18c40bedf; no voters: c0a0d121c6bb474dac7c98dc7ec52d14
W20251209 08:58:52.849401 27984 catalog_manager.cc:1568] T 00000000000000000000000000000000 P 798ad87d8a5a4392815120c18c40bedf: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20251209 08:58:52.849469 27984 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20251209 08:58:52.849452 27973 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 798ad87d8a5a4392815120c18c40bedf [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251209 08:58:52.851255 27899 master_runner.cc:428] Detected that this master 798ad87d8a5a4392815120c18c40bedf is joining an existing cluster
I20251209 08:58:52.851308 27899 master_runner.cc:432] Initiating AddMaster RPC to add 127.20.49.125:44905
I20251209 08:58:52.861487 27990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:58:52.853216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.49.126:42449 (local address 127.0.0.1:58474)
1209 08:58:52.853512 (+ 296us) negotiation.cc:107] Waiting for socket to connect
1209 08:58:52.853528 (+ 16us) client_negotiation.cc:174] Beginning negotiation
1209 08:58:52.853636 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1209 08:58:52.854060 (+ 424us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1209 08:58:52.854063 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1209 08:58:52.854341 (+ 278us) client_negotiation.cc:189] Negotiated authn=SASL
1209 08:58:52.854875 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:58:52.854882 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:58:52.856019 (+ 1137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1209 08:58:52.856022 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1209 08:58:52.856881 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:58:52.856893 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:58:52.857030 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:58:52.857719 (+ 689us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1209 08:58:52.857738 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
1209 08:58:52.858058 (+ 320us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1209 08:58:52.860089 (+ 2031us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1209 08:58:52.860092 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1209 08:58:52.860094 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1209 08:58:52.860258 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1209 08:58:52.860511 (+ 253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1209 08:58:52.860513 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1209 08:58:52.860514 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1209 08:58:52.860556 (+ 42us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1209 08:58:52.860974 (+ 418us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1209 08:58:52.860977 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1209 08:58:52.861080 (+ 103us) client_negotiation.cc:769] Sending connection context
1209 08:58:52.861202 (+ 122us) client_negotiation.cc:240] Negotiation successful
1209 08:58:52.861318 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":128,"threads_started":1}
I20251209 08:58:52.861574 27466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:58:52.853287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58474 (local address 127.20.49.126:42449)
1209 08:58:52.853639 (+ 352us) server_negotiation.cc:207] Beginning negotiation
1209 08:58:52.853644 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1209 08:58:52.853660 (+ 16us) server_negotiation.cc:408] Connection header received
1209 08:58:52.853759 (+ 99us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1209 08:58:52.853762 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1209 08:58:52.853820 (+ 58us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1209 08:58:52.853880 (+ 60us) server_negotiation.cc:227] Negotiated authn=SASL
1209 08:58:52.855061 (+ 1181us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:58:52.855864 (+ 803us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1209 08:58:52.857020 (+ 1156us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:58:52.857226 (+ 206us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:58:52.858196 (+ 970us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1209 08:58:52.858216 (+ 20us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1209 08:58:52.858220 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1209 08:58:52.858251 (+ 31us) server_negotiation.cc:923] calling sasl_server_start()
1209 08:58:52.859943 (+ 1692us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1209 08:58:52.860357 (+ 414us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1209 08:58:52.860361 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1209 08:58:52.860363 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1209 08:58:52.860424 (+ 61us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1209 08:58:52.860634 (+ 210us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1209 08:58:52.860638 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1209 08:58:52.860640 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1209 08:58:52.860842 (+ 202us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1209 08:58:52.860953 (+ 111us) server_negotiation.cc:1036] Waiting for connection context
1209 08:58:52.861270 (+ 317us) server_negotiation.cc:300] Negotiation successful
1209 08:58:52.861401 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":265}
Dec 09 08:58:52 dist-test-slave-8w6c krb5kdc[27379](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765270731, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.49.126@KRBTEST.COM for oryx/127.20.49.125@KRBTEST.COM
I20251209 08:58:52.873972 27970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:58:52.853779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36880 (local address 127.20.49.125:44905)
1209 08:58:52.853874 (+ 95us) server_negotiation.cc:207] Beginning negotiation
1209 08:58:52.853879 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1209 08:58:52.853891 (+ 12us) server_negotiation.cc:408] Connection header received
1209 08:58:52.854017 (+ 126us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1209 08:58:52.854020 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1209 08:58:52.854057 (+ 37us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1209 08:58:52.854104 (+ 47us) server_negotiation.cc:227] Negotiated authn=SASL
1209 08:58:52.854970 (+ 866us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:58:52.855423 (+ 453us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1209 08:58:52.856042 (+ 619us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:58:52.856235 (+ 193us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:58:52.857159 (+ 924us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1209 08:58:52.857180 (+ 21us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1209 08:58:52.857185 (+ 5us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1209 08:58:52.857212 (+ 27us) server_negotiation.cc:923] calling sasl_server_start()
1209 08:58:52.872226 (+ 15014us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1209 08:58:52.872711 (+ 485us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1209 08:58:52.872716 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1209 08:58:52.872718 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1209 08:58:52.872768 (+ 50us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1209 08:58:52.873060 (+ 292us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1209 08:58:52.873064 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1209 08:58:52.873066 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1209 08:58:52.873253 (+ 187us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1209 08:58:52.873348 (+ 95us) server_negotiation.cc:1036] Waiting for connection context
1209 08:58:52.873615 (+ 267us) server_negotiation.cc:300] Negotiation successful
1209 08:58:52.873768 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20251209 08:58:52.873988 27991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:58:52.853236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.49.125:44905 (local address 127.0.0.1:36880)
1209 08:58:52.853665 (+ 429us) negotiation.cc:107] Waiting for socket to connect
1209 08:58:52.853720 (+ 55us) client_negotiation.cc:174] Beginning negotiation
1209 08:58:52.853918 (+ 198us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1209 08:58:52.854219 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1209 08:58:52.854222 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1209 08:58:52.854509 (+ 287us) client_negotiation.cc:189] Negotiated authn=SASL
1209 08:58:52.854838 (+ 329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:58:52.854847 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:58:52.855536 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1209 08:58:52.855540 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1209 08:58:52.855904 (+ 364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:58:52.855911 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:58:52.856050 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:58:52.856619 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1209 08:58:52.856637 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
1209 08:58:52.857039 (+ 402us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1209 08:58:52.872338 (+ 15299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1209 08:58:52.872341 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1209 08:58:52.872344 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
1209 08:58:52.872583 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1209 08:58:52.872882 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1209 08:58:52.872889 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1209 08:58:52.872894 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
1209 08:58:52.872965 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1209 08:58:52.873369 (+ 404us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1209 08:58:52.873373 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1209 08:58:52.873463 (+ 90us) client_negotiation.cc:769] Sending connection context
1209 08:58:52.873557 (+ 94us) client_negotiation.cc:240] Negotiation successful
1209 08:58:52.873667 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":131,"threads_started":1}
I20251209 08:58:52.876515 27992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:58:52.866231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.49.125:44905 (local address 127.0.0.1:36890)
1209 08:58:52.866642 (+ 411us) negotiation.cc:107] Waiting for socket to connect
1209 08:58:52.866676 (+ 34us) client_negotiation.cc:174] Beginning negotiation
1209 08:58:52.866829 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1209 08:58:52.867165 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1209 08:58:52.867173 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
1209 08:58:52.867716 (+ 543us) client_negotiation.cc:189] Negotiated authn=SASL
1209 08:58:52.867925 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:58:52.867932 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:58:52.868708 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1209 08:58:52.868711 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1209 08:58:52.869075 (+ 364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:58:52.869082 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:58:52.869192 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:58:52.869977 (+ 785us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1209 08:58:52.870002 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
1209 08:58:52.871906 (+ 1904us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1209 08:58:52.874735 (+ 2829us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1209 08:58:52.874747 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1209 08:58:52.874753 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
1209 08:58:52.875191 (+ 438us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1209 08:58:52.875429 (+ 238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1209 08:58:52.875432 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1209 08:58:52.875434 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1209 08:58:52.875489 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1209 08:58:52.875888 (+ 399us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1209 08:58:52.875895 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1209 08:58:52.876022 (+ 127us) client_negotiation.cc:769] Sending connection context
1209 08:58:52.876169 (+ 147us) client_negotiation.cc:240] Negotiation successful
1209 08:58:52.876377 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":113,"threads_started":1}
I20251209 08:58:52.877040 27993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:58:52.866271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36890 (local address 127.20.49.125:44905)
1209 08:58:52.866560 (+ 289us) server_negotiation.cc:207] Beginning negotiation
1209 08:58:52.866565 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1209 08:58:52.866757 (+ 192us) server_negotiation.cc:408] Connection header received
1209 08:58:52.866961 (+ 204us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1209 08:58:52.866965 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1209 08:58:52.867031 (+ 66us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1209 08:58:52.867115 (+ 84us) server_negotiation.cc:227] Negotiated authn=SASL
1209 08:58:52.868052 (+ 937us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:58:52.868587 (+ 535us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1209 08:58:52.869180 (+ 593us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:58:52.869387 (+ 207us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:58:52.872119 (+ 2732us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1209 08:58:52.872146 (+ 27us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1209 08:58:52.872152 (+ 6us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1209 08:58:52.872186 (+ 34us) server_negotiation.cc:923] calling sasl_server_start()
1209 08:58:52.874596 (+ 2410us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1209 08:58:52.875292 (+ 696us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1209 08:58:52.875296 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1209 08:58:52.875298 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1209 08:58:52.875346 (+ 48us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1209 08:58:52.875599 (+ 253us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1209 08:58:52.875602 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1209 08:58:52.875604 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1209 08:58:52.875742 (+ 138us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1209 08:58:52.875829 (+ 87us) server_negotiation.cc:1036] Waiting for connection context
1209 08:58:52.876737 (+ 908us) server_negotiation.cc:300] Negotiation successful
1209 08:58:52.876864 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":210,"thread_start_us":101,"threads_started":1}
I20251209 08:58:52.878496 27427 catalog_manager.cc:7141] Initiating ChangeConfig request to add master 127.20.49.125:44905: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "798ad87d8a5a4392815120c18c40bedf"
member_type: NON_VOTER
last_known_addr {
host: "127.20.49.125"
port: 44905
}
attrs {
promote: true
}
}
dest_uuid: "c0a0d121c6bb474dac7c98dc7ec52d14"
cas_config_opid_index: -1
I20251209 08:58:52.879307 27427 consensus_queue.cc:237] T 00000000000000000000000000000000 P c0a0d121c6bb474dac7c98dc7ec52d14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c0a0d121c6bb474dac7c98dc7ec52d14" member_type: VOTER last_known_addr { host: "127.20.49.126" port: 42449 } } peers { permanent_uuid: "798ad87d8a5a4392815120c18c40bedf" member_type: NON_VOTER last_known_addr { host: "127.20.49.125" port: 44905 } attrs { promote: true } }
I20251209 08:58:52.881202 27996 raft_consensus.cc:2955] T 00000000000000000000000000000000 P c0a0d121c6bb474dac7c98dc7ec52d14 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 798ad87d8a5a4392815120c18c40bedf (127.20.49.125) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c0a0d121c6bb474dac7c98dc7ec52d14" member_type: VOTER last_known_addr { host: "127.20.49.126" port: 42449 } } peers { permanent_uuid: "798ad87d8a5a4392815120c18c40bedf" member_type: NON_VOTER last_known_addr { host: "127.20.49.125" port: 44905 } attrs { promote: true } } }
I20251209 08:58:52.882166 27941 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 798ad87d8a5a4392815120c18c40bedf [term 0 FOLLOWER]: Advancing to term 1
I20251209 08:58:52.882788 27996 catalog_manager.cc:7145] Successfully completed master ChangeConfig request to add master 127.20.49.125:44905
I20251209 08:58:52.883203 27997 sys_catalog.cc:455] T 00000000000000000000000000000000 P c0a0d121c6bb474dac7c98dc7ec52d14 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "c0a0d121c6bb474dac7c98dc7ec52d14" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c0a0d121c6bb474dac7c98dc7ec52d14" member_type: VOTER last_known_addr { host: "127.20.49.126" port: 42449 } } peers { permanent_uuid: "798ad87d8a5a4392815120c18c40bedf" member_type: NON_VOTER last_known_addr { host: "127.20.49.125" port: 44905 } attrs { promote: true } } }
I20251209 08:58:52.883455 27899 master.cc:561] Master@127.20.49.125:44905 shutting down...
I20251209 08:58:52.883651 27941 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 798ad87d8a5a4392815120c18c40bedf [term 1 FOLLOWER]: Refusing update from remote peer c0a0d121c6bb474dac7c98dc7ec52d14: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20251209 08:58:52.887795 27899 raft_consensus.cc:2243] T 00000000000000000000000000000000 P 798ad87d8a5a4392815120c18c40bedf [term 1 FOLLOWER]: Raft consensus shutting down.
I20251209 08:58:52.888213 27899 raft_consensus.cc:2272] T 00000000000000000000000000000000 P 798ad87d8a5a4392815120c18c40bedf [term 1 FOLLOWER]: Raft consensus is shut down!
I20251209 08:58:52.888343 27899 tablet_replica.cc:333] T 00000000000000000000000000000000 P 798ad87d8a5a4392815120c18c40bedf: stopping tablet replica
I20251209 08:58:52.887559 27996 consensus_queue.cc:1048] T 00000000000000000000000000000000 P c0a0d121c6bb474dac7c98dc7ec52d14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "798ad87d8a5a4392815120c18c40bedf" member_type: NON_VOTER last_known_addr { host: "127.20.49.125" port: 44905 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.001s
WARNING: no policy specified for oryx/127.20.49.124@KRBTEST.COM; defaulting to no policy
I20251209 08:58:52.887456 27997 sys_catalog.cc:458] T 00000000000000000000000000000000 P c0a0d121c6bb474dac7c98dc7ec52d14 [sys.catalog]: This master's current role is: LEADER
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.20.49.124@KRBTEST.COM" created.
W20251209 08:58:52.891690 27411 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20251209 08:58:52.892643 27411 consensus_peers.cc:597] T 00000000000000000000000000000000 P c0a0d121c6bb474dac7c98dc7ec52d14 -> Peer 798ad87d8a5a4392815120c18c40bedf (127.20.49.125:44905): Couldn't send request to peer 798ad87d8a5a4392815120c18c40bedf. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20251209 08:58:52.896027 27899 master.cc:583] Master@127.20.49.125:44905 shutdown complete.
I20251209 08:58:52.896092 27899 master_runner.cc:305] Clearing existing system tablet
I20251209 08:58:52.896329 27899 fs_manager.cc:362] Metadata directory not provided
I20251209 08:58:52.896396 27899 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251209 08:58:52.897835 27899 fs_manager.cc:714] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20251209 08:58:52.898818 28002 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251209 08:58:52.899103 27899 fs_report.cc:389] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765270668420570-20677-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251209 08:58:52.899236 27899 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20251209 08:58:52.899307 27899 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765270668420570-20677-0/minicluster-data/master-1/data,/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765270668420570-20677-0/minicluster-data/master-1/wal
uuid: "798ad87d8a5a4392815120c18c40bedf"
format_stamp: "Formatted at 2025-12-09 08:58:52 on dist-test-slave-8w6c"
server_key: "dccade62f07344634c92e79adb235c8b"
server_key_iv: "b96eee0335605981f09e7722932e896e"
server_key_version: "encryptionkey@0"
I20251209 08:58:52.901371 27899 ts_tablet_manager.cc:1916] T 00000000000000000000000000000000 P 798ad87d8a5a4392815120c18c40bedf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251209 08:58:52.904399 27899 ts_tablet_manager.cc:1929] T 00000000000000000000000000000000 P 798ad87d8a5a4392815120c18c40bedf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20251209 08:58:52.904500 27899 log.cc:1199] T 00000000000000000000000000000000 P 798ad87d8a5a4392815120c18c40bedf: Deleting WAL directory at /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765270668420570-20677-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20251209 08:58:52.904770 27899 ts_tablet_manager.cc:1950] T 00000000000000000000000000000000 P 798ad87d8a5a4392815120c18c40bedf: Deleting consensus metadata
I20251209 08:58:52.905463 27899 master_runner.cc:315] Copying system tablet from 127.20.49.126:42449
I20251209 08:58:52.906772 27899 tablet_copy_client.cc:323] T 00000000000000000000000000000000 P 798ad87d8a5a4392815120c18c40bedf: tablet copy: Beginning tablet copy session from remote peer at address 127.20.49.126:42449
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.20.49.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765270668420570-20677-0/krb5kdc/oryx.keytab.
Entry for principal oryx/127.20.49.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765270668420570-20677-0/krb5kdc/oryx.keytab.
I20251209 08:58:52.915153 28008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:58:52.907467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.49.126:42449 (local address 127.0.0.1:58486)
1209 08:58:52.907746 (+ 279us) negotiation.cc:107] Waiting for socket to connect
1209 08:58:52.907764 (+ 18us) client_negotiation.cc:174] Beginning negotiation
1209 08:58:52.907906 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1209 08:58:52.908205 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1209 08:58:52.908210 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
1209 08:58:52.908463 (+ 253us) client_negotiation.cc:189] Negotiated authn=SASL
1209 08:58:52.908752 (+ 289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:58:52.908771 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:58:52.909968 (+ 1197us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1209 08:58:52.909975 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1209 08:58:52.910580 (+ 605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:58:52.910590 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:58:52.910696 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:58:52.911260 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1209 08:58:52.911273 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
1209 08:58:52.911604 (+ 331us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1209 08:58:52.913767 (+ 2163us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1209 08:58:52.913771 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1209 08:58:52.913773 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1209 08:58:52.913985 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1209 08:58:52.914249 (+ 264us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1209 08:58:52.914252 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1209 08:58:52.914253 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1209 08:58:52.914306 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1209 08:58:52.914708 (+ 402us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1209 08:58:52.914710 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1209 08:58:52.914798 (+ 88us) client_negotiation.cc:769] Sending connection context
1209 08:58:52.914897 (+ 99us) client_negotiation.cc:240] Negotiation successful
1209 08:58:52.915014 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":105,"threads_started":1}
I20251209 08:58:52.915233 27466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:58:52.907460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58486 (local address 127.20.49.126:42449)
1209 08:58:52.907639 (+ 179us) server_negotiation.cc:207] Beginning negotiation
1209 08:58:52.907645 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1209 08:58:52.907869 (+ 224us) server_negotiation.cc:408] Connection header received
1209 08:58:52.908011 (+ 142us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1209 08:58:52.908015 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1209 08:58:52.908076 (+ 61us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1209 08:58:52.908167 (+ 91us) server_negotiation.cc:227] Negotiated authn=SASL
1209 08:58:52.908902 (+ 735us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:58:52.909833 (+ 931us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1209 08:58:52.910697 (+ 864us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:58:52.910898 (+ 201us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:58:52.911734 (+ 836us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1209 08:58:52.911757 (+ 23us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1209 08:58:52.911761 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1209 08:58:52.911789 (+ 28us) server_negotiation.cc:923] calling sasl_server_start()
1209 08:58:52.913605 (+ 1816us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1209 08:58:52.914088 (+ 483us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1209 08:58:52.914092 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1209 08:58:52.914094 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1209 08:58:52.914149 (+ 55us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1209 08:58:52.914398 (+ 249us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1209 08:58:52.914401 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1209 08:58:52.914403 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1209 08:58:52.914589 (+ 186us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1209 08:58:52.914687 (+ 98us) server_negotiation.cc:1036] Waiting for connection context
1209 08:58:52.914960 (+ 273us) server_negotiation.cc:300] Negotiation successful
1209 08:58:52.915085 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20251209 08:58:52.915874 27447 tablet_copy_service.cc:140] P c0a0d121c6bb474dac7c98dc7ec52d14: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 798ad87d8a5a4392815120c18c40bedf ({username='oryx', principal='oryx/127.20.49.125@KRBTEST.COM'} at 127.0.0.1:58486)
I20251209 08:58:52.916031 27447 tablet_copy_service.cc:161] P c0a0d121c6bb474dac7c98dc7ec52d14: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 798ad87d8a5a4392815120c18c40bedf at {username='oryx', principal='oryx/127.20.49.125@KRBTEST.COM'} at 127.0.0.1:58486: session id = 798ad87d8a5a4392815120c18c40bedf-00000000000000000000000000000000
I20251209 08:58:52.917732 27447 tablet_copy_source_session.cc:215] T 00000000000000000000000000000000 P c0a0d121c6bb474dac7c98dc7ec52d14: Tablet Copy: opened 0 blocks and 1 log segments
I20251209 08:58:52.919450 27899 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20251209 08:58:52.921753 27899 tablet_copy_client.cc:806] T 00000000000000000000000000000000 P 798ad87d8a5a4392815120c18c40bedf: tablet copy: Starting download of 0 data blocks...
I20251209 08:58:52.921944 27899 tablet_copy_client.cc:670] T 00000000000000000000000000000000 P 798ad87d8a5a4392815120c18c40bedf: tablet copy: Starting download of 1 WAL segments...
WARNING: no policy specified for HTTP/127.20.49.124@KRBTEST.COM; defaulting to no policy
I20251209 08:58:52.923906 27899 tablet_copy_client.cc:538] T 00000000000000000000000000000000 P 798ad87d8a5a4392815120c18c40bedf: tablet copy: Tablet Copy complete. Replacing tablet superblock.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.20.49.124@KRBTEST.COM" created.
I20251209 08:58:52.926492 27447 tablet_copy_service.cc:342] P c0a0d121c6bb474dac7c98dc7ec52d14: Request end of tablet copy session 798ad87d8a5a4392815120c18c40bedf-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.20.49.125@KRBTEST.COM'} at 127.0.0.1:58486
I20251209 08:58:52.926625 27447 tablet_copy_service.cc:434] P c0a0d121c6bb474dac7c98dc7ec52d14: ending tablet copy session 798ad87d8a5a4392815120c18c40bedf-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 798ad87d8a5a4392815120c18c40bedf
W20251209 08:58:52.931453 27899 builtin_ntp.cc:688] coult not shutdown socket: Network error: shutdown error: Transport endpoint is not connected (error 107)
I20251209 08:58:52.934509 27899 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251209 08:58:52.936473 28011 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251209 08:58:52.936477 28014 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251209 08:58:52.936643 27899 server_base.cc:1047] running on GCE node
W20251209 08:58:52.936522 28012 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251209 08:58:52.937022 27899 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251209 08:58:52.937315 27899 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251209 08:58:52.938468 27899 hybrid_clock.cc:648] HybridClock initialized: now 1765270732938454 us; error 37 us; skew 500 ppm
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.20.49.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765270668420570-20677-0/krb5kdc/oryx.keytab.
Entry for principal HTTP/127.20.49.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765270668420570-20677-0/krb5kdc/oryx.keytab.
Dec 09 08:58:52 dist-test-slave-8w6c krb5kdc[27379](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765270732, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.49.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251209 08:58:52.944422 27899 init.cc:377] Logged in from keytab as oryx/127.20.49.125@KRBTEST.COM (short username oryx)
I20251209 08:58:52.944507 20677 external_mini_cluster.cc:1366] Running /tmp/dist-test-task8XuDZV/build/debug/bin/kudu
/tmp/dist-test-task8XuDZV/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765270668420570-20677-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765270668420570-20677-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765270668420570-20677-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765270668420570-20677-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.49.124:44451
--webserver_interface=127.20.49.124
--webserver_port=0
--keytab_file=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765270668420570-20677-0/krb5kdc/oryx.keytab
--principal=oryx/127.20.49.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.20.49.84:40253
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.49.126:42449,127.20.49.125:44905,127.20.49.124:44451
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765270668420570-20677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765270668420570-20677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765270668420570-20677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20251209 08:58:52.945107 27899 webserver.cc:492] Webserver started at http://127.20.49.125:33095/ using document root <none> and password file <none>
I20251209 08:58:52.945343 27899 fs_manager.cc:362] Metadata directory not provided
I20251209 08:58:52.945394 27899 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251209 08:58:52.946970 27899 fs_manager.cc:714] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20251209 08:58:52.947914 28022 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251209 08:58:52.948176 27899 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20251209 08:58:52.948257 27899 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765270668420570-20677-0/minicluster-data/master-1/data,/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765270668420570-20677-0/minicluster-data/master-1/wal
uuid: "798ad87d8a5a4392815120c18c40bedf"
format_stamp: "Formatted at 2025-12-09 08:58:52 on dist-test-slave-8w6c"
server_key: "dccade62f07344634c92e79adb235c8b"
server_key_iv: "b96eee0335605981f09e7722932e896e"
server_key_version: "encryptionkey@0"
I20251209 08:58:52.948446 27899 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765270668420570-20677-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765270668420570-20677-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765270668420570-20677-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251209 08:58:52.970865 27899 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251209 08:58:52.972836 27899 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251209 08:58:52.977926 27899 rpc_server.cc:307] RPC server started. Bound to: 127.20.49.125:44905
I20251209 08:58:52.978506 27899 server_base.cc:1179] Dumped server information to /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765270668420570-20677-0/minicluster-data/master-1/data/info.pb
I20251209 08:58:52.978904 28074 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.49.125:44905 every 8 connection(s)
I20251209 08:58:52.980448 28075 sys_catalog.cc:263] Verifying existing consensus state
I20251209 08:58:52.981221 28075 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 798ad87d8a5a4392815120c18c40bedf: Bootstrap starting.
I20251209 08:58:52.989516 28075 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 798ad87d8a5a4392815120c18c40bedf: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251209 08:58:52.989879 28075 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 798ad87d8a5a4392815120c18c40bedf: Bootstrap complete.
I20251209 08:58:52.990435 28075 raft_consensus.cc:359] T 00000000000000000000000000000000 P 798ad87d8a5a4392815120c18c40bedf [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c0a0d121c6bb474dac7c98dc7ec52d14" member_type: VOTER last_known_addr { host: "127.20.49.126" port: 42449 } } peers { permanent_uuid: "798ad87d8a5a4392815120c18c40bedf" member_type: NON_VOTER last_known_addr { host: "127.20.49.125" port: 44905 } attrs { promote: true } }
I20251209 08:58:52.990607 28075 raft_consensus.cc:740] T 00000000000000000000000000000000 P 798ad87d8a5a4392815120c18c40bedf [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 798ad87d8a5a4392815120c18c40bedf, State: Initialized, Role: LEARNER
I20251209 08:58:52.990752 28075 consensus_queue.cc:260] T 00000000000000000000000000000000 P 798ad87d8a5a4392815120c18c40bedf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c0a0d121c6bb474dac7c98dc7ec52d14" member_type: VOTER last_known_addr { host: "127.20.49.126" port: 42449 } } peers { permanent_uuid: "798ad87d8a5a4392815120c18c40bedf" member_type: NON_VOTER last_known_addr { host: "127.20.49.125" port: 44905 } attrs { promote: true } }
I20251209 08:58:52.991096 28078 sys_catalog.cc:455] T 00000000000000000000000000000000 P 798ad87d8a5a4392815120c18c40bedf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c0a0d121c6bb474dac7c98dc7ec52d14" member_type: VOTER last_known_addr { host: "127.20.49.126" port: 42449 } } peers { permanent_uuid: "798ad87d8a5a4392815120c18c40bedf" member_type: NON_VOTER last_known_addr { host: "127.20.49.125" port: 44905 } attrs { promote: true } } }
I20251209 08:58:52.991261 28078 sys_catalog.cc:458] T 00000000000000000000000000000000 P 798ad87d8a5a4392815120c18c40bedf [sys.catalog]: This master's current role is: LEARNER
I20251209 08:58:52.991264 28075 sys_catalog.cc:565] T 00000000000000000000000000000000 P 798ad87d8a5a4392815120c18c40bedf [sys.catalog]: configured and running, proceeding with master startup.
I20251209 08:58:52.994777 28089 catalog_manager.cc:1269] Loaded cluster ID: 51c444202dc5408cb61915bc1a2c504c
I20251209 08:58:52.994881 28089 catalog_manager.cc:1562] T 00000000000000000000000000000000 P 798ad87d8a5a4392815120c18c40bedf: loading cluster ID for follower catalog manager: success
I20251209 08:58:52.996464 28089 catalog_manager.cc:1584] T 00000000000000000000000000000000 P 798ad87d8a5a4392815120c18c40bedf: acquiring CA information for follower catalog manager: success
I20251209 08:58:52.997576 28089 catalog_manager.cc:1612] T 00000000000000000000000000000000 P 798ad87d8a5a4392815120c18c40bedf: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20251209 08:58:53.074793 27620 heartbeater.cc:499] Master 127.20.49.126:42449 was elected leader, sending a full tablet report...
W20251209 08:58:53.092511 28020 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251209 08:58:53.092797 28020 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251209 08:58:53.092876 28020 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251209 08:58:53.096517 28020 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251209 08:58:53.096604 28020 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251209 08:58:53.096629 28020 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251209 08:58:53.096649 28020 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251209 08:58:53.096668 28020 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251209 08:58:53.101236 28020 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.49.84:40253
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765270668420570-20677-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765270668420570-20677-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.20.49.126:42449,127.20.49.125:44905,127.20.49.124:44451
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765270668420570-20677-0/krb5kdc/oryx.keytab
--principal=oryx/127.20.49.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.49.124:44451
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765270668420570-20677-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.20.49.124
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28020
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765270668420570-20677-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 468897755969d686fb08386f42f7835685e7953a
build type DEBUG
built by None at 09 Dec 2025 08:43:15 UTC on 24a791456cd2
build id 9426
I20251209 08:58:53.102459 28020 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251209 08:58:53.103369 28020 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251209 08:58:53.109249 28095 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251209 08:58:53.109253 28097 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251209 08:58:53.109274 28094 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251209 08:58:53.109650 28020 server_base.cc:1047] running on GCE node
I20251209 08:58:53.110210 28020 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251209 08:58:53.111132 28020 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251209 08:58:53.112282 28020 hybrid_clock.cc:648] HybridClock initialized: now 1765270733112276 us; error 59 us; skew 500 ppm
Dec 09 08:58:53 dist-test-slave-8w6c krb5kdc[27379](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765270733, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.49.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251209 08:58:53.115068 28020 init.cc:377] Logged in from keytab as oryx/127.20.49.124@KRBTEST.COM (short username oryx)
I20251209 08:58:53.116128 28020 webserver.cc:492] Webserver started at http://127.20.49.124:45647/ using document root <none> and password file <none>
I20251209 08:58:53.116664 28020 fs_manager.cc:362] Metadata directory not provided
I20251209 08:58:53.116710 28020 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251209 08:58:53.116870 28020 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251209 08:58:53.118757 28020 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765270668420570-20677-0/minicluster-data/master-2/data/instance:
uuid: "0364ed40f171452f9fa244fa50f6a6c3"
format_stamp: "Formatted at 2025-12-09 08:58:53 on dist-test-slave-8w6c"
server_key: "ded07db4bbf58c7410644fc982d50862"
server_key_iv: "51de27466f9232ff51b1b7f797268a16"
server_key_version: "encryptionkey@0"
I20251209 08:58:53.119240 28020 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765270668420570-20677-0/minicluster-data/master-2/wal/instance:
uuid: "0364ed40f171452f9fa244fa50f6a6c3"
format_stamp: "Formatted at 2025-12-09 08:58:53 on dist-test-slave-8w6c"
server_key: "ded07db4bbf58c7410644fc982d50862"
server_key_iv: "51de27466f9232ff51b1b7f797268a16"
server_key_version: "encryptionkey@0"
I20251209 08:58:53.122839 28020 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20251209 08:58:53.125164 28104 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251209 08:58:53.126255 28020 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20251209 08:58:53.126410 28020 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765270668420570-20677-0/minicluster-data/master-2/data,/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765270668420570-20677-0/minicluster-data/master-2/wal
uuid: "0364ed40f171452f9fa244fa50f6a6c3"
format_stamp: "Formatted at 2025-12-09 08:58:53 on dist-test-slave-8w6c"
server_key: "ded07db4bbf58c7410644fc982d50862"
server_key_iv: "51de27466f9232ff51b1b7f797268a16"
server_key_version: "encryptionkey@0"
I20251209 08:58:53.126510 28020 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765270668420570-20677-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765270668420570-20677-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765270668420570-20677-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251209 08:58:53.136083 28020 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251209 08:58:53.139499 28020 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251209 08:58:53.139685 28020 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251209 08:58:53.147073 28020 rpc_server.cc:307] RPC server started. Bound to: 127.20.49.124:44451
I20251209 08:58:53.147079 28156 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.49.124:44451 every 8 connection(s)
I20251209 08:58:53.148156 28020 server_base.cc:1179] Dumped server information to /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765270668420570-20677-0/minicluster-data/master-2/data/info.pb
I20251209 08:58:53.151059 28157 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251209 08:58:53.152388 20677 external_mini_cluster.cc:1428] Started /tmp/dist-test-task8XuDZV/build/debug/bin/kudu as pid 28020
I20251209 08:58:53.155419 28157 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.49.126" port: 42449 } has no permanent_uuid. Determining permanent_uuid...
Dec 09 08:58:53 dist-test-slave-8w6c krb5kdc[27379](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765270731, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.20.49.124@KRBTEST.COM
Dec 09 08:58:53 dist-test-slave-8w6c krb5kdc[27379](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765270733, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.49.124@KRBTEST.COM for oryx/127.20.49.126@KRBTEST.COM
I20251209 08:58:53.166173 27466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:58:53.157300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58496 (local address 127.20.49.126:42449)
1209 08:58:53.157476 (+ 176us) server_negotiation.cc:207] Beginning negotiation
1209 08:58:53.157484 (+ 8us) server_negotiation.cc:400] Waiting for connection header
1209 08:58:53.157771 (+ 287us) server_negotiation.cc:408] Connection header received
1209 08:58:53.157923 (+ 152us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1209 08:58:53.157926 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1209 08:58:53.157975 (+ 49us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1209 08:58:53.158046 (+ 71us) server_negotiation.cc:227] Negotiated authn=SASL
1209 08:58:53.159047 (+ 1001us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:58:53.159534 (+ 487us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1209 08:58:53.160289 (+ 755us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:58:53.160465 (+ 176us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:58:53.162657 (+ 2192us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1209 08:58:53.162680 (+ 23us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1209 08:58:53.162685 (+ 5us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1209 08:58:53.162724 (+ 39us) server_negotiation.cc:923] calling sasl_server_start()
1209 08:58:53.164467 (+ 1743us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1209 08:58:53.164898 (+ 431us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1209 08:58:53.164904 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1209 08:58:53.164909 (+ 5us) server_negotiation.cc:968] calling sasl_server_step()
1209 08:58:53.164965 (+ 56us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1209 08:58:53.165224 (+ 259us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1209 08:58:53.165227 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1209 08:58:53.165229 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1209 08:58:53.165412 (+ 183us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1209 08:58:53.165508 (+ 96us) server_negotiation.cc:1036] Waiting for connection context
1209 08:58:53.165944 (+ 436us) server_negotiation.cc:300] Negotiation successful
1209 08:58:53.166048 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20251209 08:58:53.166652 28159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:58:53.153810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56752 (local address 127.20.49.124:44451)
1209 08:58:53.154250 (+ 440us) server_negotiation.cc:207] Beginning negotiation
1209 08:58:53.154262 (+ 12us) server_negotiation.cc:400] Waiting for connection header
1209 08:58:53.154306 (+ 44us) server_negotiation.cc:408] Connection header received
1209 08:58:53.154996 (+ 690us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1209 08:58:53.155023 (+ 27us) server_negotiation.cc:462] Received NEGOTIATE request from client
1209 08:58:53.155404 (+ 381us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1209 08:58:53.155780 (+ 376us) server_negotiation.cc:227] Negotiated authn=SASL
1209 08:58:53.156615 (+ 835us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:58:53.157600 (+ 985us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1209 08:58:53.158306 (+ 706us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:58:53.158614 (+ 308us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:58:53.161201 (+ 2587us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1209 08:58:53.161221 (+ 20us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1209 08:58:53.161231 (+ 10us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1209 08:58:53.161255 (+ 24us) server_negotiation.cc:923] calling sasl_server_start()
1209 08:58:53.163506 (+ 2251us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1209 08:58:53.164027 (+ 521us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1209 08:58:53.164033 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1209 08:58:53.164041 (+ 8us) server_negotiation.cc:968] calling sasl_server_step()
1209 08:58:53.164139 (+ 98us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1209 08:58:53.164422 (+ 283us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1209 08:58:53.164426 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1209 08:58:53.164429 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
1209 08:58:53.164865 (+ 436us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1209 08:58:53.165059 (+ 194us) server_negotiation.cc:1036] Waiting for connection context
1209 08:58:53.165394 (+ 335us) server_negotiation.cc:300] Negotiation successful
1209 08:58:53.165765 (+ 371us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":278,"thread_start_us":120,"threads_started":1}
I20251209 08:58:53.166654 28160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:58:53.157072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.49.126:42449 (local address 127.0.0.1:58496)
1209 08:58:53.157622 (+ 550us) negotiation.cc:107] Waiting for socket to connect
1209 08:58:53.157662 (+ 40us) client_negotiation.cc:174] Beginning negotiation
1209 08:58:53.157820 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1209 08:58:53.158114 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1209 08:58:53.158121 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
1209 08:58:53.158569 (+ 448us) client_negotiation.cc:189] Negotiated authn=SASL
1209 08:58:53.158918 (+ 349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:58:53.158927 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:58:53.159638 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1209 08:58:53.159641 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1209 08:58:53.160098 (+ 457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:58:53.160109 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:58:53.160283 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:58:53.160951 (+ 668us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1209 08:58:53.160994 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
1209 08:58:53.162510 (+ 1516us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1209 08:58:53.164571 (+ 2061us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1209 08:58:53.164577 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1209 08:58:53.164593 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
1209 08:58:53.164795 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1209 08:58:53.165068 (+ 273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1209 08:58:53.165071 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1209 08:58:53.165073 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1209 08:58:53.165131 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1209 08:58:53.165563 (+ 432us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1209 08:58:53.165570 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1209 08:58:53.165725 (+ 155us) client_negotiation.cc:769] Sending connection context
1209 08:58:53.165880 (+ 155us) client_negotiation.cc:240] Negotiation successful
1209 08:58:53.166026 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":401,"thread_start_us":115,"threads_started":1}
I20251209 08:58:53.167943 28157 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.49.125" port: 44905 } has no permanent_uuid. Determining permanent_uuid...
Dec 09 08:58:53 dist-test-slave-8w6c krb5kdc[27379](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765270733, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.49.124@KRBTEST.COM for oryx/127.20.49.125@KRBTEST.COM
I20251209 08:58:53.176579 28160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:58:53.168455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.49.125:44905 (local address 127.0.0.1:36898)
1209 08:58:53.168887 (+ 432us) negotiation.cc:107] Waiting for socket to connect
1209 08:58:53.168901 (+ 14us) client_negotiation.cc:174] Beginning negotiation
1209 08:58:53.168977 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1209 08:58:53.169255 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1209 08:58:53.169258 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1209 08:58:53.169456 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
1209 08:58:53.169647 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:58:53.169653 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:58:53.170764 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1209 08:58:53.170767 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1209 08:58:53.171165 (+ 398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:58:53.171172 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:58:53.171266 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:58:53.171831 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1209 08:58:53.171844 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
1209 08:58:53.173157 (+ 1313us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1209 08:58:53.175221 (+ 2064us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1209 08:58:53.175224 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1209 08:58:53.175226 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1209 08:58:53.175402 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1209 08:58:53.175666 (+ 264us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1209 08:58:53.175669 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1209 08:58:53.175671 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1209 08:58:53.175725 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1209 08:58:53.176095 (+ 370us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1209 08:58:53.176098 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1209 08:58:53.176188 (+ 90us) client_negotiation.cc:769] Sending connection context
1209 08:58:53.176290 (+ 102us) client_negotiation.cc:240] Negotiation successful
1209 08:58:53.176418 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":351}
I20251209 08:58:53.176589 28161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:58:53.168635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36898 (local address 127.20.49.125:44905)
1209 08:58:53.168931 (+ 296us) server_negotiation.cc:207] Beginning negotiation
1209 08:58:53.168936 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1209 08:58:53.168949 (+ 13us) server_negotiation.cc:408] Connection header received
1209 08:58:53.169061 (+ 112us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1209 08:58:53.169064 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1209 08:58:53.169109 (+ 45us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1209 08:58:53.169190 (+ 81us) server_negotiation.cc:227] Negotiated authn=SASL
1209 08:58:53.169816 (+ 626us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:58:53.170643 (+ 827us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1209 08:58:53.171281 (+ 638us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:58:53.171435 (+ 154us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:58:53.173285 (+ 1850us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1209 08:58:53.173306 (+ 21us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1209 08:58:53.173309 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1209 08:58:53.173336 (+ 27us) server_negotiation.cc:923] calling sasl_server_start()
1209 08:58:53.175092 (+ 1756us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1209 08:58:53.175503 (+ 411us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1209 08:58:53.175506 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1209 08:58:53.175507 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1209 08:58:53.175556 (+ 49us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1209 08:58:53.175821 (+ 265us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1209 08:58:53.175823 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1209 08:58:53.175824 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1209 08:58:53.175980 (+ 156us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1209 08:58:53.176060 (+ 80us) server_negotiation.cc:1036] Waiting for connection context
1209 08:58:53.176342 (+ 282us) server_negotiation.cc:300] Negotiation successful
1209 08:58:53.176462 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":215,"thread_start_us":96,"threads_started":1}
I20251209 08:58:53.177417 28157 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.49.124" port: 44451 } has no permanent_uuid. Determining permanent_uuid...
Dec 09 08:58:53 dist-test-slave-8w6c krb5kdc[27379](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765270733, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.49.124@KRBTEST.COM for oryx/127.20.49.124@KRBTEST.COM
I20251209 08:58:53.184598 28160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:58:53.178017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.49.124:44451 (local address 127.0.0.1:56760)
1209 08:58:53.178136 (+ 119us) negotiation.cc:107] Waiting for socket to connect
1209 08:58:53.178151 (+ 15us) client_negotiation.cc:174] Beginning negotiation
1209 08:58:53.178232 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1209 08:58:53.178437 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1209 08:58:53.178440 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1209 08:58:53.178577 (+ 137us) client_negotiation.cc:189] Negotiated authn=SASL
1209 08:58:53.178737 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:58:53.178742 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:58:53.179396 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1209 08:58:53.179399 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1209 08:58:53.179763 (+ 364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:58:53.179771 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:58:53.179864 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:58:53.180364 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1209 08:58:53.180376 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
1209 08:58:53.181586 (+ 1210us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1209 08:58:53.183392 (+ 1806us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1209 08:58:53.183395 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1209 08:58:53.183398 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
1209 08:58:53.183554 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1209 08:58:53.183762 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1209 08:58:53.183765 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1209 08:58:53.183767 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1209 08:58:53.183804 (+ 37us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1209 08:58:53.184093 (+ 289us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1209 08:58:53.184096 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1209 08:58:53.184164 (+ 68us) client_negotiation.cc:769] Sending connection context
1209 08:58:53.184251 (+ 87us) client_negotiation.cc:240] Negotiation successful
1209 08:58:53.184394 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20251209 08:58:53.184602 28159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:58:53.178034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56760 (local address 127.20.49.124:44451)
1209 08:58:53.178143 (+ 109us) server_negotiation.cc:207] Beginning negotiation
1209 08:58:53.178146 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1209 08:58:53.178202 (+ 56us) server_negotiation.cc:408] Connection header received
1209 08:58:53.178304 (+ 102us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1209 08:58:53.178306 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1209 08:58:53.178342 (+ 36us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1209 08:58:53.178407 (+ 65us) server_negotiation.cc:227] Negotiated authn=SASL
1209 08:58:53.178819 (+ 412us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:58:53.179272 (+ 453us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1209 08:58:53.179857 (+ 585us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:58:53.179998 (+ 141us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:58:53.181742 (+ 1744us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1209 08:58:53.181759 (+ 17us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1209 08:58:53.181761 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1209 08:58:53.181782 (+ 21us) server_negotiation.cc:923] calling sasl_server_start()
1209 08:58:53.183254 (+ 1472us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1209 08:58:53.183649 (+ 395us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1209 08:58:53.183652 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1209 08:58:53.183653 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1209 08:58:53.183688 (+ 35us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1209 08:58:53.183874 (+ 186us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1209 08:58:53.183878 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1209 08:58:53.183879 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1209 08:58:53.184014 (+ 135us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1209 08:58:53.184098 (+ 84us) server_negotiation.cc:1036] Waiting for connection context
1209 08:58:53.184295 (+ 197us) server_negotiation.cc:300] Negotiation successful
1209 08:58:53.184464 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":67,"server-negotiator.queue_time_us":40}
I20251209 08:58:53.187604 28157 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0364ed40f171452f9fa244fa50f6a6c3: Bootstrap starting.
I20251209 08:58:53.190315 28157 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0364ed40f171452f9fa244fa50f6a6c3: Neither blocks nor log segments found. Creating new log.
I20251209 08:58:53.191085 28157 log.cc:826] T 00000000000000000000000000000000 P 0364ed40f171452f9fa244fa50f6a6c3: Log is configured to *not* fsync() on all Append() calls
I20251209 08:58:53.193109 28157 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0364ed40f171452f9fa244fa50f6a6c3: No bootstrap required, opened a new log
I20251209 08:58:53.195842 28157 raft_consensus.cc:359] T 00000000000000000000000000000000 P 0364ed40f171452f9fa244fa50f6a6c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0a0d121c6bb474dac7c98dc7ec52d14" member_type: VOTER last_known_addr { host: "127.20.49.126" port: 42449 } } peers { permanent_uuid: "798ad87d8a5a4392815120c18c40bedf" member_type: VOTER last_known_addr { host: "127.20.49.125" port: 44905 } } peers { permanent_uuid: "0364ed40f171452f9fa244fa50f6a6c3" member_type: VOTER last_known_addr { host: "127.20.49.124" port: 44451 } }
I20251209 08:58:53.196027 28157 raft_consensus.cc:385] T 00000000000000000000000000000000 P 0364ed40f171452f9fa244fa50f6a6c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251209 08:58:53.196069 28157 raft_consensus.cc:740] T 00000000000000000000000000000000 P 0364ed40f171452f9fa244fa50f6a6c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0364ed40f171452f9fa244fa50f6a6c3, State: Initialized, Role: FOLLOWER
I20251209 08:58:53.196508 28157 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0364ed40f171452f9fa244fa50f6a6c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0a0d121c6bb474dac7c98dc7ec52d14" member_type: VOTER last_known_addr { host: "127.20.49.126" port: 42449 } } peers { permanent_uuid: "798ad87d8a5a4392815120c18c40bedf" member_type: VOTER last_known_addr { host: "127.20.49.125" port: 44905 } } peers { permanent_uuid: "0364ed40f171452f9fa244fa50f6a6c3" member_type: VOTER last_known_addr { host: "127.20.49.124" port: 44451 } }
I20251209 08:58:53.196990 28163 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0364ed40f171452f9fa244fa50f6a6c3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0a0d121c6bb474dac7c98dc7ec52d14" member_type: VOTER last_known_addr { host: "127.20.49.126" port: 42449 } } peers { permanent_uuid: "798ad87d8a5a4392815120c18c40bedf" member_type: VOTER last_known_addr { host: "127.20.49.125" port: 44905 } } peers { permanent_uuid: "0364ed40f171452f9fa244fa50f6a6c3" member_type: VOTER last_known_addr { host: "127.20.49.124" port: 44451 } } }
I20251209 08:58:53.197108 28163 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0364ed40f171452f9fa244fa50f6a6c3 [sys.catalog]: This master's current role is: FOLLOWER
I20251209 08:58:53.197655 28157 sys_catalog.cc:565] T 00000000000000000000000000000000 P 0364ed40f171452f9fa244fa50f6a6c3 [sys.catalog]: configured and running, proceeding with master startup.
W20251209 08:58:53.202759 28174 catalog_manager.cc:1568] T 00000000000000000000000000000000 P 0364ed40f171452f9fa244fa50f6a6c3: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20251209 08:58:53.202862 28174 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20251209 08:58:53.206820 28020 master_runner.cc:428] Detected that this master 0364ed40f171452f9fa244fa50f6a6c3 is joining an existing cluster
I20251209 08:58:53.206894 28020 master_runner.cc:432] Initiating AddMaster RPC to add 127.20.49.124:44451
I20251209 08:58:53.224489 27992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:58:53.221211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.49.126:42449 (local address 127.0.0.1:58508)
1209 08:58:53.221372 (+ 161us) negotiation.cc:107] Waiting for socket to connect
1209 08:58:53.221387 (+ 15us) client_negotiation.cc:174] Beginning negotiation
1209 08:58:53.221614 (+ 227us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1209 08:58:53.221976 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1209 08:58:53.221979 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1209 08:58:53.221999 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1209 08:58:53.222203 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:58:53.222209 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:58:53.223312 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1209 08:58:53.223315 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1209 08:58:53.223931 (+ 616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:58:53.223938 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:58:53.224080 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:58:53.224160 (+ 80us) client_negotiation.cc:769] Sending connection context
1209 08:58:53.224220 (+ 60us) client_negotiation.cc:240] Negotiation successful
1209 08:58:53.224294 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":67}
I20251209 08:58:53.224790 27466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:58:53.221380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58508 (local address 127.20.49.126:42449)
1209 08:58:53.221503 (+ 123us) server_negotiation.cc:207] Beginning negotiation
1209 08:58:53.221508 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1209 08:58:53.221524 (+ 16us) server_negotiation.cc:408] Connection header received
1209 08:58:53.221800 (+ 276us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1209 08:58:53.221804 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1209 08:58:53.221855 (+ 51us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1209 08:58:53.221936 (+ 81us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1209 08:58:53.222355 (+ 419us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:58:53.223170 (+ 815us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1209 08:58:53.224075 (+ 905us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:58:53.224499 (+ 424us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:58:53.224591 (+ 92us) server_negotiation.cc:1036] Waiting for connection context
1209 08:58:53.224642 (+ 51us) server_negotiation.cc:300] Negotiation successful
1209 08:58:53.224702 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20251209 08:58:53.228607 27992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:58:53.225990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.49.125:44905 (local address 127.0.0.1:36928)
1209 08:58:53.226130 (+ 140us) negotiation.cc:107] Waiting for socket to connect
1209 08:58:53.226143 (+ 13us) client_negotiation.cc:174] Beginning negotiation
1209 08:58:53.226205 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1209 08:58:53.226662 (+ 457us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1209 08:58:53.226666 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
1209 08:58:53.226684 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1209 08:58:53.226848 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:58:53.226853 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:58:53.227572 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1209 08:58:53.227585 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1209 08:58:53.228215 (+ 630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:58:53.228224 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:58:53.228335 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:58:53.228355 (+ 20us) client_negotiation.cc:769] Sending connection context
1209 08:58:53.228408 (+ 53us) client_negotiation.cc:240] Negotiation successful
1209 08:58:53.228469 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53}
I20251209 08:58:53.229213 28182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:58:53.226123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36928 (local address 127.20.49.125:44905)
1209 08:58:53.226409 (+ 286us) server_negotiation.cc:207] Beginning negotiation
1209 08:58:53.226412 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1209 08:58:53.226424 (+ 12us) server_negotiation.cc:408] Connection header received
1209 08:58:53.226467 (+ 43us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1209 08:58:53.226469 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1209 08:58:53.226519 (+ 50us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1209 08:58:53.226617 (+ 98us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1209 08:58:53.226960 (+ 343us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:58:53.227466 (+ 506us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1209 08:58:53.228345 (+ 879us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:58:53.228899 (+ 554us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:58:53.229010 (+ 111us) server_negotiation.cc:1036] Waiting for connection context
1209 08:58:53.229056 (+ 46us) server_negotiation.cc:300] Negotiation successful
1209 08:58:53.229115 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":215,"thread_start_us":112,"threads_started":1}
I20251209 08:58:53.239784 27466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:58:53.229970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58524 (local address 127.20.49.126:42449)
1209 08:58:53.230127 (+ 157us) server_negotiation.cc:207] Beginning negotiation
1209 08:58:53.230133 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1209 08:58:53.231164 (+ 1031us) server_negotiation.cc:408] Connection header received
1209 08:58:53.231269 (+ 105us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1209 08:58:53.231272 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1209 08:58:53.231322 (+ 50us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1209 08:58:53.231414 (+ 92us) server_negotiation.cc:227] Negotiated authn=SASL
1209 08:58:53.232928 (+ 1514us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:58:53.233729 (+ 801us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1209 08:58:53.234586 (+ 857us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:58:53.234787 (+ 201us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:58:53.235963 (+ 1176us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1209 08:58:53.235980 (+ 17us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1209 08:58:53.235984 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1209 08:58:53.236009 (+ 25us) server_negotiation.cc:923] calling sasl_server_start()
1209 08:58:53.238120 (+ 2111us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1209 08:58:53.238609 (+ 489us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1209 08:58:53.238613 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1209 08:58:53.238615 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1209 08:58:53.238662 (+ 47us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1209 08:58:53.238916 (+ 254us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1209 08:58:53.238919 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1209 08:58:53.238921 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1209 08:58:53.239094 (+ 173us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1209 08:58:53.239161 (+ 67us) server_negotiation.cc:1036] Waiting for connection context
1209 08:58:53.239486 (+ 325us) server_negotiation.cc:300] Negotiation successful
1209 08:58:53.239619 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20251209 08:58:53.240008 28161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:58:53.209176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36912 (local address 127.20.49.125:44905)
1209 08:58:53.209338 (+ 162us) server_negotiation.cc:207] Beginning negotiation
1209 08:58:53.209343 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1209 08:58:53.209743 (+ 400us) server_negotiation.cc:408] Connection header received
1209 08:58:53.209959 (+ 216us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1209 08:58:53.209963 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1209 08:58:53.210021 (+ 58us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1209 08:58:53.210103 (+ 82us) server_negotiation.cc:227] Negotiated authn=SASL
1209 08:58:53.231592 (+ 21489us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:58:53.232098 (+ 506us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1209 08:58:53.232928 (+ 830us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:58:53.233116 (+ 188us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:58:53.234566 (+ 1450us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1209 08:58:53.234588 (+ 22us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1209 08:58:53.234591 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1209 08:58:53.234619 (+ 28us) server_negotiation.cc:923] calling sasl_server_start()
1209 08:58:53.237943 (+ 3324us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1209 08:58:53.238451 (+ 508us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1209 08:58:53.238455 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1209 08:58:53.238457 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1209 08:58:53.238516 (+ 59us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1209 08:58:53.239070 (+ 554us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1209 08:58:53.239079 (+ 9us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1209 08:58:53.239081 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1209 08:58:53.239270 (+ 189us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1209 08:58:53.239366 (+ 96us) server_negotiation.cc:1036] Waiting for connection context
1209 08:58:53.239696 (+ 330us) server_negotiation.cc:300] Negotiation successful
1209 08:58:53.239829 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20251209 08:58:53.241207 28180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:58:53.209120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.49.125:44905 (local address 127.0.0.1:36912)
1209 08:58:53.209580 (+ 460us) negotiation.cc:107] Waiting for socket to connect
1209 08:58:53.209597 (+ 17us) client_negotiation.cc:174] Beginning negotiation
1209 08:58:53.209745 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1209 08:58:53.210255 (+ 510us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1209 08:58:53.210258 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1209 08:58:53.229943 (+ 19685us) client_negotiation.cc:189] Negotiated authn=SASL
1209 08:58:53.231445 (+ 1502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:58:53.231451 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:58:53.232242 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1209 08:58:53.232247 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1209 08:58:53.232792 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:58:53.232804 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:58:53.233251 (+ 447us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:58:53.233951 (+ 700us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1209 08:58:53.233968 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
1209 08:58:53.234423 (+ 455us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1209 08:58:53.238067 (+ 3644us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1209 08:58:53.238071 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1209 08:58:53.238073 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1209 08:58:53.238332 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1209 08:58:53.238636 (+ 304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1209 08:58:53.238641 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1209 08:58:53.238645 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
1209 08:58:53.238701 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1209 08:58:53.239424 (+ 723us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1209 08:58:53.239427 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1209 08:58:53.239510 (+ 83us) client_negotiation.cc:769] Sending connection context
1209 08:58:53.240202 (+ 692us) client_negotiation.cc:240] Negotiation successful
1209 08:58:53.240961 (+ 759us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":374,"spinlock_wait_cycles":56576,"thread_start_us":277,"threads_started":1}
I20251209 08:58:53.241819 28183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:58:53.230243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.49.126:42449 (local address 127.0.0.1:58524)
1209 08:58:53.231063 (+ 820us) negotiation.cc:107] Waiting for socket to connect
1209 08:58:53.231078 (+ 15us) client_negotiation.cc:174] Beginning negotiation
1209 08:58:53.231181 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1209 08:58:53.232270 (+ 1089us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1209 08:58:53.232273 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1209 08:58:53.232506 (+ 233us) client_negotiation.cc:189] Negotiated authn=SASL
1209 08:58:53.232784 (+ 278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:58:53.232792 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:58:53.233868 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1209 08:58:53.233872 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1209 08:58:53.234454 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:58:53.234463 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:58:53.234900 (+ 437us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:58:53.235424 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1209 08:58:53.235440 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
1209 08:58:53.235821 (+ 381us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1209 08:58:53.238246 (+ 2425us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1209 08:58:53.238250 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1209 08:58:53.238253 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
1209 08:58:53.238487 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1209 08:58:53.238767 (+ 280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1209 08:58:53.238771 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1209 08:58:53.238773 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1209 08:58:53.238823 (+ 50us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1209 08:58:53.239230 (+ 407us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1209 08:58:53.239233 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1209 08:58:53.239314 (+ 81us) client_negotiation.cc:769] Sending connection context
1209 08:58:53.240485 (+ 1171us) client_negotiation.cc:240] Negotiation successful
1209 08:58:53.241072 (+ 587us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":647,"mutex_wait_us":424,"thread_start_us":167,"threads_started":1}
I20251209 08:58:53.243104 28181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:58:53.209143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.49.124:44451 (local address 127.0.0.1:56766)
1209 08:58:53.209717 (+ 574us) negotiation.cc:107] Waiting for socket to connect
1209 08:58:53.209729 (+ 12us) client_negotiation.cc:174] Beginning negotiation
1209 08:58:53.209898 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1209 08:58:53.210106 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1209 08:58:53.210109 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1209 08:58:53.231035 (+ 20926us) client_negotiation.cc:189] Negotiated authn=SASL
1209 08:58:53.231320 (+ 285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:58:53.231331 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:58:53.234933 (+ 3602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1209 08:58:53.234938 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1209 08:58:53.235559 (+ 621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:58:53.235572 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:58:53.236054 (+ 482us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:58:53.236682 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1209 08:58:53.236704 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
1209 08:58:53.237141 (+ 437us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1209 08:58:53.240929 (+ 3788us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1209 08:58:53.240933 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1209 08:58:53.240935 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1209 08:58:53.241180 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1209 08:58:53.241427 (+ 247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1209 08:58:53.241429 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1209 08:58:53.241430 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1209 08:58:53.241463 (+ 33us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1209 08:58:53.242603 (+ 1140us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1209 08:58:53.242606 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1209 08:58:53.242684 (+ 78us) client_negotiation.cc:769] Sending connection context
1209 08:58:53.242788 (+ 104us) client_negotiation.cc:240] Negotiation successful
1209 08:58:53.242908 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":488,"spinlock_wait_cycles":6528,"thread_start_us":309,"threads_started":1}
I20251209 08:58:53.243141 28159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:58:53.209253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56766 (local address 127.20.49.124:44451)
1209 08:58:53.209373 (+ 120us) server_negotiation.cc:207] Beginning negotiation
1209 08:58:53.209377 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1209 08:58:53.209794 (+ 417us) server_negotiation.cc:408] Connection header received
1209 08:58:53.209979 (+ 185us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1209 08:58:53.209982 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1209 08:58:53.210020 (+ 38us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1209 08:58:53.210080 (+ 60us) server_negotiation.cc:227] Negotiated authn=SASL
1209 08:58:53.231425 (+ 21345us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:58:53.232138 (+ 713us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1209 08:58:53.235713 (+ 3575us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:58:53.235941 (+ 228us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:58:53.237265 (+ 1324us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1209 08:58:53.237283 (+ 18us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1209 08:58:53.237285 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1209 08:58:53.237311 (+ 26us) server_negotiation.cc:923] calling sasl_server_start()
1209 08:58:53.240824 (+ 3513us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1209 08:58:53.241288 (+ 464us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1209 08:58:53.241292 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1209 08:58:53.241293 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1209 08:58:53.241336 (+ 43us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1209 08:58:53.241551 (+ 215us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1209 08:58:53.241554 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1209 08:58:53.241555 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1209 08:58:53.241707 (+ 152us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1209 08:58:53.241809 (+ 102us) server_negotiation.cc:1036] Waiting for connection context
1209 08:58:53.242874 (+ 1065us) server_negotiation.cc:300] Negotiation successful
1209 08:58:53.242986 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
Dec 09 08:58:53 dist-test-slave-8w6c krb5kdc[27379](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765270731, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.49.126@KRBTEST.COM for oryx/127.20.49.124@KRBTEST.COM
I20251209 08:58:53.253603 27992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:58:53.245865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.49.124:44451 (local address 127.0.0.1:56768)
1209 08:58:53.246024 (+ 159us) negotiation.cc:107] Waiting for socket to connect
1209 08:58:53.246038 (+ 14us) client_negotiation.cc:174] Beginning negotiation
1209 08:58:53.246109 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1209 08:58:53.246404 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1209 08:58:53.246408 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
1209 08:58:53.246590 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
1209 08:58:53.246764 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:58:53.246770 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:58:53.247556 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1209 08:58:53.247561 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1209 08:58:53.247915 (+ 354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:58:53.247921 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:58:53.248029 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:58:53.248611 (+ 582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1209 08:58:53.248625 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
1209 08:58:53.250096 (+ 1471us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1209 08:58:53.252189 (+ 2093us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1209 08:58:53.252193 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1209 08:58:53.252194 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1209 08:58:53.252392 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1209 08:58:53.252686 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1209 08:58:53.252690 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1209 08:58:53.252691 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1209 08:58:53.252748 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1209 08:58:53.253118 (+ 370us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1209 08:58:53.253120 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1209 08:58:53.253209 (+ 89us) client_negotiation.cc:769] Sending connection context
1209 08:58:53.253301 (+ 92us) client_negotiation.cc:240] Negotiation successful
1209 08:58:53.253439 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":76}
I20251209 08:58:53.253647 28159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:58:53.245934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56768 (local address 127.20.49.124:44451)
1209 08:58:53.246050 (+ 116us) server_negotiation.cc:207] Beginning negotiation
1209 08:58:53.246055 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1209 08:58:53.246072 (+ 17us) server_negotiation.cc:408] Connection header received
1209 08:58:53.246256 (+ 184us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1209 08:58:53.246258 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1209 08:58:53.246302 (+ 44us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1209 08:58:53.246372 (+ 70us) server_negotiation.cc:227] Negotiated authn=SASL
1209 08:58:53.246885 (+ 513us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:58:53.247461 (+ 576us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1209 08:58:53.248021 (+ 560us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:58:53.248226 (+ 205us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:58:53.250223 (+ 1997us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1209 08:58:53.250243 (+ 20us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1209 08:58:53.250247 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1209 08:58:53.250274 (+ 27us) server_negotiation.cc:923] calling sasl_server_start()
1209 08:58:53.252053 (+ 1779us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1209 08:58:53.252493 (+ 440us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1209 08:58:53.252498 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1209 08:58:53.252500 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1209 08:58:53.252550 (+ 50us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1209 08:58:53.252839 (+ 289us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1209 08:58:53.252842 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1209 08:58:53.252844 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1209 08:58:53.253029 (+ 185us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1209 08:58:53.253125 (+ 96us) server_negotiation.cc:1036] Waiting for connection context
1209 08:58:53.253360 (+ 235us) server_negotiation.cc:300] Negotiation successful
1209 08:58:53.253483 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20251209 08:58:53.254755 27425 catalog_manager.cc:7141] Initiating ChangeConfig request to add master 127.20.49.124:44451: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "0364ed40f171452f9fa244fa50f6a6c3"
member_type: NON_VOTER
last_known_addr {
host: "127.20.49.124"
port: 44451
}
attrs {
promote: true
}
}
dest_uuid: "c0a0d121c6bb474dac7c98dc7ec52d14"
cas_config_opid_index: 5
I20251209 08:58:53.255326 27425 consensus_queue.cc:237] T 00000000000000000000000000000000 P c0a0d121c6bb474dac7c98dc7ec52d14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "c0a0d121c6bb474dac7c98dc7ec52d14" member_type: VOTER last_known_addr { host: "127.20.49.126" port: 42449 } } peers { permanent_uuid: "798ad87d8a5a4392815120c18c40bedf" member_type: NON_VOTER last_known_addr { host: "127.20.49.125" port: 44905 } attrs { promote: true } } peers { permanent_uuid: "0364ed40f171452f9fa244fa50f6a6c3" member_type: NON_VOTER last_known_addr { host: "127.20.49.124" port: 44451 } attrs { promote: true } }
I20251209 08:58:53.256446 27996 raft_consensus.cc:2955] T 00000000000000000000000000000000 P c0a0d121c6bb474dac7c98dc7ec52d14 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 0364ed40f171452f9fa244fa50f6a6c3 (127.20.49.124) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "c0a0d121c6bb474dac7c98dc7ec52d14" member_type: VOTER last_known_addr { host: "127.20.49.126" port: 42449 } } peers { permanent_uuid: "798ad87d8a5a4392815120c18c40bedf" member_type: NON_VOTER last_known_addr { host: "127.20.49.125" port: 44905 } attrs { promote: true } } peers { permanent_uuid: "0364ed40f171452f9fa244fa50f6a6c3" member_type: NON_VOTER last_known_addr { host: "127.20.49.124" port: 44451 } attrs { promote: true } } }
I20251209 08:58:53.256873 28049 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 798ad87d8a5a4392815120c18c40bedf [term 1 LEARNER]: Refusing update from remote peer c0a0d121c6bb474dac7c98dc7ec52d14: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20251209 08:58:53.257875 27996 catalog_manager.cc:7145] Successfully completed master ChangeConfig request to add master 127.20.49.124:44451
I20251209 08:58:53.257973 28184 consensus_queue.cc:1048] T 00000000000000000000000000000000 P c0a0d121c6bb474dac7c98dc7ec52d14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "798ad87d8a5a4392815120c18c40bedf" member_type: NON_VOTER last_known_addr { host: "127.20.49.125" port: 44905 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20251209 08:58:53.258064 27996 sys_catalog.cc:455] T 00000000000000000000000000000000 P c0a0d121c6bb474dac7c98dc7ec52d14 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "c0a0d121c6bb474dac7c98dc7ec52d14" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "c0a0d121c6bb474dac7c98dc7ec52d14" member_type: VOTER last_known_addr { host: "127.20.49.126" port: 42449 } } peers { permanent_uuid: "798ad87d8a5a4392815120c18c40bedf" member_type: NON_VOTER last_known_addr { host: "127.20.49.125" port: 44905 } attrs { promote: true } } peers { permanent_uuid: "0364ed40f171452f9fa244fa50f6a6c3" member_type: NON_VOTER last_known_addr { host: "127.20.49.124" port: 44451 } attrs { promote: true } } }
I20251209 08:58:53.258167 27996 sys_catalog.cc:458] T 00000000000000000000000000000000 P c0a0d121c6bb474dac7c98dc7ec52d14 [sys.catalog]: This master's current role is: LEADER
I20251209 08:58:53.258698 28131 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 0364ed40f171452f9fa244fa50f6a6c3 [term 0 FOLLOWER]: Advancing to term 1
I20251209 08:58:53.260107 28049 raft_consensus.cc:2955] T 00000000000000000000000000000000 P 798ad87d8a5a4392815120c18c40bedf [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 0364ed40f171452f9fa244fa50f6a6c3 (127.20.49.124) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "c0a0d121c6bb474dac7c98dc7ec52d14" member_type: VOTER last_known_addr { host: "127.20.49.126" port: 42449 } } peers { permanent_uuid: "798ad87d8a5a4392815120c18c40bedf" member_type: NON_VOTER last_known_addr { host: "127.20.49.125" port: 44905 } attrs { promote: true } } peers { permanent_uuid: "0364ed40f171452f9fa244fa50f6a6c3" member_type: NON_VOTER last_known_addr { host: "127.20.49.124" port: 44451 } attrs { promote: true } } }
I20251209 08:58:53.260700 28131 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 0364ed40f171452f9fa244fa50f6a6c3 [term 1 FOLLOWER]: Refusing update from remote peer c0a0d121c6bb474dac7c98dc7ec52d14: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20251209 08:58:53.261341 28020 master.cc:561] Master@127.20.49.124:44451 shutting down...
I20251209 08:58:53.262678 28078 sys_catalog.cc:455] T 00000000000000000000000000000000 P 798ad87d8a5a4392815120c18c40bedf [sys.catalog]: SysCatalogTable state changed. Reason: New leader c0a0d121c6bb474dac7c98dc7ec52d14. Latest consensus state: current_term: 1 leader_uuid: "c0a0d121c6bb474dac7c98dc7ec52d14" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "c0a0d121c6bb474dac7c98dc7ec52d14" member_type: VOTER last_known_addr { host: "127.20.49.126" port: 42449 } } peers { permanent_uuid: "798ad87d8a5a4392815120c18c40bedf" member_type: NON_VOTER last_known_addr { host: "127.20.49.125" port: 44905 } attrs { promote: true } } peers { permanent_uuid: "0364ed40f171452f9fa244fa50f6a6c3" member_type: NON_VOTER last_known_addr { host: "127.20.49.124" port: 44451 } attrs { promote: true } } }
I20251209 08:58:53.262810 28078 sys_catalog.cc:458] T 00000000000000000000000000000000 P 798ad87d8a5a4392815120c18c40bedf [sys.catalog]: This master's current role is: LEARNER
I20251209 08:58:53.262914 28078 sys_catalog.cc:455] T 00000000000000000000000000000000 P 798ad87d8a5a4392815120c18c40bedf [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c0a0d121c6bb474dac7c98dc7ec52d14" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "c0a0d121c6bb474dac7c98dc7ec52d14" member_type: VOTER last_known_addr { host: "127.20.49.126" port: 42449 } } peers { permanent_uuid: "798ad87d8a5a4392815120c18c40bedf" member_type: NON_VOTER last_known_addr { host: "127.20.49.125" port: 44905 } attrs { promote: true } } peers { permanent_uuid: "0364ed40f171452f9fa244fa50f6a6c3" member_type: NON_VOTER last_known_addr { host: "127.20.49.124" port: 44451 } attrs { promote: true } } }
I20251209 08:58:53.262988 28078 sys_catalog.cc:458] T 00000000000000000000000000000000 P 798ad87d8a5a4392815120c18c40bedf [sys.catalog]: This master's current role is: LEARNER
I20251209 08:58:53.263363 28184 consensus_queue.cc:1048] T 00000000000000000000000000000000 P c0a0d121c6bb474dac7c98dc7ec52d14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0364ed40f171452f9fa244fa50f6a6c3" member_type: NON_VOTER last_known_addr { host: "127.20.49.124" port: 44451 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
W20251209 08:58:53.264886 27425 master.cc:679] Remote error: unable to get registration information for peer 0364ed40f171452f9fa244fa50f6a6c3: Service unavailable: service kudu.master.MasterService not registered on Master
W20251209 08:58:53.265422 27414 consensus_peers.cc:597] T 00000000000000000000000000000000 P c0a0d121c6bb474dac7c98dc7ec52d14 -> Peer 0364ed40f171452f9fa244fa50f6a6c3 (127.20.49.124:44451): Couldn't send request to peer 0364ed40f171452f9fa244fa50f6a6c3. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20251209 08:58:53.265499 28184 raft_consensus.cc:1064] T 00000000000000000000000000000000 P c0a0d121c6bb474dac7c98dc7ec52d14: attempting to promote NON_VOTER 798ad87d8a5a4392815120c18c40bedf to VOTER
I20251209 08:58:53.266146 28184 consensus_queue.cc:237] T 00000000000000000000000000000000 P c0a0d121c6bb474dac7c98dc7ec52d14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c0a0d121c6bb474dac7c98dc7ec52d14" member_type: VOTER last_known_addr { host: "127.20.49.126" port: 42449 } } peers { permanent_uuid: "798ad87d8a5a4392815120c18c40bedf" member_type: VOTER last_known_addr { host: "127.20.49.125" port: 44905 } attrs { promote: false } } peers { permanent_uuid: "0364ed40f171452f9fa244fa50f6a6c3" member_type: NON_VOTER last_known_addr { host: "127.20.49.124" port: 44451 } attrs { promote: true } }
I20251209 08:58:53.267364 28049 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 798ad87d8a5a4392815120c18c40bedf [term 1 LEARNER]: Refusing update from remote peer c0a0d121c6bb474dac7c98dc7ec52d14: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20251209 08:58:53.267836 27997 consensus_queue.cc:1048] T 00000000000000000000000000000000 P c0a0d121c6bb474dac7c98dc7ec52d14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "798ad87d8a5a4392815120c18c40bedf" member_type: VOTER last_known_addr { host: "127.20.49.125" port: 44905 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20251209 08:58:53.267954 27414 consensus_peers.cc:597] T 00000000000000000000000000000000 P c0a0d121c6bb474dac7c98dc7ec52d14 -> Peer 0364ed40f171452f9fa244fa50f6a6c3 (127.20.49.124:44451): Couldn't send request to peer 0364ed40f171452f9fa244fa50f6a6c3. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20251209 08:58:53.268675 28020 raft_consensus.cc:2243] T 00000000000000000000000000000000 P 0364ed40f171452f9fa244fa50f6a6c3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251209 08:58:53.268949 28020 raft_consensus.cc:2272] T 00000000000000000000000000000000 P 0364ed40f171452f9fa244fa50f6a6c3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251209 08:58:53.269065 28020 tablet_replica.cc:333] T 00000000000000000000000000000000 P 0364ed40f171452f9fa244fa50f6a6c3: stopping tablet replica
I20251209 08:58:53.269714 27996 raft_consensus.cc:2955] T 00000000000000000000000000000000 P c0a0d121c6bb474dac7c98dc7ec52d14 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 798ad87d8a5a4392815120c18c40bedf (127.20.49.125) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c0a0d121c6bb474dac7c98dc7ec52d14" member_type: VOTER last_known_addr { host: "127.20.49.126" port: 42449 } } peers { permanent_uuid: "798ad87d8a5a4392815120c18c40bedf" member_type: VOTER last_known_addr { host: "127.20.49.125" port: 44905 } attrs { promote: false } } peers { permanent_uuid: "0364ed40f171452f9fa244fa50f6a6c3" member_type: NON_VOTER last_known_addr { host: "127.20.49.124" port: 44451 } attrs { promote: true } } }
I20251209 08:58:53.270228 28049 raft_consensus.cc:2955] T 00000000000000000000000000000000 P 798ad87d8a5a4392815120c18c40bedf [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 798ad87d8a5a4392815120c18c40bedf (127.20.49.125) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c0a0d121c6bb474dac7c98dc7ec52d14" member_type: VOTER last_known_addr { host: "127.20.49.126" port: 42449 } } peers { permanent_uuid: "798ad87d8a5a4392815120c18c40bedf" member_type: VOTER last_known_addr { host: "127.20.49.125" port: 44905 } attrs { promote: false } } peers { permanent_uuid: "0364ed40f171452f9fa244fa50f6a6c3" member_type: NON_VOTER last_known_addr { host: "127.20.49.124" port: 44451 } attrs { promote: true } } }
I20251209 08:58:53.271126 27996 sys_catalog.cc:455] T 00000000000000000000000000000000 P c0a0d121c6bb474dac7c98dc7ec52d14 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "c0a0d121c6bb474dac7c98dc7ec52d14" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c0a0d121c6bb474dac7c98dc7ec52d14" member_type: VOTER last_known_addr { host: "127.20.49.126" port: 42449 } } peers { permanent_uuid: "798ad87d8a5a4392815120c18c40bedf" member_type: VOTER last_known_addr { host: "127.20.49.125" port: 44905 } attrs { promote: false } } peers { permanent_uuid: "0364ed40f171452f9fa244fa50f6a6c3" member_type: NON_VOTER last_known_addr { host: "127.20.49.124" port: 44451 } attrs { promote: true } } }
I20251209 08:58:53.271219 27996 sys_catalog.cc:458] T 00000000000000000000000000000000 P c0a0d121c6bb474dac7c98dc7ec52d14 [sys.catalog]: This master's current role is: LEADER
I20251209 08:58:53.271256 27997 sys_catalog.cc:455] T 00000000000000000000000000000000 P c0a0d121c6bb474dac7c98dc7ec52d14 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c0a0d121c6bb474dac7c98dc7ec52d14" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c0a0d121c6bb474dac7c98dc7ec52d14" member_type: VOTER last_known_addr { host: "127.20.49.126" port: 42449 } } peers { permanent_uuid: "798ad87d8a5a4392815120c18c40bedf" member_type: VOTER last_known_addr { host: "127.20.49.125" port: 44905 } attrs { promote: false } } peers { permanent_uuid: "0364ed40f171452f9fa244fa50f6a6c3" member_type: NON_VOTER last_known_addr { host: "127.20.49.124" port: 44451 } attrs { promote: true } } }
I20251209 08:58:53.271337 27997 sys_catalog.cc:458] T 00000000000000000000000000000000 P c0a0d121c6bb474dac7c98dc7ec52d14 [sys.catalog]: This master's current role is: LEADER
I20251209 08:58:53.271548 28186 sys_catalog.cc:455] T 00000000000000000000000000000000 P 798ad87d8a5a4392815120c18c40bedf [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c0a0d121c6bb474dac7c98dc7ec52d14" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c0a0d121c6bb474dac7c98dc7ec52d14" member_type: VOTER last_known_addr { host: "127.20.49.126" port: 42449 } } peers { permanent_uuid: "798ad87d8a5a4392815120c18c40bedf" member_type: VOTER last_known_addr { host: "127.20.49.125" port: 44905 } attrs { promote: false } } peers { permanent_uuid: "0364ed40f171452f9fa244fa50f6a6c3" member_type: NON_VOTER last_known_addr { host: "127.20.49.124" port: 44451 } attrs { promote: true } } }
I20251209 08:58:53.271721 28186 sys_catalog.cc:458] T 00000000000000000000000000000000 P 798ad87d8a5a4392815120c18c40bedf [sys.catalog]: This master's current role is: FOLLOWER
I20251209 08:58:53.284235 28020 master.cc:583] Master@127.20.49.124:44451 shutdown complete.
I20251209 08:58:53.284301 28020 master_runner.cc:305] Clearing existing system tablet
I20251209 08:58:53.284533 28020 fs_manager.cc:362] Metadata directory not provided
I20251209 08:58:53.284600 28020 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251209 08:58:53.285881 28020 fs_manager.cc:714] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20251209 08:58:53.286996 28188 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251209 08:58:53.287318 28020 fs_report.cc:389] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765270668420570-20677-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251209 08:58:53.287456 28020 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20251209 08:58:53.287498 28020 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765270668420570-20677-0/minicluster-data/master-2/data,/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765270668420570-20677-0/minicluster-data/master-2/wal
uuid: "0364ed40f171452f9fa244fa50f6a6c3"
format_stamp: "Formatted at 2025-12-09 08:58:53 on dist-test-slave-8w6c"
server_key: "ded07db4bbf58c7410644fc982d50862"
server_key_iv: "51de27466f9232ff51b1b7f797268a16"
server_key_version: "encryptionkey@0"
I20251209 08:58:53.288445 27992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:58:53.288124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.49.124:44451 (local address 127.0.0.1:56776)
1209 08:58:53.288260 (+ 136us) negotiation.cc:107] Waiting for socket to connect
1209 08:58:53.288353 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.49.124:44451: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20251209 08:58:53.288749 27425 master.cc:679] Network error: unable to get registration information for peer 0364ed40f171452f9fa244fa50f6a6c3: Client connection negotiation failed: client connection to 127.20.49.124:44451: connect: Connection refused (error 111)
I20251209 08:58:53.289556 28020 ts_tablet_manager.cc:1916] T 00000000000000000000000000000000 P 0364ed40f171452f9fa244fa50f6a6c3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251209 08:58:53.291663 28020 ts_tablet_manager.cc:1929] T 00000000000000000000000000000000 P 0364ed40f171452f9fa244fa50f6a6c3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20251209 08:58:53.291745 28020 log.cc:1199] T 00000000000000000000000000000000 P 0364ed40f171452f9fa244fa50f6a6c3: Deleting WAL directory at /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765270668420570-20677-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20251209 08:58:53.291980 28020 ts_tablet_manager.cc:1950] T 00000000000000000000000000000000 P 0364ed40f171452f9fa244fa50f6a6c3: Deleting consensus metadata
I20251209 08:58:53.292145 28020 master_runner.cc:315] Copying system tablet from 127.20.49.126:42449
I20251209 08:58:53.293346 28020 tablet_copy_client.cc:323] T 00000000000000000000000000000000 P 0364ed40f171452f9fa244fa50f6a6c3: tablet copy: Beginning tablet copy session from remote peer at address 127.20.49.126:42449
I20251209 08:58:53.300957 27466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:58:53.294010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58536 (local address 127.20.49.126:42449)
1209 08:58:53.294164 (+ 154us) server_negotiation.cc:207] Beginning negotiation
1209 08:58:53.294171 (+ 7us) server_negotiation.cc:400] Waiting for connection header
1209 08:58:53.294240 (+ 69us) server_negotiation.cc:408] Connection header received
1209 08:58:53.294390 (+ 150us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1209 08:58:53.294393 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1209 08:58:53.294452 (+ 59us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1209 08:58:53.294557 (+ 105us) server_negotiation.cc:227] Negotiated authn=SASL
1209 08:58:53.295106 (+ 549us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:58:53.295942 (+ 836us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1209 08:58:53.296637 (+ 695us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:58:53.296841 (+ 204us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:58:53.297736 (+ 895us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1209 08:58:53.297757 (+ 21us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1209 08:58:53.297760 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1209 08:58:53.297790 (+ 30us) server_negotiation.cc:923] calling sasl_server_start()
1209 08:58:53.299377 (+ 1587us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1209 08:58:53.299843 (+ 466us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1209 08:58:53.299846 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1209 08:58:53.299848 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1209 08:58:53.299892 (+ 44us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1209 08:58:53.300115 (+ 223us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1209 08:58:53.300118 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1209 08:58:53.300120 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1209 08:58:53.300271 (+ 151us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1209 08:58:53.300429 (+ 158us) server_negotiation.cc:1036] Waiting for connection context
1209 08:58:53.300691 (+ 262us) server_negotiation.cc:300] Negotiation successful
1209 08:58:53.300810 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20251209 08:58:53.300972 28194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:58:53.293903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.49.126:42449 (local address 127.0.0.1:58536)
1209 08:58:53.294168 (+ 265us) negotiation.cc:107] Waiting for socket to connect
1209 08:58:53.294182 (+ 14us) client_negotiation.cc:174] Beginning negotiation
1209 08:58:53.294283 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1209 08:58:53.294567 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1209 08:58:53.294569 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1209 08:58:53.294775 (+ 206us) client_negotiation.cc:189] Negotiated authn=SASL
1209 08:58:53.294994 (+ 219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:58:53.295001 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:58:53.296085 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1209 08:58:53.296114 (+ 29us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1209 08:58:53.296516 (+ 402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:58:53.296524 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:58:53.296627 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:58:53.297226 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1209 08:58:53.297239 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
1209 08:58:53.297562 (+ 323us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1209 08:58:53.299532 (+ 1970us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1209 08:58:53.299535 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1209 08:58:53.299536 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1209 08:58:53.299746 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1209 08:58:53.299986 (+ 240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1209 08:58:53.299989 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1209 08:58:53.299991 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1209 08:58:53.300037 (+ 46us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1209 08:58:53.300424 (+ 387us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1209 08:58:53.300427 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1209 08:58:53.300516 (+ 89us) client_negotiation.cc:769] Sending connection context
1209 08:58:53.300629 (+ 113us) client_negotiation.cc:240] Negotiation successful
1209 08:58:53.300783 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":97,"threads_started":1}
I20251209 08:58:53.301378 27447 tablet_copy_service.cc:140] P c0a0d121c6bb474dac7c98dc7ec52d14: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 0364ed40f171452f9fa244fa50f6a6c3 ({username='oryx', principal='oryx/127.20.49.124@KRBTEST.COM'} at 127.0.0.1:58536)
I20251209 08:58:53.301465 27447 tablet_copy_service.cc:161] P c0a0d121c6bb474dac7c98dc7ec52d14: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 0364ed40f171452f9fa244fa50f6a6c3 at {username='oryx', principal='oryx/127.20.49.124@KRBTEST.COM'} at 127.0.0.1:58536: session id = 0364ed40f171452f9fa244fa50f6a6c3-00000000000000000000000000000000
I20251209 08:58:53.302322 27447 tablet_copy_source_session.cc:215] T 00000000000000000000000000000000 P c0a0d121c6bb474dac7c98dc7ec52d14: Tablet Copy: opened 0 blocks and 1 log segments
I20251209 08:58:53.303509 28020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20251209 08:58:53.305255 28020 tablet_copy_client.cc:806] T 00000000000000000000000000000000 P 0364ed40f171452f9fa244fa50f6a6c3: tablet copy: Starting download of 0 data blocks...
I20251209 08:58:53.305439 28020 tablet_copy_client.cc:670] T 00000000000000000000000000000000 P 0364ed40f171452f9fa244fa50f6a6c3: tablet copy: Starting download of 1 WAL segments...
I20251209 08:58:53.306958 28020 tablet_copy_client.cc:538] T 00000000000000000000000000000000 P 0364ed40f171452f9fa244fa50f6a6c3: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20251209 08:58:53.308887 27447 tablet_copy_service.cc:342] P c0a0d121c6bb474dac7c98dc7ec52d14: Request end of tablet copy session 0364ed40f171452f9fa244fa50f6a6c3-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.20.49.124@KRBTEST.COM'} at 127.0.0.1:58536
I20251209 08:58:53.308972 27447 tablet_copy_service.cc:434] P c0a0d121c6bb474dac7c98dc7ec52d14: ending tablet copy session 0364ed40f171452f9fa244fa50f6a6c3-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 0364ed40f171452f9fa244fa50f6a6c3
W20251209 08:58:53.313812 28020 builtin_ntp.cc:688] coult not shutdown socket: Network error: shutdown error: Transport endpoint is not connected (error 107)
I20251209 08:58:53.316282 28020 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251209 08:58:53.318045 28196 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251209 08:58:53.318046 28195 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251209 08:58:53.318260 28198 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251209 08:58:53.318423 28020 server_base.cc:1047] running on GCE node
I20251209 08:58:53.318591 28020 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251209 08:58:53.318878 28020 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251209 08:58:53.320047 28020 hybrid_clock.cc:648] HybridClock initialized: now 1765270733320039 us; error 30 us; skew 500 ppm
Dec 09 08:58:53 dist-test-slave-8w6c krb5kdc[27379](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765270733, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.49.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251209 08:58:53.321828 28020 init.cc:377] Logged in from keytab as oryx/127.20.49.124@KRBTEST.COM (short username oryx)
I20251209 08:58:53.322333 28020 webserver.cc:492] Webserver started at http://127.20.49.124:42749/ using document root <none> and password file <none>
I20251209 08:58:53.322544 28020 fs_manager.cc:362] Metadata directory not provided
I20251209 08:58:53.322585 28020 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251209 08:58:53.323920 28020 fs_manager.cc:714] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20251209 08:58:53.325078 28205 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251209 08:58:53.325181 27992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:58:53.324761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.49.124:44451 (local address 127.0.0.1:56788)
1209 08:58:53.324883 (+ 122us) negotiation.cc:107] Waiting for socket to connect
1209 08:58:53.325055 (+ 172us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.49.124:44451: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20251209 08:58:53.325393 27425 master.cc:679] Network error: unable to get registration information for peer 0364ed40f171452f9fa244fa50f6a6c3: Client connection negotiation failed: client connection to 127.20.49.124:44451: connect: Connection refused (error 111)
I20251209 08:58:53.325810 28020 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20251209 08:58:53.325891 28020 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765270668420570-20677-0/minicluster-data/master-2/data,/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765270668420570-20677-0/minicluster-data/master-2/wal
uuid: "0364ed40f171452f9fa244fa50f6a6c3"
format_stamp: "Formatted at 2025-12-09 08:58:53 on dist-test-slave-8w6c"
server_key: "ded07db4bbf58c7410644fc982d50862"
server_key_iv: "51de27466f9232ff51b1b7f797268a16"
server_key_version: "encryptionkey@0"
I20251209 08:58:53.325968 28020 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765270668420570-20677-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765270668420570-20677-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765270668420570-20677-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251209 08:58:53.329797 27756 heartbeater.cc:499] Master 127.20.49.126:42449 was elected leader, sending a full tablet report...
I20251209 08:58:53.346197 28020 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251209 08:58:53.348376 28020 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251209 08:58:53.353773 28020 rpc_server.cc:307] RPC server started. Bound to: 127.20.49.124:44451
I20251209 08:58:53.353891 28257 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.49.124:44451 every 8 connection(s)
I20251209 08:58:53.354423 28020 server_base.cc:1179] Dumped server information to /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765270668420570-20677-0/minicluster-data/master-2/data/info.pb
I20251209 08:58:53.356091 28258 sys_catalog.cc:263] Verifying existing consensus state
I20251209 08:58:53.356915 28258 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0364ed40f171452f9fa244fa50f6a6c3: Bootstrap starting.
I20251209 08:58:53.366394 28258 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0364ed40f171452f9fa244fa50f6a6c3: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251209 08:58:53.366696 28258 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0364ed40f171452f9fa244fa50f6a6c3: Bootstrap complete.
I20251209 08:58:53.367278 28258 raft_consensus.cc:359] T 00000000000000000000000000000000 P 0364ed40f171452f9fa244fa50f6a6c3 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c0a0d121c6bb474dac7c98dc7ec52d14" member_type: VOTER last_known_addr { host: "127.20.49.126" port: 42449 } } peers { permanent_uuid: "798ad87d8a5a4392815120c18c40bedf" member_type: VOTER last_known_addr { host: "127.20.49.125" port: 44905 } attrs { promote: false } } peers { permanent_uuid: "0364ed40f171452f9fa244fa50f6a6c3" member_type: NON_VOTER last_known_addr { host: "127.20.49.124" port: 44451 } attrs { promote: true } }
I20251209 08:58:53.367403 28258 raft_consensus.cc:740] T 00000000000000000000000000000000 P 0364ed40f171452f9fa244fa50f6a6c3 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 0364ed40f171452f9fa244fa50f6a6c3, State: Initialized, Role: LEARNER
I20251209 08:58:53.367522 28258 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0364ed40f171452f9fa244fa50f6a6c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c0a0d121c6bb474dac7c98dc7ec52d14" member_type: VOTER last_known_addr { host: "127.20.49.126" port: 42449 } } peers { permanent_uuid: "798ad87d8a5a4392815120c18c40bedf" member_type: VOTER last_known_addr { host: "127.20.49.125" port: 44905 } attrs { promote: false } } peers { permanent_uuid: "0364ed40f171452f9fa244fa50f6a6c3" member_type: NON_VOTER last_known_addr { host: "127.20.49.124" port: 44451 } attrs { promote: true } }
I20251209 08:58:53.367858 28261 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0364ed40f171452f9fa244fa50f6a6c3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c0a0d121c6bb474dac7c98dc7ec52d14" member_type: VOTER last_known_addr { host: "127.20.49.126" port: 42449 } } peers { permanent_uuid: "798ad87d8a5a4392815120c18c40bedf" member_type: VOTER last_known_addr { host: "127.20.49.125" port: 44905 } attrs { promote: false } } peers { permanent_uuid: "0364ed40f171452f9fa244fa50f6a6c3" member_type: NON_VOTER last_known_addr { host: "127.20.49.124" port: 44451 } attrs { promote: true } } }
I20251209 08:58:53.367966 28261 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0364ed40f171452f9fa244fa50f6a6c3 [sys.catalog]: This master's current role is: LEARNER
I20251209 08:58:53.368045 28258 sys_catalog.cc:565] T 00000000000000000000000000000000 P 0364ed40f171452f9fa244fa50f6a6c3 [sys.catalog]: configured and running, proceeding with master startup.
I20251209 08:58:53.371174 28272 catalog_manager.cc:1269] Loaded cluster ID: 51c444202dc5408cb61915bc1a2c504c
I20251209 08:58:53.371240 28272 catalog_manager.cc:1562] T 00000000000000000000000000000000 P 0364ed40f171452f9fa244fa50f6a6c3: loading cluster ID for follower catalog manager: success
I20251209 08:58:53.373106 28272 catalog_manager.cc:1584] T 00000000000000000000000000000000 P 0364ed40f171452f9fa244fa50f6a6c3: acquiring CA information for follower catalog manager: success
I20251209 08:58:53.374470 28272 catalog_manager.cc:1612] T 00000000000000000000000000000000 P 0364ed40f171452f9fa244fa50f6a6c3: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20251209 08:58:53.396867 27992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:58:53.393566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.49.124:44451 (local address 127.0.0.1:56792)
1209 08:58:53.393807 (+ 241us) negotiation.cc:107] Waiting for socket to connect
1209 08:58:53.393825 (+ 18us) client_negotiation.cc:174] Beginning negotiation
1209 08:58:53.393918 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1209 08:58:53.394268 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1209 08:58:53.394270 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1209 08:58:53.394285 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1209 08:58:53.394512 (+ 227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:58:53.394522 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:58:53.395806 (+ 1284us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1209 08:58:53.395810 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1209 08:58:53.396435 (+ 625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:58:53.396443 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:58:53.396565 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:58:53.396590 (+ 25us) client_negotiation.cc:769] Sending connection context
1209 08:58:53.396646 (+ 56us) client_negotiation.cc:240] Negotiation successful
1209 08:58:53.396718 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135}
I20251209 08:58:53.397389 28273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:58:53.393722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56792 (local address 127.20.49.124:44451)
1209 08:58:53.393988 (+ 266us) server_negotiation.cc:207] Beginning negotiation
1209 08:58:53.393994 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1209 08:58:53.394010 (+ 16us) server_negotiation.cc:408] Connection header received
1209 08:58:53.394066 (+ 56us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1209 08:58:53.394076 (+ 10us) server_negotiation.cc:462] Received NEGOTIATE request from client
1209 08:58:53.394138 (+ 62us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1209 08:58:53.394248 (+ 110us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1209 08:58:53.394655 (+ 407us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:58:53.395689 (+ 1034us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1209 08:58:53.396552 (+ 863us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:58:53.397021 (+ 469us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:58:53.397167 (+ 146us) server_negotiation.cc:1036] Waiting for connection context
1209 08:58:53.397220 (+ 53us) server_negotiation.cc:300] Negotiation successful
1209 08:58:53.397276 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":174,"thread_start_us":84,"threads_started":1}
I20251209 08:58:53.565981 27892 heartbeater.cc:499] Master 127.20.49.126:42449 was elected leader, sending a full tablet report...
I20251209 08:58:53.722892 28232 raft_consensus.cc:1217] T 00000000000000000000000000000000 P 0364ed40f171452f9fa244fa50f6a6c3 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20251209 08:58:53.723320 28261 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0364ed40f171452f9fa244fa50f6a6c3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c0a0d121c6bb474dac7c98dc7ec52d14. Latest consensus state: current_term: 1 leader_uuid: "c0a0d121c6bb474dac7c98dc7ec52d14" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c0a0d121c6bb474dac7c98dc7ec52d14" member_type: VOTER last_known_addr { host: "127.20.49.126" port: 42449 } } peers { permanent_uuid: "798ad87d8a5a4392815120c18c40bedf" member_type: VOTER last_known_addr { host: "127.20.49.125" port: 44905 } attrs { promote: false } } peers { permanent_uuid: "0364ed40f171452f9fa244fa50f6a6c3" member_type: NON_VOTER last_known_addr { host: "127.20.49.124" port: 44451 } attrs { promote: true } } }
I20251209 08:58:53.723466 28261 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0364ed40f171452f9fa244fa50f6a6c3 [sys.catalog]: This master's current role is: LEARNER
I20251209 08:58:54.271282 27997 sys_catalog.cc:455] T 00000000000000000000000000000000 P c0a0d121c6bb474dac7c98dc7ec52d14 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c0a0d121c6bb474dac7c98dc7ec52d14" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c0a0d121c6bb474dac7c98dc7ec52d14" member_type: VOTER last_known_addr { host: "127.20.49.126" port: 42449 } } peers { permanent_uuid: "798ad87d8a5a4392815120c18c40bedf" member_type: VOTER last_known_addr { host: "127.20.49.125" port: 44905 } attrs { promote: false } } peers { permanent_uuid: "0364ed40f171452f9fa244fa50f6a6c3" member_type: NON_VOTER last_known_addr { host: "127.20.49.124" port: 44451 } attrs { promote: true } } }
I20251209 08:58:54.271490 27997 sys_catalog.cc:458] T 00000000000000000000000000000000 P c0a0d121c6bb474dac7c98dc7ec52d14 [sys.catalog]: This master's current role is: LEADER
I20251209 08:58:54.272468 28275 raft_consensus.cc:1064] T 00000000000000000000000000000000 P c0a0d121c6bb474dac7c98dc7ec52d14: attempting to promote NON_VOTER 0364ed40f171452f9fa244fa50f6a6c3 to VOTER
I20251209 08:58:54.272850 28275 consensus_queue.cc:237] T 00000000000000000000000000000000 P c0a0d121c6bb474dac7c98dc7ec52d14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "c0a0d121c6bb474dac7c98dc7ec52d14" member_type: VOTER last_known_addr { host: "127.20.49.126" port: 42449 } } peers { permanent_uuid: "798ad87d8a5a4392815120c18c40bedf" member_type: VOTER last_known_addr { host: "127.20.49.125" port: 44905 } attrs { promote: false } } peers { permanent_uuid: "0364ed40f171452f9fa244fa50f6a6c3" member_type: VOTER last_known_addr { host: "127.20.49.124" port: 44451 } attrs { promote: false } }
I20251209 08:58:54.274057 28232 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 0364ed40f171452f9fa244fa50f6a6c3 [term 1 LEARNER]: Refusing update from remote peer c0a0d121c6bb474dac7c98dc7ec52d14: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20251209 08:58:54.274096 28049 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 798ad87d8a5a4392815120c18c40bedf [term 1 FOLLOWER]: Refusing update from remote peer c0a0d121c6bb474dac7c98dc7ec52d14: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20251209 08:58:54.274502 27997 consensus_queue.cc:1048] T 00000000000000000000000000000000 P c0a0d121c6bb474dac7c98dc7ec52d14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0364ed40f171452f9fa244fa50f6a6c3" member_type: VOTER last_known_addr { host: "127.20.49.124" port: 44451 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20251209 08:58:54.274748 28275 consensus_queue.cc:1048] T 00000000000000000000000000000000 P c0a0d121c6bb474dac7c98dc7ec52d14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "798ad87d8a5a4392815120c18c40bedf" member_type: VOTER last_known_addr { host: "127.20.49.125" port: 44905 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20251209 08:58:54.276546 27997 raft_consensus.cc:2955] T 00000000000000000000000000000000 P c0a0d121c6bb474dac7c98dc7ec52d14 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 0364ed40f171452f9fa244fa50f6a6c3 (127.20.49.124) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "c0a0d121c6bb474dac7c98dc7ec52d14" member_type: VOTER last_known_addr { host: "127.20.49.126" port: 42449 } } peers { permanent_uuid: "798ad87d8a5a4392815120c18c40bedf" member_type: VOTER last_known_addr { host: "127.20.49.125" port: 44905 } attrs { promote: false } } peers { permanent_uuid: "0364ed40f171452f9fa244fa50f6a6c3" member_type: VOTER last_known_addr { host: "127.20.49.124" port: 44451 } attrs { promote: false } } }
I20251209 08:58:54.276990 28049 raft_consensus.cc:2955] T 00000000000000000000000000000000 P 798ad87d8a5a4392815120c18c40bedf [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 0364ed40f171452f9fa244fa50f6a6c3 (127.20.49.124) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "c0a0d121c6bb474dac7c98dc7ec52d14" member_type: VOTER last_known_addr { host: "127.20.49.126" port: 42449 } } peers { permanent_uuid: "798ad87d8a5a4392815120c18c40bedf" member_type: VOTER last_known_addr { host: "127.20.49.125" port: 44905 } attrs { promote: false } } peers { permanent_uuid: "0364ed40f171452f9fa244fa50f6a6c3" member_type: VOTER last_known_addr { host: "127.20.49.124" port: 44451 } attrs { promote: false } } }
I20251209 08:58:54.278307 28275 sys_catalog.cc:455] T 00000000000000000000000000000000 P c0a0d121c6bb474dac7c98dc7ec52d14 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "c0a0d121c6bb474dac7c98dc7ec52d14" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "c0a0d121c6bb474dac7c98dc7ec52d14" member_type: VOTER last_known_addr { host: "127.20.49.126" port: 42449 } } peers { permanent_uuid: "798ad87d8a5a4392815120c18c40bedf" member_type: VOTER last_known_addr { host: "127.20.49.125" port: 44905 } attrs { promote: false } } peers { permanent_uuid: "0364ed40f171452f9fa244fa50f6a6c3" member_type: VOTER last_known_addr { host: "127.20.49.124" port: 44451 } attrs { promote: false } } }
I20251209 08:58:54.278366 28274 sys_catalog.cc:455] T 00000000000000000000000000000000 P c0a0d121c6bb474dac7c98dc7ec52d14 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c0a0d121c6bb474dac7c98dc7ec52d14" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "c0a0d121c6bb474dac7c98dc7ec52d14" member_type: VOTER last_known_addr { host: "127.20.49.126" port: 42449 } } peers { permanent_uuid: "798ad87d8a5a4392815120c18c40bedf" member_type: VOTER last_known_addr { host: "127.20.49.125" port: 44905 } attrs { promote: false } } peers { permanent_uuid: "0364ed40f171452f9fa244fa50f6a6c3" member_type: VOTER last_known_addr { host: "127.20.49.124" port: 44451 } attrs { promote: false } } }
I20251209 08:58:54.278416 28275 sys_catalog.cc:458] T 00000000000000000000000000000000 P c0a0d121c6bb474dac7c98dc7ec52d14 [sys.catalog]: This master's current role is: LEADER
I20251209 08:58:54.278437 28274 sys_catalog.cc:458] T 00000000000000000000000000000000 P c0a0d121c6bb474dac7c98dc7ec52d14 [sys.catalog]: This master's current role is: LEADER
I20251209 08:58:54.278080 28232 raft_consensus.cc:2955] T 00000000000000000000000000000000 P 0364ed40f171452f9fa244fa50f6a6c3 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 0364ed40f171452f9fa244fa50f6a6c3 (127.20.49.124) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "c0a0d121c6bb474dac7c98dc7ec52d14" member_type: VOTER last_known_addr { host: "127.20.49.126" port: 42449 } } peers { permanent_uuid: "798ad87d8a5a4392815120c18c40bedf" member_type: VOTER last_known_addr { host: "127.20.49.125" port: 44905 } attrs { promote: false } } peers { permanent_uuid: "0364ed40f171452f9fa244fa50f6a6c3" member_type: VOTER last_known_addr { host: "127.20.49.124" port: 44451 } attrs { promote: false } } }
I20251209 08:58:54.278578 28274 sys_catalog.cc:455] T 00000000000000000000000000000000 P c0a0d121c6bb474dac7c98dc7ec52d14 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c0a0d121c6bb474dac7c98dc7ec52d14" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "c0a0d121c6bb474dac7c98dc7ec52d14" member_type: VOTER last_known_addr { host: "127.20.49.126" port: 42449 } } peers { permanent_uuid: "798ad87d8a5a4392815120c18c40bedf" member_type: VOTER last_known_addr { host: "127.20.49.125" port: 44905 } attrs { promote: false } } peers { permanent_uuid: "0364ed40f171452f9fa244fa50f6a6c3" member_type: VOTER last_known_addr { host: "127.20.49.124" port: 44451 } attrs { promote: false } } }
I20251209 08:58:54.278630 28274 sys_catalog.cc:458] T 00000000000000000000000000000000 P c0a0d121c6bb474dac7c98dc7ec52d14 [sys.catalog]: This master's current role is: LEADER
I20251209 08:58:54.279134 28280 sys_catalog.cc:455] T 00000000000000000000000000000000 P 798ad87d8a5a4392815120c18c40bedf [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c0a0d121c6bb474dac7c98dc7ec52d14" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "c0a0d121c6bb474dac7c98dc7ec52d14" member_type: VOTER last_known_addr { host: "127.20.49.126" port: 42449 } } peers { permanent_uuid: "798ad87d8a5a4392815120c18c40bedf" member_type: VOTER last_known_addr { host: "127.20.49.125" port: 44905 } attrs { promote: false } } peers { permanent_uuid: "0364ed40f171452f9fa244fa50f6a6c3" member_type: VOTER last_known_addr { host: "127.20.49.124" port: 44451 } attrs { promote: false } } }
I20251209 08:58:54.279243 28280 sys_catalog.cc:458] T 00000000000000000000000000000000 P 798ad87d8a5a4392815120c18c40bedf [sys.catalog]: This master's current role is: FOLLOWER
I20251209 08:58:54.279932 28281 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0364ed40f171452f9fa244fa50f6a6c3 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c0a0d121c6bb474dac7c98dc7ec52d14" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "c0a0d121c6bb474dac7c98dc7ec52d14" member_type: VOTER last_known_addr { host: "127.20.49.126" port: 42449 } } peers { permanent_uuid: "798ad87d8a5a4392815120c18c40bedf" member_type: VOTER last_known_addr { host: "127.20.49.125" port: 44905 } attrs { promote: false } } peers { permanent_uuid: "0364ed40f171452f9fa244fa50f6a6c3" member_type: VOTER last_known_addr { host: "127.20.49.124" port: 44451 } attrs { promote: false } } }
I20251209 08:58:54.280059 28281 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0364ed40f171452f9fa244fa50f6a6c3 [sys.catalog]: This master's current role is: FOLLOWER
I20251209 08:58:54.316607 28283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:58:54.309153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36942 (local address 127.20.49.125:44905)
1209 08:58:54.309435 (+ 282us) server_negotiation.cc:207] Beginning negotiation
1209 08:58:54.309439 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1209 08:58:54.309453 (+ 14us) server_negotiation.cc:408] Connection header received
1209 08:58:54.309490 (+ 37us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1209 08:58:54.309492 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1209 08:58:54.309533 (+ 41us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1209 08:58:54.309618 (+ 85us) server_negotiation.cc:227] Negotiated authn=SASL
1209 08:58:54.310706 (+ 1088us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:58:54.311286 (+ 580us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1209 08:58:54.311941 (+ 655us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:58:54.312113 (+ 172us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:58:54.313387 (+ 1274us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1209 08:58:54.313407 (+ 20us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1209 08:58:54.313410 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1209 08:58:54.313434 (+ 24us) server_negotiation.cc:923] calling sasl_server_start()
1209 08:58:54.315124 (+ 1690us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1209 08:58:54.315545 (+ 421us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1209 08:58:54.315549 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1209 08:58:54.315550 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1209 08:58:54.315595 (+ 45us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1209 08:58:54.315818 (+ 223us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1209 08:58:54.315821 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1209 08:58:54.315823 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1209 08:58:54.315976 (+ 153us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1209 08:58:54.316099 (+ 123us) server_negotiation.cc:1036] Waiting for connection context
1209 08:58:54.316307 (+ 208us) server_negotiation.cc:300] Negotiation successful
1209 08:58:54.316437 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":208,"thread_start_us":97,"threads_started":1}
I20251209 08:58:54.320286 28284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:58:54.317393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.49.126:42449 (local address 127.0.0.1:58544)
1209 08:58:54.317666 (+ 273us) negotiation.cc:107] Waiting for socket to connect
1209 08:58:54.317704 (+ 38us) client_negotiation.cc:174] Beginning negotiation
1209 08:58:54.317798 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1209 08:58:54.318053 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1209 08:58:54.318057 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
1209 08:58:54.318075 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1209 08:58:54.318323 (+ 248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:58:54.318330 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:58:54.319100 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1209 08:58:54.319189 (+ 89us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1209 08:58:54.319843 (+ 654us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:58:54.319853 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:58:54.319991 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:58:54.320012 (+ 21us) client_negotiation.cc:769] Sending connection context
1209 08:58:54.320067 (+ 55us) client_negotiation.cc:240] Negotiation successful
1209 08:58:54.320130 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":93,"threads_started":1}
I20251209 08:58:54.320765 28285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:58:54.317554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58544 (local address 127.20.49.126:42449)
1209 08:58:54.317849 (+ 295us) server_negotiation.cc:207] Beginning negotiation
1209 08:58:54.317854 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1209 08:58:54.317869 (+ 15us) server_negotiation.cc:408] Connection header received
1209 08:58:54.317916 (+ 47us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1209 08:58:54.317918 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1209 08:58:54.317962 (+ 44us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1209 08:58:54.318040 (+ 78us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1209 08:58:54.318451 (+ 411us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:58:54.318984 (+ 533us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1209 08:58:54.319992 (+ 1008us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:58:54.320502 (+ 510us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:58:54.320541 (+ 39us) server_negotiation.cc:1036] Waiting for connection context
1209 08:58:54.320610 (+ 69us) server_negotiation.cc:300] Negotiation successful
1209 08:58:54.320665 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":206,"thread_start_us":83,"threads_started":1}
I20251209 08:58:54.324055 28284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:58:54.321967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.49.125:44905 (local address 127.0.0.1:36956)
1209 08:58:54.322084 (+ 117us) negotiation.cc:107] Waiting for socket to connect
1209 08:58:54.322099 (+ 15us) client_negotiation.cc:174] Beginning negotiation
1209 08:58:54.322165 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1209 08:58:54.322351 (+ 186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1209 08:58:54.322353 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1209 08:58:54.322374 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1209 08:58:54.322515 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:58:54.322519 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:58:54.323151 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1209 08:58:54.323154 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1209 08:58:54.323761 (+ 607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:58:54.323769 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:58:54.323862 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:58:54.323875 (+ 13us) client_negotiation.cc:769] Sending connection context
1209 08:58:54.323919 (+ 44us) client_negotiation.cc:240] Negotiation successful
1209 08:58:54.323966 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42,"spinlock_wait_cycles":1792}
I20251209 08:58:54.324432 28283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:58:54.322049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36956 (local address 127.20.49.125:44905)
1209 08:58:54.322153 (+ 104us) server_negotiation.cc:207] Beginning negotiation
1209 08:58:54.322157 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1209 08:58:54.322169 (+ 12us) server_negotiation.cc:408] Connection header received
1209 08:58:54.322241 (+ 72us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1209 08:58:54.322244 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1209 08:58:54.322282 (+ 38us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1209 08:58:54.322338 (+ 56us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1209 08:58:54.322593 (+ 255us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:58:54.323050 (+ 457us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1209 08:58:54.323862 (+ 812us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:58:54.324213 (+ 351us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:58:54.324246 (+ 33us) server_negotiation.cc:1036] Waiting for connection context
1209 08:58:54.324288 (+ 42us) server_negotiation.cc:300] Negotiation successful
1209 08:58:54.324343 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20251209 08:58:54.327971 28284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:58:54.325588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.49.124:44451 (local address 127.0.0.1:56794)
1209 08:58:54.325751 (+ 163us) negotiation.cc:107] Waiting for socket to connect
1209 08:58:54.325764 (+ 13us) client_negotiation.cc:174] Beginning negotiation
1209 08:58:54.325832 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1209 08:58:54.326223 (+ 391us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1209 08:58:54.326225 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1209 08:58:54.326236 (+ 11us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1209 08:58:54.326386 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:58:54.326392 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:58:54.327101 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1209 08:58:54.327104 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1209 08:58:54.327701 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:58:54.327709 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:58:54.327791 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:58:54.327804 (+ 13us) client_negotiation.cc:769] Sending connection context
1209 08:58:54.327842 (+ 38us) client_negotiation.cc:240] Negotiation successful
1209 08:58:54.327884 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":68}
I20251209 08:58:54.328490 28286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:58:54.325789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56794 (local address 127.20.49.124:44451)
1209 08:58:54.326031 (+ 242us) server_negotiation.cc:207] Beginning negotiation
1209 08:58:54.326035 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1209 08:58:54.326047 (+ 12us) server_negotiation.cc:408] Connection header received
1209 08:58:54.326089 (+ 42us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1209 08:58:54.326092 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1209 08:58:54.326133 (+ 41us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1209 08:58:54.326214 (+ 81us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1209 08:58:54.326472 (+ 258us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:58:54.327011 (+ 539us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1209 08:58:54.327814 (+ 803us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:58:54.328206 (+ 392us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:58:54.328256 (+ 50us) server_negotiation.cc:1036] Waiting for connection context
1209 08:58:54.328301 (+ 45us) server_negotiation.cc:300] Negotiation successful
1209 08:58:54.328344 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":176,"thread_start_us":91,"threads_started":1}
I20251209 08:58:54.337026 28286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:58:54.330256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56808 (local address 127.20.49.124:44451)
1209 08:58:54.330390 (+ 134us) server_negotiation.cc:207] Beginning negotiation
1209 08:58:54.330393 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1209 08:58:54.330410 (+ 17us) server_negotiation.cc:408] Connection header received
1209 08:58:54.330515 (+ 105us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1209 08:58:54.330517 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1209 08:58:54.330553 (+ 36us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1209 08:58:54.330616 (+ 63us) server_negotiation.cc:227] Negotiated authn=SASL
1209 08:58:54.331355 (+ 739us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:58:54.331889 (+ 534us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1209 08:58:54.332564 (+ 675us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:58:54.332763 (+ 199us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:58:54.333976 (+ 1213us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1209 08:58:54.333992 (+ 16us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1209 08:58:54.333995 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1209 08:58:54.334019 (+ 24us) server_negotiation.cc:923] calling sasl_server_start()
1209 08:58:54.335600 (+ 1581us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1209 08:58:54.335980 (+ 380us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1209 08:58:54.335984 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1209 08:58:54.335988 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
1209 08:58:54.336030 (+ 42us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1209 08:58:54.336246 (+ 216us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1209 08:58:54.336249 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1209 08:58:54.336251 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1209 08:58:54.336402 (+ 151us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1209 08:58:54.336528 (+ 126us) server_negotiation.cc:1036] Waiting for connection context
1209 08:58:54.336738 (+ 210us) server_negotiation.cc:300] Negotiation successful
1209 08:58:54.336867 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20251209 08:58:54.340231 28287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:58:54.337952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.49.126:42449 (local address 127.0.0.1:58560)
1209 08:58:54.338205 (+ 253us) negotiation.cc:107] Waiting for socket to connect
1209 08:58:54.338217 (+ 12us) client_negotiation.cc:174] Beginning negotiation
1209 08:58:54.338313 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1209 08:58:54.338507 (+ 194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1209 08:58:54.338509 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1209 08:58:54.338522 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1209 08:58:54.338693 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:58:54.338697 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:58:54.339359 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1209 08:58:54.339361 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1209 08:58:54.339935 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:58:54.339942 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:58:54.340029 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:58:54.340044 (+ 15us) client_negotiation.cc:769] Sending connection context
1209 08:58:54.340081 (+ 37us) client_negotiation.cc:240] Negotiation successful
1209 08:58:54.340124 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":89,"threads_started":1}
I20251209 08:58:54.340690 28285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:58:54.338048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58560 (local address 127.20.49.126:42449)
1209 08:58:54.338208 (+ 160us) server_negotiation.cc:207] Beginning negotiation
1209 08:58:54.338212 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1209 08:58:54.338279 (+ 67us) server_negotiation.cc:408] Connection header received
1209 08:58:54.338384 (+ 105us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1209 08:58:54.338386 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1209 08:58:54.338429 (+ 43us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1209 08:58:54.338495 (+ 66us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1209 08:58:54.338775 (+ 280us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:58:54.339274 (+ 499us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1209 08:58:54.340052 (+ 778us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:58:54.340435 (+ 383us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:58:54.340483 (+ 48us) server_negotiation.cc:1036] Waiting for connection context
1209 08:58:54.340536 (+ 53us) server_negotiation.cc:300] Negotiation successful
1209 08:58:54.340584 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20251209 08:58:54.344187 28287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:58:54.341920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.49.125:44905 (local address 127.0.0.1:36966)
1209 08:58:54.342042 (+ 122us) negotiation.cc:107] Waiting for socket to connect
1209 08:58:54.342058 (+ 16us) client_negotiation.cc:174] Beginning negotiation
1209 08:58:54.342119 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1209 08:58:54.342326 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1209 08:58:54.342328 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1209 08:58:54.342341 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1209 08:58:54.342490 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:58:54.342494 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:58:54.343275 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1209 08:58:54.343278 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1209 08:58:54.343884 (+ 606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:58:54.343892 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:58:54.343993 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:58:54.344008 (+ 15us) client_negotiation.cc:769] Sending connection context
1209 08:58:54.344046 (+ 38us) client_negotiation.cc:240] Negotiation successful
1209 08:58:54.344089 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46}
I20251209 08:58:54.344692 28283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:58:54.342033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36966 (local address 127.20.49.125:44905)
1209 08:58:54.342147 (+ 114us) server_negotiation.cc:207] Beginning negotiation
1209 08:58:54.342150 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1209 08:58:54.342162 (+ 12us) server_negotiation.cc:408] Connection header received
1209 08:58:54.342206 (+ 44us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1209 08:58:54.342208 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1209 08:58:54.342250 (+ 42us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1209 08:58:54.342316 (+ 66us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1209 08:58:54.342666 (+ 350us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:58:54.343146 (+ 480us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1209 08:58:54.344019 (+ 873us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:58:54.344451 (+ 432us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:58:54.344493 (+ 42us) server_negotiation.cc:1036] Waiting for connection context
1209 08:58:54.344537 (+ 44us) server_negotiation.cc:300] Negotiation successful
1209 08:58:54.344590 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20251209 08:58:54.348205 28287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:58:54.346025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.49.124:44451 (local address 127.0.0.1:56816)
1209 08:58:54.346144 (+ 119us) negotiation.cc:107] Waiting for socket to connect
1209 08:58:54.346164 (+ 20us) client_negotiation.cc:174] Beginning negotiation
1209 08:58:54.346247 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1209 08:58:54.346492 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1209 08:58:54.346495 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1209 08:58:54.346512 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1209 08:58:54.346663 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:58:54.346668 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:58:54.347326 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1209 08:58:54.347330 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1209 08:58:54.347902 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:58:54.347910 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:58:54.348006 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:58:54.348020 (+ 14us) client_negotiation.cc:769] Sending connection context
1209 08:58:54.348056 (+ 36us) client_negotiation.cc:240] Negotiation successful
1209 08:58:54.348098 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20251209 08:58:54.348656 28286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:58:54.346041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56816 (local address 127.20.49.124:44451)
1209 08:58:54.346161 (+ 120us) server_negotiation.cc:207] Beginning negotiation
1209 08:58:54.346164 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1209 08:58:54.346218 (+ 54us) server_negotiation.cc:408] Connection header received
1209 08:58:54.346316 (+ 98us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1209 08:58:54.346319 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1209 08:58:54.346355 (+ 36us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1209 08:58:54.346441 (+ 86us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1209 08:58:54.346764 (+ 323us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:58:54.347217 (+ 453us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1209 08:58:54.348020 (+ 803us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:58:54.348455 (+ 435us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:58:54.348490 (+ 35us) server_negotiation.cc:1036] Waiting for connection context
1209 08:58:54.348529 (+ 39us) server_negotiation.cc:300] Negotiation successful
1209 08:58:54.348569 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20251209 08:58:54.350186 20677 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task8XuDZV/build/debug/bin/kudu with pid 27491
I20251209 08:58:54.357223 20677 external_mini_cluster.cc:1366] Running /tmp/dist-test-task8XuDZV/build/debug/bin/kudu
/tmp/dist-test-task8XuDZV/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765270668420570-20677-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765270668420570-20677-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765270668420570-20677-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765270668420570-20677-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.49.65:38709
--local_ip_for_outbound_sockets=127.20.49.65
--tserver_master_addrs=127.20.49.126:42449
--webserver_port=37597
--webserver_interface=127.20.49.65
--keytab_file=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765270668420570-20677-0/krb5kdc/oryx.keytab
--principal=oryx/127.20.49.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.20.49.84:40253
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.20.49.126:42449,127.20.49.125:44905
--tserver_master_addrs=127.20.49.126:42449,127.20.49.125:44905,127.20.49.124:44451 with env {KRB5CCNAME=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765270668420570-20677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765270668420570-20677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765270668420570-20677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251209 08:58:54.472476 28288 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251209 08:58:54.472725 28288 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251209 08:58:54.472788 28288 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251209 08:58:54.476292 28288 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251209 08:58:54.476369 28288 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251209 08:58:54.476450 28288 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.49.65
I20251209 08:58:54.480863 28288 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.49.84:40253
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765270668420570-20677-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765270668420570-20677-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765270668420570-20677-0/krb5kdc/oryx.keytab
--principal=oryx/127.20.49.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.49.65:38709
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765270668420570-20677-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.49.65
--webserver_port=37597
--webserver_require_spnego=true
--tserver_master_addrs=127.20.49.126:42449,127.20.49.125:44905,127.20.49.124:44451
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28288
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.49.65
--log_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765270668420570-20677-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 468897755969d686fb08386f42f7835685e7953a
build type DEBUG
built by None at 09 Dec 2025 08:43:15 UTC on 24a791456cd2
build id 9426
I20251209 08:58:54.481953 28288 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251209 08:58:54.482764 28288 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251209 08:58:54.489243 28293 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251209 08:58:54.489248 28294 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251209 08:58:54.489248 28296 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251209 08:58:54.490165 28288 server_base.cc:1047] running on GCE node
I20251209 08:58:54.490685 28288 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251209 08:58:54.491279 28288 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251209 08:58:54.492486 28288 hybrid_clock.cc:648] HybridClock initialized: now 1765270734492468 us; error 26 us; skew 500 ppm
Dec 09 08:58:54 dist-test-slave-8w6c krb5kdc[27379](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765270734, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.49.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251209 08:58:54.495756 28288 init.cc:377] Logged in from keytab as oryx/127.20.49.65@KRBTEST.COM (short username oryx)
I20251209 08:58:54.496831 28288 webserver.cc:492] Webserver started at http://127.20.49.65:37597/ using document root <none> and password file <none>
I20251209 08:58:54.497368 28288 fs_manager.cc:362] Metadata directory not provided
I20251209 08:58:54.497431 28288 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251209 08:58:54.501123 28288 fs_manager.cc:714] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20251209 08:58:54.503114 28303 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251209 08:58:54.504184 28288 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20251209 08:58:54.504292 28288 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765270668420570-20677-0/minicluster-data/ts-0/data,/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765270668420570-20677-0/minicluster-data/ts-0/wal
uuid: "c7b2b642c9144d568877db72b8553ac5"
format_stamp: "Formatted at 2025-12-09 08:58:51 on dist-test-slave-8w6c"
server_key: "204ae2da64bef4c03034d89940ac1943"
server_key_iv: "d69e132ff5074ea344155fba327b27e7"
server_key_version: "encryptionkey@0"
I20251209 08:58:54.504699 28288 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765270668420570-20677-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765270668420570-20677-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765270668420570-20677-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251209 08:58:54.521601 28288 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251209 08:58:54.524523 28288 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251209 08:58:54.524695 28288 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251209 08:58:54.525218 28288 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251209 08:58:54.526217 28288 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251209 08:58:54.526268 28288 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251209 08:58:54.526307 28288 ts_tablet_manager.cc:616] Registered 0 tablets
I20251209 08:58:54.526330 28288 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251209 08:58:54.535436 28288 rpc_server.cc:307] RPC server started. Bound to: 127.20.49.65:38709
I20251209 08:58:54.535486 28416 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.49.65:38709 every 8 connection(s)
I20251209 08:58:54.536476 28288 server_base.cc:1179] Dumped server information to /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765270668420570-20677-0/minicluster-data/ts-0/data/info.pb
Dec 09 08:58:54 dist-test-slave-8w6c krb5kdc[27379](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765270734, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.49.65@KRBTEST.COM for oryx/127.20.49.126@KRBTEST.COM
Dec 09 08:58:54 dist-test-slave-8w6c krb5kdc[27379](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765270734, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.49.65@KRBTEST.COM for oryx/127.20.49.125@KRBTEST.COM
I20251209 08:58:54.546275 20677 external_mini_cluster.cc:1428] Started /tmp/dist-test-task8XuDZV/build/debug/bin/kudu as pid 28288
I20251209 08:58:54.546447 20677 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task8XuDZV/build/debug/bin/kudu with pid 27627
Dec 09 08:58:54 dist-test-slave-8w6c krb5kdc[27379](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765270734, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.49.65@KRBTEST.COM for oryx/127.20.49.124@KRBTEST.COM
I20251209 08:58:54.551293 28285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:58:54.538889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.49.65:41397 (local address 127.20.49.126:42449)
1209 08:58:54.539021 (+ 132us) server_negotiation.cc:207] Beginning negotiation
1209 08:58:54.539026 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1209 08:58:54.539905 (+ 879us) server_negotiation.cc:408] Connection header received
1209 08:58:54.540458 (+ 553us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1209 08:58:54.540465 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
1209 08:58:54.540524 (+ 59us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1209 08:58:54.540597 (+ 73us) server_negotiation.cc:227] Negotiated authn=SASL
1209 08:58:54.542001 (+ 1404us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:58:54.542814 (+ 813us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1209 08:58:54.543481 (+ 667us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:58:54.543672 (+ 191us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:58:54.546276 (+ 2604us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1209 08:58:54.546299 (+ 23us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1209 08:58:54.546304 (+ 5us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1209 08:58:54.546346 (+ 42us) server_negotiation.cc:923] calling sasl_server_start()
1209 08:58:54.548872 (+ 2526us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1209 08:58:54.549656 (+ 784us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1209 08:58:54.549660 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1209 08:58:54.549662 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1209 08:58:54.549754 (+ 92us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1209 08:58:54.550144 (+ 390us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1209 08:58:54.550147 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1209 08:58:54.550150 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
1209 08:58:54.550321 (+ 171us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1209 08:58:54.550415 (+ 94us) server_negotiation.cc:1036] Waiting for connection context
1209 08:58:54.550967 (+ 552us) server_negotiation.cc:300] Negotiation successful
1209 08:58:54.551122 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20251209 08:58:54.551756 28283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:58:54.538756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.49.65:50291 (local address 127.20.49.125:44905)
1209 08:58:54.538901 (+ 145us) server_negotiation.cc:207] Beginning negotiation
1209 08:58:54.538906 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1209 08:58:54.539659 (+ 753us) server_negotiation.cc:408] Connection header received
1209 08:58:54.540437 (+ 778us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1209 08:58:54.540441 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1209 08:58:54.540494 (+ 53us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1209 08:58:54.540548 (+ 54us) server_negotiation.cc:227] Negotiated authn=SASL
1209 08:58:54.542058 (+ 1510us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:58:54.542812 (+ 754us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1209 08:58:54.543613 (+ 801us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:58:54.543799 (+ 186us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:58:54.547190 (+ 3391us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1209 08:58:54.547214 (+ 24us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1209 08:58:54.547217 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1209 08:58:54.547253 (+ 36us) server_negotiation.cc:923] calling sasl_server_start()
1209 08:58:54.549096 (+ 1843us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1209 08:58:54.549661 (+ 565us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1209 08:58:54.549664 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1209 08:58:54.549666 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1209 08:58:54.549786 (+ 120us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1209 08:58:54.550123 (+ 337us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1209 08:58:54.550129 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1209 08:58:54.550133 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
1209 08:58:54.550367 (+ 234us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1209 08:58:54.550492 (+ 125us) server_negotiation.cc:1036] Waiting for connection context
1209 08:58:54.550997 (+ 505us) server_negotiation.cc:300] Negotiation successful
1209 08:58:54.551148 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20251209 08:58:54.551936 28422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:58:54.538534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.49.126:42449 (local address 127.20.49.65:41397)
1209 08:58:54.539773 (+ 1239us) negotiation.cc:107] Waiting for socket to connect
1209 08:58:54.539787 (+ 14us) client_negotiation.cc:174] Beginning negotiation
1209 08:58:54.540201 (+ 414us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1209 08:58:54.540782 (+ 581us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1209 08:58:54.540791 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
1209 08:58:54.541228 (+ 437us) client_negotiation.cc:189] Negotiated authn=SASL
1209 08:58:54.541853 (+ 625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:58:54.541866 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:58:54.542980 (+ 1114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1209 08:58:54.542983 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1209 08:58:54.543355 (+ 372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:58:54.543361 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:58:54.543576 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:58:54.544235 (+ 659us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1209 08:58:54.544283 (+ 48us) client_negotiation.cc:656] calling sasl_client_start()
1209 08:58:54.546067 (+ 1784us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1209 08:58:54.549048 (+ 2981us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1209 08:58:54.549055 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1209 08:58:54.549069 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
1209 08:58:54.549527 (+ 458us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1209 08:58:54.549880 (+ 353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1209 08:58:54.549884 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1209 08:58:54.549887 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
1209 08:58:54.550030 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1209 08:58:54.550442 (+ 412us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1209 08:58:54.550450 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1209 08:58:54.550712 (+ 262us) client_negotiation.cc:769] Sending connection context
1209 08:58:54.550916 (+ 204us) client_negotiation.cc:240] Negotiation successful
1209 08:58:54.551218 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1166,"mutex_wait_us":485,"spinlock_wait_cycles":16000,"thread_start_us":95,"threads_started":1}
I20251209 08:58:54.552740 28423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:58:54.538534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.49.125:44905 (local address 127.20.49.65:50291)
1209 08:58:54.539549 (+ 1015us) negotiation.cc:107] Waiting for socket to connect
1209 08:58:54.539562 (+ 13us) client_negotiation.cc:174] Beginning negotiation
1209 08:58:54.540198 (+ 636us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1209 08:58:54.540783 (+ 585us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1209 08:58:54.540791 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
1209 08:58:54.541251 (+ 460us) client_negotiation.cc:189] Negotiated authn=SASL
1209 08:58:54.541867 (+ 616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:58:54.541876 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:58:54.542919 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1209 08:58:54.542923 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1209 08:58:54.543483 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:58:54.543494 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:58:54.543906 (+ 412us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:58:54.544558 (+ 652us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1209 08:58:54.544571 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
1209 08:58:54.547016 (+ 2445us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1209 08:58:54.549221 (+ 2205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1209 08:58:54.549224 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1209 08:58:54.549226 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1209 08:58:54.549527 (+ 301us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1209 08:58:54.549895 (+ 368us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1209 08:58:54.549897 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1209 08:58:54.549899 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1209 08:58:54.550015 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1209 08:58:54.550545 (+ 530us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1209 08:58:54.550551 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1209 08:58:54.550735 (+ 184us) client_negotiation.cc:769] Sending connection context
1209 08:58:54.552070 (+ 1335us) client_negotiation.cc:240] Negotiation successful
1209 08:58:54.552233 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":931,"mutex_wait_us":46,"spinlock_wait_cycles":37376,"thread_start_us":135,"threads_started":1}
I20251209 08:58:54.553139 28417 heartbeater.cc:344] Connected to a master server at 127.20.49.126:42449
I20251209 08:58:54.553418 28417 heartbeater.cc:461] Registering TS with master...
I20251209 08:58:54.554127 28417 heartbeater.cc:507] Master 127.20.49.126:42449 requested a full tablet report, sending...
I20251209 08:58:54.555285 28419 heartbeater.cc:344] Connected to a master server at 127.20.49.125:44905
I20251209 08:58:54.555431 28419 heartbeater.cc:461] Registering TS with master...
I20251209 08:58:54.555513 27425 ts_manager.cc:194] Re-registered known tserver with Master: c7b2b642c9144d568877db72b8553ac5 (127.20.49.65:38709)
I20251209 08:58:54.555660 28419 heartbeater.cc:507] Master 127.20.49.125:44905 requested a full tablet report, sending...
I20251209 08:58:54.556322 27425 master_service.cc:502] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.20.49.65@KRBTEST.COM'} at 127.20.49.65:41397
I20251209 08:58:54.556854 28030 ts_manager.cc:194] Registered new tserver with Master: c7b2b642c9144d568877db72b8553ac5 (127.20.49.65:38709)
I20251209 08:58:54.557607 20677 external_mini_cluster.cc:1366] Running /tmp/dist-test-task8XuDZV/build/debug/bin/kudu
/tmp/dist-test-task8XuDZV/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765270668420570-20677-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765270668420570-20677-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765270668420570-20677-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765270668420570-20677-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.49.66:35715
--local_ip_for_outbound_sockets=127.20.49.66
--tserver_master_addrs=127.20.49.126:42449
--webserver_port=33885
--webserver_interface=127.20.49.66
--keytab_file=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765270668420570-20677-0/krb5kdc/oryx.keytab
--principal=oryx/127.20.49.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.20.49.84:40253
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.20.49.126:42449,127.20.49.125:44905
--tserver_master_addrs=127.20.49.126:42449,127.20.49.125:44905,127.20.49.124:44451 with env {KRB5CCNAME=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765270668420570-20677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765270668420570-20677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765270668420570-20677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20251209 08:58:54.558655 28421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:58:54.538533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.49.124:44451 (local address 127.20.49.65:38383)
1209 08:58:54.539202 (+ 669us) negotiation.cc:107] Waiting for socket to connect
1209 08:58:54.539256 (+ 54us) client_negotiation.cc:174] Beginning negotiation
1209 08:58:54.540192 (+ 936us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1209 08:58:54.540783 (+ 591us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1209 08:58:54.540791 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
1209 08:58:54.541796 (+ 1005us) client_negotiation.cc:189] Negotiated authn=SASL
1209 08:58:54.542078 (+ 282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:58:54.542085 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:58:54.542807 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1209 08:58:54.542811 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1209 08:58:54.543385 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:58:54.543393 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:58:54.543870 (+ 477us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:58:54.544706 (+ 836us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1209 08:58:54.544721 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
1209 08:58:54.554413 (+ 9692us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1209 08:58:54.557126 (+ 2713us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1209 08:58:54.557129 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1209 08:58:54.557131 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1209 08:58:54.557367 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1209 08:58:54.557618 (+ 251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1209 08:58:54.557621 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1209 08:58:54.557623 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1209 08:58:54.557710 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1209 08:58:54.558150 (+ 440us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1209 08:58:54.558152 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1209 08:58:54.558218 (+ 66us) client_negotiation.cc:769] Sending connection context
1209 08:58:54.558315 (+ 97us) client_negotiation.cc:240] Negotiation successful
1209 08:58:54.558466 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":460,"thread_start_us":121,"threads_started":1}
I20251209 08:58:54.558657 28286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:58:54.538756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.49.65:38383 (local address 127.20.49.124:44451)
1209 08:58:54.539516 (+ 760us) server_negotiation.cc:207] Beginning negotiation
1209 08:58:54.539520 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1209 08:58:54.539535 (+ 15us) server_negotiation.cc:408] Connection header received
1209 08:58:54.540431 (+ 896us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1209 08:58:54.540435 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1209 08:58:54.540494 (+ 59us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1209 08:58:54.540548 (+ 54us) server_negotiation.cc:227] Negotiated authn=SASL
1209 08:58:54.542192 (+ 1644us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:58:54.542685 (+ 493us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1209 08:58:54.543529 (+ 844us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:58:54.543741 (+ 212us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:58:54.554767 (+ 11026us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1209 08:58:54.554789 (+ 22us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1209 08:58:54.554792 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1209 08:58:54.554828 (+ 36us) server_negotiation.cc:923] calling sasl_server_start()
1209 08:58:54.556982 (+ 2154us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1209 08:58:54.557474 (+ 492us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1209 08:58:54.557478 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1209 08:58:54.557480 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1209 08:58:54.557533 (+ 53us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1209 08:58:54.557844 (+ 311us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1209 08:58:54.557849 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1209 08:58:54.557851 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1209 08:58:54.558024 (+ 173us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1209 08:58:54.558089 (+ 65us) server_negotiation.cc:1036] Waiting for connection context
1209 08:58:54.558405 (+ 316us) server_negotiation.cc:300] Negotiation successful
1209 08:58:54.558524 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":674}
I20251209 08:58:54.559406 28418 heartbeater.cc:344] Connected to a master server at 127.20.49.124:44451
I20251209 08:58:54.559512 28418 heartbeater.cc:461] Registering TS with master...
I20251209 08:58:54.559744 28418 heartbeater.cc:507] Master 127.20.49.124:44451 requested a full tablet report, sending...
I20251209 08:58:54.561017 28222 ts_manager.cc:194] Registered new tserver with Master: c7b2b642c9144d568877db72b8553ac5 (127.20.49.65:38709)
W20251209 08:58:54.672338 28424 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251209 08:58:54.672595 28424 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251209 08:58:54.672655 28424 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251209 08:58:54.676095 28424 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251209 08:58:54.676162 28424 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251209 08:58:54.676244 28424 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.49.66
I20251209 08:58:54.680560 28424 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.49.84:40253
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765270668420570-20677-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765270668420570-20677-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765270668420570-20677-0/krb5kdc/oryx.keytab
--principal=oryx/127.20.49.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.49.66:35715
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765270668420570-20677-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.49.66
--webserver_port=33885
--webserver_require_spnego=true
--tserver_master_addrs=127.20.49.126:42449,127.20.49.125:44905,127.20.49.124:44451
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28424
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.49.66
--log_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765270668420570-20677-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 468897755969d686fb08386f42f7835685e7953a
build type DEBUG
built by None at 09 Dec 2025 08:43:15 UTC on 24a791456cd2
build id 9426
I20251209 08:58:54.681650 28424 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251209 08:58:54.682503 28424 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251209 08:58:54.689270 28430 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251209 08:58:54.689263 28432 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251209 08:58:54.689240 28429 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251209 08:58:54.689659 28424 server_base.cc:1047] running on GCE node
I20251209 08:58:54.690048 28424 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251209 08:58:54.690632 28424 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251209 08:58:54.691782 28424 hybrid_clock.cc:648] HybridClock initialized: now 1765270734691753 us; error 39 us; skew 500 ppm
Dec 09 08:58:54 dist-test-slave-8w6c krb5kdc[27379](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765270734, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.49.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251209 08:58:54.694844 28424 init.cc:377] Logged in from keytab as oryx/127.20.49.66@KRBTEST.COM (short username oryx)
I20251209 08:58:54.695997 28424 webserver.cc:492] Webserver started at http://127.20.49.66:33885/ using document root <none> and password file <none>
I20251209 08:58:54.696587 28424 fs_manager.cc:362] Metadata directory not provided
I20251209 08:58:54.696660 28424 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251209 08:58:54.700362 28424 fs_manager.cc:714] Time spent opening directory manager: real 0.002s user 0.000s sys 0.002s
I20251209 08:58:54.702380 28439 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251209 08:58:54.703660 28424 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20251209 08:58:54.703786 28424 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765270668420570-20677-0/minicluster-data/ts-1/data,/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765270668420570-20677-0/minicluster-data/ts-1/wal
uuid: "7182d3e0faba4ceaa1acc9b8b3674cbb"
format_stamp: "Formatted at 2025-12-09 08:58:52 on dist-test-slave-8w6c"
server_key: "271312f522119c7b1a805989f64aed85"
server_key_iv: "397121a43a64c9c935d465b9a1923d49"
server_key_version: "encryptionkey@0"
I20251209 08:58:54.704222 28424 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765270668420570-20677-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765270668420570-20677-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765270668420570-20677-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251209 08:58:54.730628 28424 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251209 08:58:54.733986 28424 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251209 08:58:54.734196 28424 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251209 08:58:54.734805 28424 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251209 08:58:54.735730 28424 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251209 08:58:54.735783 28424 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251209 08:58:54.735844 28424 ts_tablet_manager.cc:616] Registered 0 tablets
I20251209 08:58:54.735878 28424 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251209 08:58:54.746549 28424 rpc_server.cc:307] RPC server started. Bound to: 127.20.49.66:35715
I20251209 08:58:54.746572 28552 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.49.66:35715 every 8 connection(s)
I20251209 08:58:54.747607 28424 server_base.cc:1179] Dumped server information to /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765270668420570-20677-0/minicluster-data/ts-1/data/info.pb
I20251209 08:58:54.755429 20677 external_mini_cluster.cc:1428] Started /tmp/dist-test-task8XuDZV/build/debug/bin/kudu as pid 28424
I20251209 08:58:54.755611 20677 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task8XuDZV/build/debug/bin/kudu with pid 27763
Dec 09 08:58:54 dist-test-slave-8w6c krb5kdc[27379](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765270734, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.49.66@KRBTEST.COM for oryx/127.20.49.126@KRBTEST.COM
Dec 09 08:58:54 dist-test-slave-8w6c krb5kdc[27379](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765270734, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.49.66@KRBTEST.COM for oryx/127.20.49.125@KRBTEST.COM
I20251209 08:58:54.763047 28283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:58:54.750907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.49.66:46575 (local address 127.20.49.125:44905)
1209 08:58:54.751035 (+ 128us) server_negotiation.cc:207] Beginning negotiation
1209 08:58:54.751039 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1209 08:58:54.751702 (+ 663us) server_negotiation.cc:408] Connection header received
1209 08:58:54.751797 (+ 95us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1209 08:58:54.751800 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1209 08:58:54.751842 (+ 42us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1209 08:58:54.751906 (+ 64us) server_negotiation.cc:227] Negotiated authn=SASL
1209 08:58:54.753474 (+ 1568us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:58:54.754219 (+ 745us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1209 08:58:54.755056 (+ 837us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:58:54.755239 (+ 183us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:58:54.759017 (+ 3778us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1209 08:58:54.759036 (+ 19us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1209 08:58:54.759038 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1209 08:58:54.759065 (+ 27us) server_negotiation.cc:923] calling sasl_server_start()
1209 08:58:54.760961 (+ 1896us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1209 08:58:54.761476 (+ 515us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1209 08:58:54.761482 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1209 08:58:54.761484 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1209 08:58:54.761541 (+ 57us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1209 08:58:54.761898 (+ 357us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1209 08:58:54.761905 (+ 7us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1209 08:58:54.761910 (+ 5us) server_negotiation.cc:968] calling sasl_server_step()
1209 08:58:54.762117 (+ 207us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1209 08:58:54.762195 (+ 78us) server_negotiation.cc:1036] Waiting for connection context
1209 08:58:54.762635 (+ 440us) server_negotiation.cc:300] Negotiation successful
1209 08:58:54.762793 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
Dec 09 08:58:54 dist-test-slave-8w6c krb5kdc[27379](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765270734, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.49.66@KRBTEST.COM for oryx/127.20.49.124@KRBTEST.COM
I20251209 08:58:54.763885 28559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:58:54.750717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.49.125:44905 (local address 127.20.49.66:46575)
1209 08:58:54.751617 (+ 900us) negotiation.cc:107] Waiting for socket to connect
1209 08:58:54.751630 (+ 13us) client_negotiation.cc:174] Beginning negotiation
1209 08:58:54.751718 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1209 08:58:54.751947 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1209 08:58:54.751951 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
1209 08:58:54.752746 (+ 795us) client_negotiation.cc:189] Negotiated authn=SASL
1209 08:58:54.753257 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:58:54.753276 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:58:54.754358 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1209 08:58:54.754361 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1209 08:58:54.754931 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:58:54.754941 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:58:54.755051 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:58:54.757481 (+ 2430us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1209 08:58:54.757503 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
1209 08:58:54.758886 (+ 1383us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1209 08:58:54.761102 (+ 2216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1209 08:58:54.761109 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1209 08:58:54.761114 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
1209 08:58:54.761356 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1209 08:58:54.761655 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1209 08:58:54.761662 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1209 08:58:54.761666 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
1209 08:58:54.761767 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1209 08:58:54.762269 (+ 502us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1209 08:58:54.762275 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1209 08:58:54.762366 (+ 91us) client_negotiation.cc:769] Sending connection context
1209 08:58:54.763266 (+ 900us) client_negotiation.cc:240] Negotiation successful
1209 08:58:54.763435 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":828,"thread_start_us":95,"threads_started":1}
I20251209 08:58:54.763971 20677 external_mini_cluster.cc:1366] Running /tmp/dist-test-task8XuDZV/build/debug/bin/kudu
/tmp/dist-test-task8XuDZV/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765270668420570-20677-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765270668420570-20677-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765270668420570-20677-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765270668420570-20677-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.49.67:33177
--local_ip_for_outbound_sockets=127.20.49.67
--tserver_master_addrs=127.20.49.126:42449
--webserver_port=44899
--webserver_interface=127.20.49.67
--keytab_file=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765270668420570-20677-0/krb5kdc/oryx.keytab
--principal=oryx/127.20.49.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.20.49.84:40253
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.20.49.126:42449,127.20.49.125:44905
--tserver_master_addrs=127.20.49.126:42449,127.20.49.125:44905,127.20.49.124:44451 with env {KRB5CCNAME=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765270668420570-20677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765270668420570-20677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765270668420570-20677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20251209 08:58:54.765007 28555 heartbeater.cc:344] Connected to a master server at 127.20.49.125:44905
I20251209 08:58:54.765231 28555 heartbeater.cc:461] Registering TS with master...
I20251209 08:58:54.765897 28555 heartbeater.cc:507] Master 127.20.49.125:44905 requested a full tablet report, sending...
I20251209 08:58:54.766108 28285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:58:54.750056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.49.66:36637 (local address 127.20.49.126:42449)
1209 08:58:54.750175 (+ 119us) server_negotiation.cc:207] Beginning negotiation
1209 08:58:54.750179 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1209 08:58:54.750421 (+ 242us) server_negotiation.cc:408] Connection header received
1209 08:58:54.751423 (+ 1002us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1209 08:58:54.751427 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1209 08:58:54.751484 (+ 57us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1209 08:58:54.751565 (+ 81us) server_negotiation.cc:227] Negotiated authn=SASL
1209 08:58:54.753450 (+ 1885us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:58:54.754037 (+ 587us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1209 08:58:54.754701 (+ 664us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:58:54.754886 (+ 185us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:58:54.758344 (+ 3458us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1209 08:58:54.758371 (+ 27us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1209 08:58:54.758377 (+ 6us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1209 08:58:54.758424 (+ 47us) server_negotiation.cc:923] calling sasl_server_start()
1209 08:58:54.760515 (+ 2091us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1209 08:58:54.761135 (+ 620us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1209 08:58:54.761139 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1209 08:58:54.761142 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
1209 08:58:54.761195 (+ 53us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1209 08:58:54.761509 (+ 314us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1209 08:58:54.761512 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1209 08:58:54.761514 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1209 08:58:54.761778 (+ 264us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1209 08:58:54.761880 (+ 102us) server_negotiation.cc:1036] Waiting for connection context
1209 08:58:54.765806 (+ 3926us) server_negotiation.cc:300] Negotiation successful
1209 08:58:54.765962 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20251209 08:58:54.766352 28558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:58:54.749941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.49.126:42449 (local address 127.20.49.66:36637)
1209 08:58:54.750316 (+ 375us) negotiation.cc:107] Waiting for socket to connect
1209 08:58:54.750329 (+ 13us) client_negotiation.cc:174] Beginning negotiation
1209 08:58:54.751184 (+ 855us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1209 08:58:54.751789 (+ 605us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1209 08:58:54.751802 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
1209 08:58:54.752384 (+ 582us) client_negotiation.cc:189] Negotiated authn=SASL
1209 08:58:54.753258 (+ 874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:58:54.753274 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:58:54.754167 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1209 08:58:54.754170 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1209 08:58:54.754579 (+ 409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:58:54.754585 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:58:54.754808 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:58:54.756494 (+ 1686us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1209 08:58:54.756519 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
1209 08:58:54.758127 (+ 1608us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1209 08:58:54.760664 (+ 2537us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1209 08:58:54.760672 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1209 08:58:54.760689 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
1209 08:58:54.761043 (+ 354us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1209 08:58:54.761284 (+ 241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1209 08:58:54.761288 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1209 08:58:54.761290 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1209 08:58:54.761396 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1209 08:58:54.761941 (+ 545us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1209 08:58:54.761951 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1209 08:58:54.762286 (+ 335us) client_negotiation.cc:769] Sending connection context
1209 08:58:54.762577 (+ 291us) client_negotiation.cc:240] Negotiation successful
1209 08:58:54.762897 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":295,"spinlock_wait_cycles":12672,"thread_start_us":99,"threads_started":1}
I20251209 08:58:54.767326 28030 ts_manager.cc:194] Registered new tserver with Master: 7182d3e0faba4ceaa1acc9b8b3674cbb (127.20.49.66:35715)
I20251209 08:58:54.767361 28553 heartbeater.cc:344] Connected to a master server at 127.20.49.126:42449
I20251209 08:58:54.767457 28553 heartbeater.cc:461] Registering TS with master...
I20251209 08:58:54.767707 28553 heartbeater.cc:507] Master 127.20.49.126:42449 requested a full tablet report, sending...
I20251209 08:58:54.768481 27425 ts_manager.cc:194] Re-registered known tserver with Master: 7182d3e0faba4ceaa1acc9b8b3674cbb (127.20.49.66:35715)
I20251209 08:58:54.769282 27425 master_service.cc:502] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.20.49.66@KRBTEST.COM'} at 127.20.49.66:36637
I20251209 08:58:54.770434 28286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:58:54.749348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.49.66:39023 (local address 127.20.49.124:44451)
1209 08:58:54.749510 (+ 162us) server_negotiation.cc:207] Beginning negotiation
1209 08:58:54.749513 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1209 08:58:54.750310 (+ 797us) server_negotiation.cc:408] Connection header received
1209 08:58:54.751423 (+ 1113us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1209 08:58:54.751427 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1209 08:58:54.751485 (+ 58us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1209 08:58:54.751568 (+ 83us) server_negotiation.cc:227] Negotiated authn=SASL
1209 08:58:54.753765 (+ 2197us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:58:54.754483 (+ 718us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1209 08:58:54.755602 (+ 1119us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:58:54.755834 (+ 232us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:58:54.766043 (+ 10209us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1209 08:58:54.766066 (+ 23us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1209 08:58:54.766069 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1209 08:58:54.766114 (+ 45us) server_negotiation.cc:923] calling sasl_server_start()
1209 08:58:54.768679 (+ 2565us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1209 08:58:54.769162 (+ 483us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1209 08:58:54.769167 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1209 08:58:54.769169 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1209 08:58:54.769224 (+ 55us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1209 08:58:54.769512 (+ 288us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1209 08:58:54.769516 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1209 08:58:54.769517 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1209 08:58:54.769751 (+ 234us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1209 08:58:54.769842 (+ 91us) server_negotiation.cc:1036] Waiting for connection context
1209 08:58:54.770173 (+ 331us) server_negotiation.cc:300] Negotiation successful
1209 08:58:54.770295 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20251209 08:58:54.770435 28557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:58:54.749628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.49.124:44451 (local address 127.20.49.66:39023)
1209 08:58:54.750164 (+ 536us) negotiation.cc:107] Waiting for socket to connect
1209 08:58:54.750211 (+ 47us) client_negotiation.cc:174] Beginning negotiation
1209 08:58:54.751193 (+ 982us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1209 08:58:54.753124 (+ 1931us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1209 08:58:54.753128 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
1209 08:58:54.753295 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
1209 08:58:54.753600 (+ 305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:58:54.753609 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:58:54.754607 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1209 08:58:54.754610 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1209 08:58:54.755415 (+ 805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:58:54.755426 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:58:54.755552 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:58:54.762345 (+ 6793us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1209 08:58:54.762377 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
1209 08:58:54.765830 (+ 3453us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1209 08:58:54.768824 (+ 2994us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1209 08:58:54.768828 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1209 08:58:54.768830 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1209 08:58:54.769047 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1209 08:58:54.769347 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1209 08:58:54.769350 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1209 08:58:54.769352 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1209 08:58:54.769403 (+ 51us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1209 08:58:54.769896 (+ 493us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1209 08:58:54.769899 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1209 08:58:54.769994 (+ 95us) client_negotiation.cc:769] Sending connection context
1209 08:58:54.770084 (+ 90us) client_negotiation.cc:240] Negotiation successful
1209 08:58:54.770240 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":329,"spinlock_wait_cycles":18816,"thread_start_us":135,"threads_started":1}
I20251209 08:58:54.772303 28554 heartbeater.cc:344] Connected to a master server at 127.20.49.124:44451
I20251209 08:58:54.772396 28554 heartbeater.cc:461] Registering TS with master...
I20251209 08:58:54.772626 28554 heartbeater.cc:507] Master 127.20.49.124:44451 requested a full tablet report, sending...
I20251209 08:58:54.773394 28222 ts_manager.cc:194] Registered new tserver with Master: 7182d3e0faba4ceaa1acc9b8b3674cbb (127.20.49.66:35715)
W20251209 08:58:54.879637 28560 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251209 08:58:54.879885 28560 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251209 08:58:54.879967 28560 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251209 08:58:54.883423 28560 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251209 08:58:54.883505 28560 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251209 08:58:54.883627 28560 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.49.67
I20251209 08:58:54.888072 28560 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.49.84:40253
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765270668420570-20677-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765270668420570-20677-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765270668420570-20677-0/krb5kdc/oryx.keytab
--principal=oryx/127.20.49.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.49.67:33177
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765270668420570-20677-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.49.67
--webserver_port=44899
--webserver_require_spnego=true
--tserver_master_addrs=127.20.49.126:42449,127.20.49.125:44905,127.20.49.124:44451
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28560
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.49.67
--log_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765270668420570-20677-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 468897755969d686fb08386f42f7835685e7953a
build type DEBUG
built by None at 09 Dec 2025 08:43:15 UTC on 24a791456cd2
build id 9426
I20251209 08:58:54.889230 28560 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251209 08:58:54.890151 28560 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251209 08:58:54.897224 28565 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251209 08:58:54.897218 28568 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251209 08:58:54.897228 28566 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251209 08:58:54.897661 28560 server_base.cc:1047] running on GCE node
I20251209 08:58:54.898072 28560 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251209 08:58:54.898666 28560 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251209 08:58:54.899860 28560 hybrid_clock.cc:648] HybridClock initialized: now 1765270734899838 us; error 33 us; skew 500 ppm
Dec 09 08:58:54 dist-test-slave-8w6c krb5kdc[27379](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765270734, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.49.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251209 08:58:54.902771 28560 init.cc:377] Logged in from keytab as oryx/127.20.49.67@KRBTEST.COM (short username oryx)
I20251209 08:58:54.903903 28560 webserver.cc:492] Webserver started at http://127.20.49.67:44899/ using document root <none> and password file <none>
I20251209 08:58:54.904444 28560 fs_manager.cc:362] Metadata directory not provided
I20251209 08:58:54.904505 28560 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251209 08:58:54.908197 28560 fs_manager.cc:714] Time spent opening directory manager: real 0.002s user 0.001s sys 0.000s
I20251209 08:58:54.910234 28575 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251209 08:58:54.911288 28560 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20251209 08:58:54.911427 28560 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765270668420570-20677-0/minicluster-data/ts-2/data,/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765270668420570-20677-0/minicluster-data/ts-2/wal
uuid: "9787869327f04a04a23ea007848ae36b"
format_stamp: "Formatted at 2025-12-09 08:58:52 on dist-test-slave-8w6c"
server_key: "041248036dac2a232e81804fbb047a0c"
server_key_iv: "760473bb4c6402d90e6e2f5ce047ae1b"
server_key_version: "encryptionkey@0"
I20251209 08:58:54.911841 28560 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765270668420570-20677-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765270668420570-20677-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765270668420570-20677-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251209 08:58:54.945421 28560 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251209 08:58:54.949004 28560 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251209 08:58:54.949184 28560 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251209 08:58:54.949909 28560 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251209 08:58:54.950922 28560 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251209 08:58:54.950974 28560 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251209 08:58:54.951031 28560 ts_tablet_manager.cc:616] Registered 0 tablets
I20251209 08:58:54.951066 28560 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251209 08:58:54.960355 28560 rpc_server.cc:307] RPC server started. Bound to: 127.20.49.67:33177
I20251209 08:58:54.960393 28688 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.49.67:33177 every 8 connection(s)
I20251209 08:58:54.961416 28560 server_base.cc:1179] Dumped server information to /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765270668420570-20677-0/minicluster-data/ts-2/data/info.pb
Dec 09 08:58:54 dist-test-slave-8w6c krb5kdc[27379](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765270734, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.49.67@KRBTEST.COM for oryx/127.20.49.126@KRBTEST.COM
Dec 09 08:58:54 dist-test-slave-8w6c krb5kdc[27379](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765270734, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.49.67@KRBTEST.COM for oryx/127.20.49.124@KRBTEST.COM
I20251209 08:58:54.971719 20677 external_mini_cluster.cc:1428] Started /tmp/dist-test-task8XuDZV/build/debug/bin/kudu as pid 28560
Dec 09 08:58:54 dist-test-slave-8w6c krb5kdc[27379](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765270734, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.49.67@KRBTEST.COM for oryx/127.20.49.125@KRBTEST.COM
I20251209 08:58:54.976321 28285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:58:54.963490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.49.67:46769 (local address 127.20.49.126:42449)
1209 08:58:54.964006 (+ 516us) server_negotiation.cc:207] Beginning negotiation
1209 08:58:54.964012 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1209 08:58:54.964437 (+ 425us) server_negotiation.cc:408] Connection header received
1209 08:58:54.965623 (+ 1186us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1209 08:58:54.965627 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1209 08:58:54.965723 (+ 96us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1209 08:58:54.965807 (+ 84us) server_negotiation.cc:227] Negotiated authn=SASL
1209 08:58:54.967261 (+ 1454us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:58:54.968073 (+ 812us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1209 08:58:54.968778 (+ 705us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:58:54.968964 (+ 186us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:58:54.971965 (+ 3001us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1209 08:58:54.971988 (+ 23us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1209 08:58:54.971992 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1209 08:58:54.972026 (+ 34us) server_negotiation.cc:923] calling sasl_server_start()
1209 08:58:54.974022 (+ 1996us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1209 08:58:54.974640 (+ 618us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1209 08:58:54.974644 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1209 08:58:54.974646 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1209 08:58:54.974699 (+ 53us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1209 08:58:54.975008 (+ 309us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1209 08:58:54.975011 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1209 08:58:54.975013 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1209 08:58:54.975224 (+ 211us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1209 08:58:54.975355 (+ 131us) server_negotiation.cc:1036] Waiting for connection context
1209 08:58:54.975993 (+ 638us) server_negotiation.cc:300] Negotiation successful
1209 08:58:54.976132 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":395}
I20251209 08:58:54.977084 28283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:58:54.963383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.49.67:55089 (local address 127.20.49.125:44905)
1209 08:58:54.963694 (+ 311us) server_negotiation.cc:207] Beginning negotiation
1209 08:58:54.963699 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1209 08:58:54.964850 (+ 1151us) server_negotiation.cc:408] Connection header received
1209 08:58:54.965789 (+ 939us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1209 08:58:54.965793 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1209 08:58:54.965843 (+ 50us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1209 08:58:54.965914 (+ 71us) server_negotiation.cc:227] Negotiated authn=SASL
1209 08:58:54.967403 (+ 1489us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:58:54.967906 (+ 503us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1209 08:58:54.968828 (+ 922us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:58:54.969023 (+ 195us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:58:54.972350 (+ 3327us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1209 08:58:54.972372 (+ 22us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1209 08:58:54.972376 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1209 08:58:54.972410 (+ 34us) server_negotiation.cc:923] calling sasl_server_start()
1209 08:58:54.974355 (+ 1945us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1209 08:58:54.975187 (+ 832us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1209 08:58:54.975192 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1209 08:58:54.975196 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
1209 08:58:54.975255 (+ 59us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1209 08:58:54.975592 (+ 337us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1209 08:58:54.975597 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1209 08:58:54.975601 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
1209 08:58:54.975819 (+ 218us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1209 08:58:54.975923 (+ 104us) server_negotiation.cc:1036] Waiting for connection context
1209 08:58:54.976777 (+ 854us) server_negotiation.cc:300] Negotiation successful
1209 08:58:54.976897 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":209}
I20251209 08:58:54.977187 28695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:58:54.963668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.49.126:42449 (local address 127.20.49.67:46769)
1209 08:58:54.964279 (+ 611us) negotiation.cc:107] Waiting for socket to connect
1209 08:58:54.964326 (+ 47us) client_negotiation.cc:174] Beginning negotiation
1209 08:58:54.965454 (+ 1128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1209 08:58:54.965994 (+ 540us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1209 08:58:54.966002 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
1209 08:58:54.966498 (+ 496us) client_negotiation.cc:189] Negotiated authn=SASL
1209 08:58:54.967102 (+ 604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:58:54.967115 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:58:54.968233 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1209 08:58:54.968236 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1209 08:58:54.968659 (+ 423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:58:54.968667 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:58:54.968875 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:58:54.969574 (+ 699us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1209 08:58:54.969621 (+ 47us) client_negotiation.cc:656] calling sasl_client_start()
1209 08:58:54.971359 (+ 1738us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1209 08:58:54.974152 (+ 2793us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1209 08:58:54.974158 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1209 08:58:54.974170 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
1209 08:58:54.974517 (+ 347us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1209 08:58:54.974788 (+ 271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1209 08:58:54.974791 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1209 08:58:54.974793 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1209 08:58:54.974919 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1209 08:58:54.975356 (+ 437us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1209 08:58:54.975364 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1209 08:58:54.975707 (+ 343us) client_negotiation.cc:769] Sending connection context
1209 08:58:54.975961 (+ 254us) client_negotiation.cc:240] Negotiation successful
1209 08:58:54.976255 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":407,"thread_start_us":131,"threads_started":1}
I20251209 08:58:54.978480 28689 heartbeater.cc:344] Connected to a master server at 127.20.49.126:42449
I20251209 08:58:54.978531 28286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:58:54.963524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.49.67:46443 (local address 127.20.49.124:44451)
1209 08:58:54.963852 (+ 328us) server_negotiation.cc:207] Beginning negotiation
1209 08:58:54.963858 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1209 08:58:54.964686 (+ 828us) server_negotiation.cc:408] Connection header received
1209 08:58:54.965770 (+ 1084us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1209 08:58:54.965774 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1209 08:58:54.965833 (+ 59us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1209 08:58:54.965914 (+ 81us) server_negotiation.cc:227] Negotiated authn=SASL
1209 08:58:54.967340 (+ 1426us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:58:54.968066 (+ 726us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1209 08:58:54.968846 (+ 780us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:58:54.969032 (+ 186us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:58:54.974199 (+ 5167us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1209 08:58:54.974225 (+ 26us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1209 08:58:54.974231 (+ 6us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1209 08:58:54.974276 (+ 45us) server_negotiation.cc:923] calling sasl_server_start()
1209 08:58:54.976369 (+ 2093us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1209 08:58:54.977141 (+ 772us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1209 08:58:54.977145 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1209 08:58:54.977147 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1209 08:58:54.977200 (+ 53us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1209 08:58:54.977551 (+ 351us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1209 08:58:54.977558 (+ 7us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1209 08:58:54.977563 (+ 5us) server_negotiation.cc:968] calling sasl_server_step()
1209 08:58:54.977798 (+ 235us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1209 08:58:54.977896 (+ 98us) server_negotiation.cc:1036] Waiting for connection context
1209 08:58:54.978239 (+ 343us) server_negotiation.cc:300] Negotiation successful
1209 08:58:54.978382 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":212}
I20251209 08:58:54.978750 28689 heartbeater.cc:461] Registering TS with master...
I20251209 08:58:54.979008 28694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:58:54.963668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.49.124:44451 (local address 127.20.49.67:46443)
1209 08:58:54.964574 (+ 906us) negotiation.cc:107] Waiting for socket to connect
1209 08:58:54.964587 (+ 13us) client_negotiation.cc:174] Beginning negotiation
1209 08:58:54.965410 (+ 823us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1209 08:58:54.966030 (+ 620us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1209 08:58:54.966034 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
1209 08:58:54.966518 (+ 484us) client_negotiation.cc:189] Negotiated authn=SASL
1209 08:58:54.967142 (+ 624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:58:54.967150 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:58:54.968174 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1209 08:58:54.968177 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1209 08:58:54.968719 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:58:54.968730 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:58:54.969156 (+ 426us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:58:54.970018 (+ 862us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1209 08:58:54.970036 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
1209 08:58:54.974009 (+ 3973us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1209 08:58:54.976748 (+ 2739us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1209 08:58:54.976752 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1209 08:58:54.976754 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1209 08:58:54.977011 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1209 08:58:54.977337 (+ 326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1209 08:58:54.977342 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1209 08:58:54.977346 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
1209 08:58:54.977403 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1209 08:58:54.977911 (+ 508us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1209 08:58:54.977915 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1209 08:58:54.977997 (+ 82us) client_negotiation.cc:769] Sending connection context
1209 08:58:54.978707 (+ 710us) client_negotiation.cc:240] Negotiation successful
1209 08:58:54.978848 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":814,"mutex_wait_us":62,"spinlock_wait_cycles":62208,"thread_start_us":148,"threads_started":1}
I20251209 08:58:54.979406 28689 heartbeater.cc:507] Master 127.20.49.126:42449 requested a full tablet report, sending...
I20251209 08:58:54.979696 28690 heartbeater.cc:344] Connected to a master server at 127.20.49.124:44451
I20251209 08:58:54.979770 28690 heartbeater.cc:461] Registering TS with master...
I20251209 08:58:54.979954 28690 heartbeater.cc:507] Master 127.20.49.124:44451 requested a full tablet report, sending...
I20251209 08:58:54.980556 27425 ts_manager.cc:194] Re-registered known tserver with Master: 9787869327f04a04a23ea007848ae36b (127.20.49.67:33177)
I20251209 08:58:54.980659 28222 ts_manager.cc:194] Registered new tserver with Master: 9787869327f04a04a23ea007848ae36b (127.20.49.67:33177)
I20251209 08:58:54.981209 27425 master_service.cc:502] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.20.49.67@KRBTEST.COM'} at 127.20.49.67:46769
I20251209 08:58:54.987191 28701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:58:54.974292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58576 (local address 127.20.49.126:42449)
1209 08:58:54.975386 (+ 1094us) server_negotiation.cc:207] Beginning negotiation
1209 08:58:54.975389 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1209 08:58:54.975400 (+ 11us) server_negotiation.cc:408] Connection header received
1209 08:58:54.975579 (+ 179us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1209 08:58:54.975582 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1209 08:58:54.975627 (+ 45us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1209 08:58:54.975711 (+ 84us) server_negotiation.cc:227] Negotiated authn=SASL
1209 08:58:54.977791 (+ 2080us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:58:54.978742 (+ 951us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1209 08:58:54.981521 (+ 2779us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:58:54.981785 (+ 264us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:58:54.983246 (+ 1461us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1209 08:58:54.983263 (+ 17us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1209 08:58:54.983267 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1209 08:58:54.983292 (+ 25us) server_negotiation.cc:923] calling sasl_server_start()
1209 08:58:54.985129 (+ 1837us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1209 08:58:54.985763 (+ 634us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1209 08:58:54.985767 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1209 08:58:54.985769 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1209 08:58:54.985818 (+ 49us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1209 08:58:54.986066 (+ 248us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1209 08:58:54.986068 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1209 08:58:54.986071 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
1209 08:58:54.986233 (+ 162us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1209 08:58:54.986820 (+ 587us) server_negotiation.cc:1036] Waiting for connection context
1209 08:58:54.986894 (+ 74us) server_negotiation.cc:300] Negotiation successful
1209 08:58:54.987019 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":193,"thread_start_us":84,"threads_started":1}
I20251209 08:58:54.987867 28693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:58:54.963668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.49.125:44905 (local address 127.20.49.67:55089)
1209 08:58:54.964737 (+ 1069us) negotiation.cc:107] Waiting for socket to connect
1209 08:58:54.964749 (+ 12us) client_negotiation.cc:174] Beginning negotiation
1209 08:58:54.965399 (+ 650us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1209 08:58:54.966030 (+ 631us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1209 08:58:54.966034 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
1209 08:58:54.966990 (+ 956us) client_negotiation.cc:189] Negotiated authn=SASL
1209 08:58:54.967221 (+ 231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:58:54.967228 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:58:54.968030 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1209 08:58:54.968034 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1209 08:58:54.968663 (+ 629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:58:54.968673 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:58:54.969158 (+ 485us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:58:54.970053 (+ 895us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1209 08:58:54.970065 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
1209 08:58:54.972177 (+ 2112us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1209 08:58:54.974498 (+ 2321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1209 08:58:54.974504 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1209 08:58:54.974507 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
1209 08:58:54.975050 (+ 543us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1209 08:58:54.975383 (+ 333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1209 08:58:54.975387 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1209 08:58:54.975391 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
1209 08:58:54.975461 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1209 08:58:54.975970 (+ 509us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1209 08:58:54.975973 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1209 08:58:54.976060 (+ 87us) client_negotiation.cc:769] Sending connection context
1209 08:58:54.976187 (+ 127us) client_negotiation.cc:240] Negotiation successful
1209 08:58:54.976327 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":988,"mutex_wait_us":185,"spinlock_wait_cycles":19328,"thread_start_us":113,"threads_started":1}
I20251209 08:58:54.988646 28691 heartbeater.cc:344] Connected to a master server at 127.20.49.125:44905
I20251209 08:58:54.988758 28691 heartbeater.cc:461] Registering TS with master...
I20251209 08:58:54.989010 28691 heartbeater.cc:507] Master 127.20.49.125:44905 requested a full tablet report, sending...
I20251209 08:58:54.989893 28030 ts_manager.cc:194] Registered new tserver with Master: 9787869327f04a04a23ea007848ae36b (127.20.49.67:33177)
I20251209 08:58:54.989949 28283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:58:54.974789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36974 (local address 127.20.49.125:44905)
1209 08:58:54.977295 (+ 2506us) server_negotiation.cc:207] Beginning negotiation
1209 08:58:54.977302 (+ 7us) server_negotiation.cc:400] Waiting for connection header
1209 08:58:54.982246 (+ 4944us) server_negotiation.cc:408] Connection header received
1209 08:58:54.982447 (+ 201us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1209 08:58:54.982451 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1209 08:58:54.982505 (+ 54us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1209 08:58:54.982559 (+ 54us) server_negotiation.cc:227] Negotiated authn=SASL
1209 08:58:54.983496 (+ 937us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:58:54.983976 (+ 480us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1209 08:58:54.984775 (+ 799us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:58:54.985005 (+ 230us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:58:54.986263 (+ 1258us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1209 08:58:54.986281 (+ 18us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1209 08:58:54.986283 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1209 08:58:54.986308 (+ 25us) server_negotiation.cc:923] calling sasl_server_start()
1209 08:58:54.988268 (+ 1960us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1209 08:58:54.988783 (+ 515us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1209 08:58:54.988787 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1209 08:58:54.988788 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1209 08:58:54.988842 (+ 54us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1209 08:58:54.989132 (+ 290us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1209 08:58:54.989135 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1209 08:58:54.989137 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1209 08:58:54.989280 (+ 143us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1209 08:58:54.989404 (+ 124us) server_negotiation.cc:1036] Waiting for connection context
1209 08:58:54.989622 (+ 218us) server_negotiation.cc:300] Negotiation successful
1209 08:58:54.989764 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2416,"thread_start_us":76,"threads_started":1}
I20251209 08:58:54.991195 27425 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58576:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251209 08:58:54.994108 28286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:58:54.988706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56822 (local address 127.20.49.124:44451)
1209 08:58:54.988839 (+ 133us) server_negotiation.cc:207] Beginning negotiation
1209 08:58:54.988843 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1209 08:58:54.990469 (+ 1626us) server_negotiation.cc:408] Connection header received
1209 08:58:54.990641 (+ 172us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1209 08:58:54.990645 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1209 08:58:54.990872 (+ 227us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1209 08:58:54.990999 (+ 127us) server_negotiation.cc:227] Negotiated authn=TOKEN
1209 08:58:54.991338 (+ 339us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:58:54.992063 (+ 725us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1209 08:58:54.992783 (+ 720us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:58:54.992981 (+ 198us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:58:54.993048 (+ 67us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1209 08:58:54.993651 (+ 603us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1209 08:58:54.993821 (+ 170us) server_negotiation.cc:1036] Waiting for connection context
1209 08:58:54.993938 (+ 117us) server_negotiation.cc:300] Negotiation successful
1209 08:58:54.993990 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
W20251209 08:58:54.994436 27425 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251209 08:58:55.012138 28712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:58:55.007728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.49.66:35715 (local address 127.0.0.1:49872)
1209 08:58:55.008020 (+ 292us) negotiation.cc:107] Waiting for socket to connect
1209 08:58:55.008037 (+ 17us) client_negotiation.cc:174] Beginning negotiation
1209 08:58:55.008134 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1209 08:58:55.009214 (+ 1080us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1209 08:58:55.009217 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1209 08:58:55.009238 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1209 08:58:55.009536 (+ 298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:58:55.009547 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:58:55.010780 (+ 1233us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1209 08:58:55.010784 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1209 08:58:55.011746 (+ 962us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:58:55.011756 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:58:55.011844 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:58:55.011867 (+ 23us) client_negotiation.cc:769] Sending connection context
1209 08:58:55.011913 (+ 46us) client_negotiation.cc:240] Negotiation successful
1209 08:58:55.011987 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":133,"threads_started":1}
I20251209 08:58:55.012322 28713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:58:55.008093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.49.65:38709 (local address 127.0.0.1:43026)
1209 08:58:55.008921 (+ 828us) negotiation.cc:107] Waiting for socket to connect
1209 08:58:55.008935 (+ 14us) client_negotiation.cc:174] Beginning negotiation
1209 08:58:55.009042 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1209 08:58:55.009883 (+ 841us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1209 08:58:55.009886 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1209 08:58:55.009904 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1209 08:58:55.010172 (+ 268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:58:55.010182 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:58:55.011333 (+ 1151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1209 08:58:55.011337 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1209 08:58:55.011917 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:58:55.011925 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:58:55.012034 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:58:55.012056 (+ 22us) client_negotiation.cc:769] Sending connection context
1209 08:58:55.012119 (+ 63us) client_negotiation.cc:240] Negotiation successful
1209 08:58:55.012180 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":742,"thread_start_us":98,"threads_started":1}
I20251209 08:58:55.013118 28717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:58:55.008862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43026 (local address 127.20.49.65:38709)
1209 08:58:55.009234 (+ 372us) server_negotiation.cc:207] Beginning negotiation
1209 08:58:55.009241 (+ 7us) server_negotiation.cc:400] Waiting for connection header
1209 08:58:55.009262 (+ 21us) server_negotiation.cc:408] Connection header received
1209 08:58:55.009396 (+ 134us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1209 08:58:55.009400 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1209 08:58:55.009540 (+ 140us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1209 08:58:55.009720 (+ 180us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1209 08:58:55.010290 (+ 570us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:58:55.011218 (+ 928us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1209 08:58:55.012065 (+ 847us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:58:55.012793 (+ 728us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:58:55.012891 (+ 98us) server_negotiation.cc:1036] Waiting for connection context
1209 08:58:55.012947 (+ 56us) server_negotiation.cc:300] Negotiation successful
1209 08:58:55.013011 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":223,"thread_start_us":105,"threads_started":1}
I20251209 08:58:55.013458 28714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:58:55.008375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49872 (local address 127.20.49.66:35715)
1209 08:58:55.008718 (+ 343us) server_negotiation.cc:207] Beginning negotiation
1209 08:58:55.008726 (+ 8us) server_negotiation.cc:400] Waiting for connection header
1209 08:58:55.008751 (+ 25us) server_negotiation.cc:408] Connection header received
1209 08:58:55.008847 (+ 96us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1209 08:58:55.008853 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
1209 08:58:55.009066 (+ 213us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1209 08:58:55.009226 (+ 160us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1209 08:58:55.009702 (+ 476us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:58:55.010654 (+ 952us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1209 08:58:55.012440 (+ 1786us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:58:55.013001 (+ 561us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:58:55.013110 (+ 109us) server_negotiation.cc:1036] Waiting for connection context
1209 08:58:55.013189 (+ 79us) server_negotiation.cc:300] Negotiation successful
1209 08:58:55.013285 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":232,"thread_start_us":107,"threads_started":1}
I20251209 08:58:55.013533 28715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:58:55.008423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.49.67:33177 (local address 127.0.0.1:55022)
1209 08:58:55.009489 (+ 1066us) negotiation.cc:107] Waiting for socket to connect
1209 08:58:55.009502 (+ 13us) client_negotiation.cc:174] Beginning negotiation
1209 08:58:55.009605 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1209 08:58:55.010099 (+ 494us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1209 08:58:55.010102 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1209 08:58:55.010129 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1209 08:58:55.010530 (+ 401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:58:55.010541 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:58:55.011994 (+ 1453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1209 08:58:55.011996 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1209 08:58:55.013154 (+ 1158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:58:55.013164 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:58:55.013272 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:58:55.013287 (+ 15us) client_negotiation.cc:769] Sending connection context
1209 08:58:55.013337 (+ 50us) client_negotiation.cc:240] Negotiation successful
1209 08:58:55.013392 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":999,"thread_start_us":92,"threads_started":1}
I20251209 08:58:55.014714 28716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:58:55.008656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55022 (local address 127.20.49.67:33177)
1209 08:58:55.009293 (+ 637us) server_negotiation.cc:207] Beginning negotiation
1209 08:58:55.009299 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1209 08:58:55.009599 (+ 300us) server_negotiation.cc:408] Connection header received
1209 08:58:55.009776 (+ 177us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1209 08:58:55.009781 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1209 08:58:55.009949 (+ 168us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1209 08:58:55.010096 (+ 147us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1209 08:58:55.010693 (+ 597us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:58:55.011865 (+ 1172us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1209 08:58:55.013789 (+ 1924us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:58:55.014362 (+ 573us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:58:55.014456 (+ 94us) server_negotiation.cc:1036] Waiting for connection context
1209 08:58:55.014513 (+ 57us) server_negotiation.cc:300] Negotiation successful
1209 08:58:55.014606 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":530,"thread_start_us":107,"threads_started":1}
I20251209 08:58:55.016775 28351 tablet_service.cc:1505] Processing CreateTablet for tablet 6774c15d83e043ab9b2b2e0718bca5b9 (DEFAULT_TABLE table=test-table [id=a4ed318cde3e45c79d14c1485d489eb9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251209 08:58:55.016958 28623 tablet_service.cc:1505] Processing CreateTablet for tablet 6774c15d83e043ab9b2b2e0718bca5b9 (DEFAULT_TABLE table=test-table [id=a4ed318cde3e45c79d14c1485d489eb9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251209 08:58:55.017809 28623 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6774c15d83e043ab9b2b2e0718bca5b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251209 08:58:55.019593 28487 tablet_service.cc:1505] Processing CreateTablet for tablet 6774c15d83e043ab9b2b2e0718bca5b9 (DEFAULT_TABLE table=test-table [id=a4ed318cde3e45c79d14c1485d489eb9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251209 08:58:55.020417 28487 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6774c15d83e043ab9b2b2e0718bca5b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251209 08:58:55.021791 28351 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6774c15d83e043ab9b2b2e0718bca5b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251209 08:58:55.023963 28718 tablet_bootstrap.cc:492] T 6774c15d83e043ab9b2b2e0718bca5b9 P 9787869327f04a04a23ea007848ae36b: Bootstrap starting.
I20251209 08:58:55.026448 28718 tablet_bootstrap.cc:654] T 6774c15d83e043ab9b2b2e0718bca5b9 P 9787869327f04a04a23ea007848ae36b: Neither blocks nor log segments found. Creating new log.
I20251209 08:58:55.027246 28718 log.cc:826] T 6774c15d83e043ab9b2b2e0718bca5b9 P 9787869327f04a04a23ea007848ae36b: Log is configured to *not* fsync() on all Append() calls
I20251209 08:58:55.029059 28720 tablet_bootstrap.cc:492] T 6774c15d83e043ab9b2b2e0718bca5b9 P 7182d3e0faba4ceaa1acc9b8b3674cbb: Bootstrap starting.
I20251209 08:58:55.029299 28721 tablet_bootstrap.cc:492] T 6774c15d83e043ab9b2b2e0718bca5b9 P c7b2b642c9144d568877db72b8553ac5: Bootstrap starting.
I20251209 08:58:55.029651 28718 tablet_bootstrap.cc:492] T 6774c15d83e043ab9b2b2e0718bca5b9 P 9787869327f04a04a23ea007848ae36b: No bootstrap required, opened a new log
I20251209 08:58:55.029881 28718 ts_tablet_manager.cc:1403] T 6774c15d83e043ab9b2b2e0718bca5b9 P 9787869327f04a04a23ea007848ae36b: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20251209 08:58:55.031252 28720 tablet_bootstrap.cc:654] T 6774c15d83e043ab9b2b2e0718bca5b9 P 7182d3e0faba4ceaa1acc9b8b3674cbb: Neither blocks nor log segments found. Creating new log.
I20251209 08:58:55.031508 28721 tablet_bootstrap.cc:654] T 6774c15d83e043ab9b2b2e0718bca5b9 P c7b2b642c9144d568877db72b8553ac5: Neither blocks nor log segments found. Creating new log.
I20251209 08:58:55.031982 28720 log.cc:826] T 6774c15d83e043ab9b2b2e0718bca5b9 P 7182d3e0faba4ceaa1acc9b8b3674cbb: Log is configured to *not* fsync() on all Append() calls
I20251209 08:58:55.032292 28721 log.cc:826] T 6774c15d83e043ab9b2b2e0718bca5b9 P c7b2b642c9144d568877db72b8553ac5: Log is configured to *not* fsync() on all Append() calls
I20251209 08:58:55.033185 28718 raft_consensus.cc:359] T 6774c15d83e043ab9b2b2e0718bca5b9 P 9787869327f04a04a23ea007848ae36b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7182d3e0faba4ceaa1acc9b8b3674cbb" member_type: VOTER last_known_addr { host: "127.20.49.66" port: 35715 } } peers { permanent_uuid: "c7b2b642c9144d568877db72b8553ac5" member_type: VOTER last_known_addr { host: "127.20.49.65" port: 38709 } } peers { permanent_uuid: "9787869327f04a04a23ea007848ae36b" member_type: VOTER last_known_addr { host: "127.20.49.67" port: 33177 } }
I20251209 08:58:55.033421 28718 raft_consensus.cc:385] T 6774c15d83e043ab9b2b2e0718bca5b9 P 9787869327f04a04a23ea007848ae36b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251209 08:58:55.033479 28718 raft_consensus.cc:740] T 6774c15d83e043ab9b2b2e0718bca5b9 P 9787869327f04a04a23ea007848ae36b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9787869327f04a04a23ea007848ae36b, State: Initialized, Role: FOLLOWER
I20251209 08:58:55.033977 28721 tablet_bootstrap.cc:492] T 6774c15d83e043ab9b2b2e0718bca5b9 P c7b2b642c9144d568877db72b8553ac5: No bootstrap required, opened a new log
I20251209 08:58:55.033991 28720 tablet_bootstrap.cc:492] T 6774c15d83e043ab9b2b2e0718bca5b9 P 7182d3e0faba4ceaa1acc9b8b3674cbb: No bootstrap required, opened a new log
I20251209 08:58:55.033954 28718 consensus_queue.cc:260] T 6774c15d83e043ab9b2b2e0718bca5b9 P 9787869327f04a04a23ea007848ae36b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7182d3e0faba4ceaa1acc9b8b3674cbb" member_type: VOTER last_known_addr { host: "127.20.49.66" port: 35715 } } peers { permanent_uuid: "c7b2b642c9144d568877db72b8553ac5" member_type: VOTER last_known_addr { host: "127.20.49.65" port: 38709 } } peers { permanent_uuid: "9787869327f04a04a23ea007848ae36b" member_type: VOTER last_known_addr { host: "127.20.49.67" port: 33177 } }
I20251209 08:58:55.034168 28721 ts_tablet_manager.cc:1403] T 6774c15d83e043ab9b2b2e0718bca5b9 P c7b2b642c9144d568877db72b8553ac5: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20251209 08:58:55.034168 28720 ts_tablet_manager.cc:1403] T 6774c15d83e043ab9b2b2e0718bca5b9 P 7182d3e0faba4ceaa1acc9b8b3674cbb: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.001s
I20251209 08:58:55.034627 28689 heartbeater.cc:499] Master 127.20.49.126:42449 was elected leader, sending a full tablet report...
I20251209 08:58:55.034848 28718 ts_tablet_manager.cc:1434] T 6774c15d83e043ab9b2b2e0718bca5b9 P 9787869327f04a04a23ea007848ae36b: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20251209 08:58:55.037257 28720 raft_consensus.cc:359] T 6774c15d83e043ab9b2b2e0718bca5b9 P 7182d3e0faba4ceaa1acc9b8b3674cbb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7182d3e0faba4ceaa1acc9b8b3674cbb" member_type: VOTER last_known_addr { host: "127.20.49.66" port: 35715 } } peers { permanent_uuid: "c7b2b642c9144d568877db72b8553ac5" member_type: VOTER last_known_addr { host: "127.20.49.65" port: 38709 } } peers { permanent_uuid: "9787869327f04a04a23ea007848ae36b" member_type: VOTER last_known_addr { host: "127.20.49.67" port: 33177 } }
I20251209 08:58:55.037259 28721 raft_consensus.cc:359] T 6774c15d83e043ab9b2b2e0718bca5b9 P c7b2b642c9144d568877db72b8553ac5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7182d3e0faba4ceaa1acc9b8b3674cbb" member_type: VOTER last_known_addr { host: "127.20.49.66" port: 35715 } } peers { permanent_uuid: "c7b2b642c9144d568877db72b8553ac5" member_type: VOTER last_known_addr { host: "127.20.49.65" port: 38709 } } peers { permanent_uuid: "9787869327f04a04a23ea007848ae36b" member_type: VOTER last_known_addr { host: "127.20.49.67" port: 33177 } }
I20251209 08:58:55.037475 28720 raft_consensus.cc:385] T 6774c15d83e043ab9b2b2e0718bca5b9 P 7182d3e0faba4ceaa1acc9b8b3674cbb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251209 08:58:55.037473 28721 raft_consensus.cc:385] T 6774c15d83e043ab9b2b2e0718bca5b9 P c7b2b642c9144d568877db72b8553ac5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251209 08:58:55.037531 28720 raft_consensus.cc:740] T 6774c15d83e043ab9b2b2e0718bca5b9 P 7182d3e0faba4ceaa1acc9b8b3674cbb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7182d3e0faba4ceaa1acc9b8b3674cbb, State: Initialized, Role: FOLLOWER
I20251209 08:58:55.037539 28721 raft_consensus.cc:740] T 6774c15d83e043ab9b2b2e0718bca5b9 P c7b2b642c9144d568877db72b8553ac5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c7b2b642c9144d568877db72b8553ac5, State: Initialized, Role: FOLLOWER
I20251209 08:58:55.038053 28721 consensus_queue.cc:260] T 6774c15d83e043ab9b2b2e0718bca5b9 P c7b2b642c9144d568877db72b8553ac5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7182d3e0faba4ceaa1acc9b8b3674cbb" member_type: VOTER last_known_addr { host: "127.20.49.66" port: 35715 } } peers { permanent_uuid: "c7b2b642c9144d568877db72b8553ac5" member_type: VOTER last_known_addr { host: "127.20.49.65" port: 38709 } } peers { permanent_uuid: "9787869327f04a04a23ea007848ae36b" member_type: VOTER last_known_addr { host: "127.20.49.67" port: 33177 } }
I20251209 08:58:55.038053 28720 consensus_queue.cc:260] T 6774c15d83e043ab9b2b2e0718bca5b9 P 7182d3e0faba4ceaa1acc9b8b3674cbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7182d3e0faba4ceaa1acc9b8b3674cbb" member_type: VOTER last_known_addr { host: "127.20.49.66" port: 35715 } } peers { permanent_uuid: "c7b2b642c9144d568877db72b8553ac5" member_type: VOTER last_known_addr { host: "127.20.49.65" port: 38709 } } peers { permanent_uuid: "9787869327f04a04a23ea007848ae36b" member_type: VOTER last_known_addr { host: "127.20.49.67" port: 33177 } }
I20251209 08:58:55.039000 28720 ts_tablet_manager.cc:1434] T 6774c15d83e043ab9b2b2e0718bca5b9 P 7182d3e0faba4ceaa1acc9b8b3674cbb: Time spent starting tablet: real 0.005s user 0.003s sys 0.002s
I20251209 08:58:55.039000 28721 ts_tablet_manager.cc:1434] T 6774c15d83e043ab9b2b2e0718bca5b9 P c7b2b642c9144d568877db72b8553ac5: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20251209 08:58:55.039290 28553 heartbeater.cc:499] Master 127.20.49.126:42449 was elected leader, sending a full tablet report...
I20251209 08:58:55.039299 28417 heartbeater.cc:499] Master 127.20.49.126:42449 was elected leader, sending a full tablet report...
W20251209 08:58:55.040359 28420 tablet.cc:2378] T 6774c15d83e043ab9b2b2e0718bca5b9 P c7b2b642c9144d568877db72b8553ac5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20251209 08:58:55.040356 28725 raft_consensus.cc:493] T 6774c15d83e043ab9b2b2e0718bca5b9 P 9787869327f04a04a23ea007848ae36b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251209 08:58:55.040496 28725 raft_consensus.cc:515] T 6774c15d83e043ab9b2b2e0718bca5b9 P 9787869327f04a04a23ea007848ae36b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7182d3e0faba4ceaa1acc9b8b3674cbb" member_type: VOTER last_known_addr { host: "127.20.49.66" port: 35715 } } peers { permanent_uuid: "c7b2b642c9144d568877db72b8553ac5" member_type: VOTER last_known_addr { host: "127.20.49.65" port: 38709 } } peers { permanent_uuid: "9787869327f04a04a23ea007848ae36b" member_type: VOTER last_known_addr { host: "127.20.49.67" port: 33177 } }
I20251209 08:58:55.041729 28725 leader_election.cc:290] T 6774c15d83e043ab9b2b2e0718bca5b9 P 9787869327f04a04a23ea007848ae36b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7182d3e0faba4ceaa1acc9b8b3674cbb (127.20.49.66:35715), c7b2b642c9144d568877db72b8553ac5 (127.20.49.65:38709)
I20251209 08:58:55.045509 28694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:58:55.041971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.49.66:35715 (local address 127.20.49.67:58221)
1209 08:58:55.042327 (+ 356us) negotiation.cc:107] Waiting for socket to connect
1209 08:58:55.042342 (+ 15us) client_negotiation.cc:174] Beginning negotiation
1209 08:58:55.042441 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1209 08:58:55.042788 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1209 08:58:55.042791 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1209 08:58:55.042811 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1209 08:58:55.043066 (+ 255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:58:55.043080 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:58:55.044208 (+ 1128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1209 08:58:55.044212 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1209 08:58:55.045083 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:58:55.045095 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:58:55.045213 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:58:55.045238 (+ 25us) client_negotiation.cc:769] Sending connection context
1209 08:58:55.045305 (+ 67us) client_negotiation.cc:240] Negotiation successful
1209 08:58:55.045386 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279}
I20251209 08:58:55.046731 28714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:58:55.042043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.49.67:58221 (local address 127.20.49.66:35715)
1209 08:58:55.042197 (+ 154us) server_negotiation.cc:207] Beginning negotiation
1209 08:58:55.042201 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1209 08:58:55.042415 (+ 214us) server_negotiation.cc:408] Connection header received
1209 08:58:55.042534 (+ 119us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1209 08:58:55.042540 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
1209 08:58:55.042588 (+ 48us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1209 08:58:55.042669 (+ 81us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1209 08:58:55.043204 (+ 535us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:58:55.043944 (+ 740us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1209 08:58:55.045913 (+ 1969us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:58:55.046441 (+ 528us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:58:55.046479 (+ 38us) server_negotiation.cc:1036] Waiting for connection context
1209 08:58:55.046531 (+ 52us) server_negotiation.cc:300] Negotiation successful
1209 08:58:55.046598 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20251209 08:58:55.048252 28693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:58:55.042225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.49.65:38709 (local address 127.20.49.67:54885)
1209 08:58:55.042391 (+ 166us) negotiation.cc:107] Waiting for socket to connect
1209 08:58:55.042403 (+ 12us) client_negotiation.cc:174] Beginning negotiation
1209 08:58:55.042452 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1209 08:58:55.044758 (+ 2306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1209 08:58:55.044762 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
1209 08:58:55.044784 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1209 08:58:55.045017 (+ 233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:58:55.045026 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:58:55.046982 (+ 1956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1209 08:58:55.046986 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1209 08:58:55.047903 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:58:55.047914 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:58:55.048050 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:58:55.048070 (+ 20us) client_negotiation.cc:769] Sending connection context
1209 08:58:55.048116 (+ 46us) client_negotiation.cc:240] Negotiation successful
1209 08:58:55.048165 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":90}
I20251209 08:58:55.048735 28717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:58:55.042652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.49.67:54885 (local address 127.20.49.65:38709)
1209 08:58:55.042762 (+ 110us) server_negotiation.cc:207] Beginning negotiation
1209 08:58:55.042766 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1209 08:58:55.042779 (+ 13us) server_negotiation.cc:408] Connection header received
1209 08:58:55.042820 (+ 41us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1209 08:58:55.042823 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1209 08:58:55.042871 (+ 48us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1209 08:58:55.042945 (+ 74us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1209 08:58:55.045891 (+ 2946us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:58:55.046588 (+ 697us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1209 08:58:55.048056 (+ 1468us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:58:55.048500 (+ 444us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:58:55.048533 (+ 33us) server_negotiation.cc:1036] Waiting for connection context
1209 08:58:55.048577 (+ 44us) server_negotiation.cc:300] Negotiation successful
1209 08:58:55.048622 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20251209 08:58:55.049144 28507 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6774c15d83e043ab9b2b2e0718bca5b9" candidate_uuid: "9787869327f04a04a23ea007848ae36b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7182d3e0faba4ceaa1acc9b8b3674cbb" is_pre_election: true
I20251209 08:58:55.049327 28370 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6774c15d83e043ab9b2b2e0718bca5b9" candidate_uuid: "9787869327f04a04a23ea007848ae36b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c7b2b642c9144d568877db72b8553ac5" is_pre_election: true
I20251209 08:58:55.049465 28507 raft_consensus.cc:2468] T 6774c15d83e043ab9b2b2e0718bca5b9 P 7182d3e0faba4ceaa1acc9b8b3674cbb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9787869327f04a04a23ea007848ae36b in term 0.
I20251209 08:58:55.049723 28370 raft_consensus.cc:2468] T 6774c15d83e043ab9b2b2e0718bca5b9 P c7b2b642c9144d568877db72b8553ac5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9787869327f04a04a23ea007848ae36b in term 0.
I20251209 08:58:55.050117 28579 leader_election.cc:304] T 6774c15d83e043ab9b2b2e0718bca5b9 P 9787869327f04a04a23ea007848ae36b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7182d3e0faba4ceaa1acc9b8b3674cbb, 9787869327f04a04a23ea007848ae36b; no voters:
I20251209 08:58:55.050359 28724 raft_consensus.cc:2804] T 6774c15d83e043ab9b2b2e0718bca5b9 P 9787869327f04a04a23ea007848ae36b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251209 08:58:55.050443 28724 raft_consensus.cc:493] T 6774c15d83e043ab9b2b2e0718bca5b9 P 9787869327f04a04a23ea007848ae36b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251209 08:58:55.050485 28724 raft_consensus.cc:3060] T 6774c15d83e043ab9b2b2e0718bca5b9 P 9787869327f04a04a23ea007848ae36b [term 0 FOLLOWER]: Advancing to term 1
I20251209 08:58:55.051514 28724 raft_consensus.cc:515] T 6774c15d83e043ab9b2b2e0718bca5b9 P 9787869327f04a04a23ea007848ae36b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7182d3e0faba4ceaa1acc9b8b3674cbb" member_type: VOTER last_known_addr { host: "127.20.49.66" port: 35715 } } peers { permanent_uuid: "c7b2b642c9144d568877db72b8553ac5" member_type: VOTER last_known_addr { host: "127.20.49.65" port: 38709 } } peers { permanent_uuid: "9787869327f04a04a23ea007848ae36b" member_type: VOTER last_known_addr { host: "127.20.49.67" port: 33177 } }
I20251209 08:58:55.051847 28724 leader_election.cc:290] T 6774c15d83e043ab9b2b2e0718bca5b9 P 9787869327f04a04a23ea007848ae36b [CANDIDATE]: Term 1 election: Requested vote from peers 7182d3e0faba4ceaa1acc9b8b3674cbb (127.20.49.66:35715), c7b2b642c9144d568877db72b8553ac5 (127.20.49.65:38709)
I20251209 08:58:55.052199 28507 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6774c15d83e043ab9b2b2e0718bca5b9" candidate_uuid: "9787869327f04a04a23ea007848ae36b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7182d3e0faba4ceaa1acc9b8b3674cbb"
I20251209 08:58:55.052232 28370 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6774c15d83e043ab9b2b2e0718bca5b9" candidate_uuid: "9787869327f04a04a23ea007848ae36b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c7b2b642c9144d568877db72b8553ac5"
I20251209 08:58:55.052309 28507 raft_consensus.cc:3060] T 6774c15d83e043ab9b2b2e0718bca5b9 P 7182d3e0faba4ceaa1acc9b8b3674cbb [term 0 FOLLOWER]: Advancing to term 1
I20251209 08:58:55.052353 28370 raft_consensus.cc:3060] T 6774c15d83e043ab9b2b2e0718bca5b9 P c7b2b642c9144d568877db72b8553ac5 [term 0 FOLLOWER]: Advancing to term 1
I20251209 08:58:55.053331 28507 raft_consensus.cc:2468] T 6774c15d83e043ab9b2b2e0718bca5b9 P 7182d3e0faba4ceaa1acc9b8b3674cbb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9787869327f04a04a23ea007848ae36b in term 1.
I20251209 08:58:55.053381 28370 raft_consensus.cc:2468] T 6774c15d83e043ab9b2b2e0718bca5b9 P c7b2b642c9144d568877db72b8553ac5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9787869327f04a04a23ea007848ae36b in term 1.
I20251209 08:58:55.053647 28579 leader_election.cc:304] T 6774c15d83e043ab9b2b2e0718bca5b9 P 9787869327f04a04a23ea007848ae36b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7182d3e0faba4ceaa1acc9b8b3674cbb, 9787869327f04a04a23ea007848ae36b; no voters:
I20251209 08:58:55.053858 28724 raft_consensus.cc:2804] T 6774c15d83e043ab9b2b2e0718bca5b9 P 9787869327f04a04a23ea007848ae36b [term 1 FOLLOWER]: Leader election won for term 1
I20251209 08:58:55.053954 28724 raft_consensus.cc:697] T 6774c15d83e043ab9b2b2e0718bca5b9 P 9787869327f04a04a23ea007848ae36b [term 1 LEADER]: Becoming Leader. State: Replica: 9787869327f04a04a23ea007848ae36b, State: Running, Role: LEADER
I20251209 08:58:55.054287 28724 consensus_queue.cc:237] T 6774c15d83e043ab9b2b2e0718bca5b9 P 9787869327f04a04a23ea007848ae36b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7182d3e0faba4ceaa1acc9b8b3674cbb" member_type: VOTER last_known_addr { host: "127.20.49.66" port: 35715 } } peers { permanent_uuid: "c7b2b642c9144d568877db72b8553ac5" member_type: VOTER last_known_addr { host: "127.20.49.65" port: 38709 } } peers { permanent_uuid: "9787869327f04a04a23ea007848ae36b" member_type: VOTER last_known_addr { host: "127.20.49.67" port: 33177 } }
I20251209 08:58:55.057170 27427 catalog_manager.cc:5654] T 6774c15d83e043ab9b2b2e0718bca5b9 P 9787869327f04a04a23ea007848ae36b reported cstate change: term changed from 0 to 1, leader changed from <none> to 9787869327f04a04a23ea007848ae36b (127.20.49.67). New cstate: current_term: 1 leader_uuid: "9787869327f04a04a23ea007848ae36b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7182d3e0faba4ceaa1acc9b8b3674cbb" member_type: VOTER last_known_addr { host: "127.20.49.66" port: 35715 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c7b2b642c9144d568877db72b8553ac5" member_type: VOTER last_known_addr { host: "127.20.49.65" port: 38709 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9787869327f04a04a23ea007848ae36b" member_type: VOTER last_known_addr { host: "127.20.49.67" port: 33177 } health_report { overall_health: HEALTHY } } }
I20251209 08:58:55.077606 28716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:58:55.074595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55024 (local address 127.20.49.67:33177)
1209 08:58:55.074746 (+ 151us) server_negotiation.cc:207] Beginning negotiation
1209 08:58:55.074750 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1209 08:58:55.074765 (+ 15us) server_negotiation.cc:408] Connection header received
1209 08:58:55.074841 (+ 76us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1209 08:58:55.074843 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1209 08:58:55.074898 (+ 55us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1209 08:58:55.074981 (+ 83us) server_negotiation.cc:227] Negotiated authn=TOKEN
1209 08:58:55.075328 (+ 347us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:58:55.075871 (+ 543us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1209 08:58:55.076488 (+ 617us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:58:55.076665 (+ 177us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:58:55.076775 (+ 110us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1209 08:58:55.077229 (+ 454us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1209 08:58:55.077344 (+ 115us) server_negotiation.cc:1036] Waiting for connection context
1209 08:58:55.077395 (+ 51us) server_negotiation.cc:300] Negotiation successful
1209 08:58:55.077445 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20251209 08:58:55.083684 28507 raft_consensus.cc:1275] T 6774c15d83e043ab9b2b2e0718bca5b9 P 7182d3e0faba4ceaa1acc9b8b3674cbb [term 1 FOLLOWER]: Refusing update from remote peer 9787869327f04a04a23ea007848ae36b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251209 08:58:55.083684 28370 raft_consensus.cc:1275] T 6774c15d83e043ab9b2b2e0718bca5b9 P c7b2b642c9144d568877db72b8553ac5 [term 1 FOLLOWER]: Refusing update from remote peer 9787869327f04a04a23ea007848ae36b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251209 08:58:55.084400 28724 consensus_queue.cc:1048] T 6774c15d83e043ab9b2b2e0718bca5b9 P 9787869327f04a04a23ea007848ae36b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c7b2b642c9144d568877db72b8553ac5" member_type: VOTER last_known_addr { host: "127.20.49.65" port: 38709 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251209 08:58:55.084612 28725 consensus_queue.cc:1048] T 6774c15d83e043ab9b2b2e0718bca5b9 P 9787869327f04a04a23ea007848ae36b [LEADER]: Connected to new peer: Peer: permanent_uuid: "7182d3e0faba4ceaa1acc9b8b3674cbb" member_type: VOTER last_known_addr { host: "127.20.49.66" port: 35715 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251209 08:58:55.092806 28730 mvcc.cc:204] Tried to move back new op lower bound from 7230548930894815232 to 7230548930784989184. Current Snapshot: MvccSnapshot[applied={T|T < 7230548930894815232}]
I20251209 08:58:55.093811 28731 mvcc.cc:204] Tried to move back new op lower bound from 7230548930894815232 to 7230548930784989184. Current Snapshot: MvccSnapshot[applied={T|T < 7230548930894815232}]
I20251209 08:58:55.099072 28732 mvcc.cc:204] Tried to move back new op lower bound from 7230548930894815232 to 7230548930784989184. Current Snapshot: MvccSnapshot[applied={T|T < 7230548930894815232}]
I20251209 08:58:55.102844 27425 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58576:
table { table_name: "test-table" } modify_external_catalogs: true
I20251209 08:58:55.102986 27425 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58576:
table { table_name: "test-table" } modify_external_catalogs: true
I20251209 08:58:55.107384 27425 catalog_manager.cc:5941] T 00000000000000000000000000000000 P c0a0d121c6bb474dac7c98dc7ec52d14: Sending DeleteTablet for 3 replicas of tablet 6774c15d83e043ab9b2b2e0718bca5b9
I20251209 08:58:55.108456 28351 tablet_service.cc:1552] Processing DeleteTablet for tablet 6774c15d83e043ab9b2b2e0718bca5b9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-09 08:58:55 UTC) from {username='oryx', principal='oryx/127.20.49.126@KRBTEST.COM'} at 127.0.0.1:43026
I20251209 08:58:55.109166 28739 tablet_replica.cc:333] T 6774c15d83e043ab9b2b2e0718bca5b9 P c7b2b642c9144d568877db72b8553ac5: stopping tablet replica
I20251209 08:58:55.109359 28487 tablet_service.cc:1552] Processing DeleteTablet for tablet 6774c15d83e043ab9b2b2e0718bca5b9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-09 08:58:55 UTC) from {username='oryx', principal='oryx/127.20.49.126@KRBTEST.COM'} at 127.0.0.1:49872
I20251209 08:58:55.109488 28739 raft_consensus.cc:2243] T 6774c15d83e043ab9b2b2e0718bca5b9 P c7b2b642c9144d568877db72b8553ac5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251209 08:58:55.109913 28739 raft_consensus.cc:2272] T 6774c15d83e043ab9b2b2e0718bca5b9 P c7b2b642c9144d568877db72b8553ac5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251209 08:58:55.110352 28623 tablet_service.cc:1552] Processing DeleteTablet for tablet 6774c15d83e043ab9b2b2e0718bca5b9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-09 08:58:55 UTC) from {username='oryx', principal='oryx/127.20.49.126@KRBTEST.COM'} at 127.0.0.1:55022
I20251209 08:58:55.110961 28740 tablet_replica.cc:333] T 6774c15d83e043ab9b2b2e0718bca5b9 P 7182d3e0faba4ceaa1acc9b8b3674cbb: stopping tablet replica
I20251209 08:58:55.111284 28740 raft_consensus.cc:2243] T 6774c15d83e043ab9b2b2e0718bca5b9 P 7182d3e0faba4ceaa1acc9b8b3674cbb [term 1 FOLLOWER]: Raft consensus shutting down.
I20251209 08:58:55.111673 20677 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task8XuDZV/build/debug/bin/kudu with pid 27395
I20251209 08:58:55.111676 28740 raft_consensus.cc:2272] T 6774c15d83e043ab9b2b2e0718bca5b9 P 7182d3e0faba4ceaa1acc9b8b3674cbb [term 1 FOLLOWER]: Raft consensus is shut down!
I20251209 08:58:55.112792 28739 ts_tablet_manager.cc:1916] T 6774c15d83e043ab9b2b2e0718bca5b9 P c7b2b642c9144d568877db72b8553ac5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251209 08:58:55.115271 28739 ts_tablet_manager.cc:1929] T 6774c15d83e043ab9b2b2e0718bca5b9 P c7b2b642c9144d568877db72b8553ac5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251209 08:58:55.115375 28739 log.cc:1199] T 6774c15d83e043ab9b2b2e0718bca5b9 P c7b2b642c9144d568877db72b8553ac5: Deleting WAL directory at /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765270668420570-20677-0/minicluster-data/ts-0/wal/wals/6774c15d83e043ab9b2b2e0718bca5b9
I20251209 08:58:55.115687 28739 ts_tablet_manager.cc:1950] T 6774c15d83e043ab9b2b2e0718bca5b9 P c7b2b642c9144d568877db72b8553ac5: Deleting consensus metadata
I20251209 08:58:55.116999 28741 tablet_replica.cc:333] T 6774c15d83e043ab9b2b2e0718bca5b9 P 9787869327f04a04a23ea007848ae36b: stopping tablet replica
I20251209 08:58:55.117264 28741 raft_consensus.cc:2243] T 6774c15d83e043ab9b2b2e0718bca5b9 P 9787869327f04a04a23ea007848ae36b [term 1 LEADER]: Raft consensus shutting down.
I20251209 08:58:55.117599 28741 raft_consensus.cc:2272] T 6774c15d83e043ab9b2b2e0718bca5b9 P 9787869327f04a04a23ea007848ae36b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251209 08:58:55.118863 28741 ts_tablet_manager.cc:1916] T 6774c15d83e043ab9b2b2e0718bca5b9 P 9787869327f04a04a23ea007848ae36b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251209 08:58:55.120877 28741 ts_tablet_manager.cc:1929] T 6774c15d83e043ab9b2b2e0718bca5b9 P 9787869327f04a04a23ea007848ae36b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251209 08:58:55.120967 28741 log.cc:1199] T 6774c15d83e043ab9b2b2e0718bca5b9 P 9787869327f04a04a23ea007848ae36b: Deleting WAL directory at /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765270668420570-20677-0/minicluster-data/ts-2/wal/wals/6774c15d83e043ab9b2b2e0718bca5b9
I20251209 08:58:55.121300 28741 ts_tablet_manager.cc:1950] T 6774c15d83e043ab9b2b2e0718bca5b9 P 9787869327f04a04a23ea007848ae36b: Deleting consensus metadata
W20251209 08:58:55.121459 28305 connection.cc:537] server connection from 127.0.0.1:43026 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20251209 08:58:55.122701 28576 connection.cc:441] server connection from 127.0.0.1:55022 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55022 (request call id 1) could send its response
I20251209 08:58:55.123889 28740 ts_tablet_manager.cc:1916] T 6774c15d83e043ab9b2b2e0718bca5b9 P 7182d3e0faba4ceaa1acc9b8b3674cbb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251209 08:58:55.126892 28740 ts_tablet_manager.cc:1929] T 6774c15d83e043ab9b2b2e0718bca5b9 P 7182d3e0faba4ceaa1acc9b8b3674cbb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251209 08:58:55.127068 28740 log.cc:1199] T 6774c15d83e043ab9b2b2e0718bca5b9 P 7182d3e0faba4ceaa1acc9b8b3674cbb: Deleting WAL directory at /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765270668420570-20677-0/minicluster-data/ts-1/wal/wals/6774c15d83e043ab9b2b2e0718bca5b9
I20251209 08:58:55.127805 28740 ts_tablet_manager.cc:1950] T 6774c15d83e043ab9b2b2e0718bca5b9 P 7182d3e0faba4ceaa1acc9b8b3674cbb: Deleting consensus metadata
W20251209 08:58:55.129053 28443 connection.cc:441] server connection from 127.0.0.1:49872 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:49872 (request call id 1) could send its response
I20251209 08:58:55.134583 28286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:58:55.124159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56830 (local address 127.20.49.124:44451)
1209 08:58:55.124289 (+ 130us) server_negotiation.cc:207] Beginning negotiation
1209 08:58:55.124294 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1209 08:58:55.125003 (+ 709us) server_negotiation.cc:408] Connection header received
1209 08:58:55.125205 (+ 202us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1209 08:58:55.125209 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1209 08:58:55.125265 (+ 56us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1209 08:58:55.125344 (+ 79us) server_negotiation.cc:227] Negotiated authn=SASL
1209 08:58:55.126633 (+ 1289us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:58:55.127473 (+ 840us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1209 08:58:55.128412 (+ 939us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:58:55.128581 (+ 169us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:58:55.130511 (+ 1930us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1209 08:58:55.130550 (+ 39us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1209 08:58:55.130552 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1209 08:58:55.130580 (+ 28us) server_negotiation.cc:923] calling sasl_server_start()
1209 08:58:55.132747 (+ 2167us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1209 08:58:55.133367 (+ 620us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1209 08:58:55.133371 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1209 08:58:55.133373 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1209 08:58:55.133430 (+ 57us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1209 08:58:55.133669 (+ 239us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1209 08:58:55.133707 (+ 38us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1209 08:58:55.133712 (+ 5us) server_negotiation.cc:968] calling sasl_server_step()
1209 08:58:55.133909 (+ 197us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1209 08:58:55.134049 (+ 140us) server_negotiation.cc:1036] Waiting for connection context
1209 08:58:55.134291 (+ 242us) server_negotiation.cc:300] Negotiation successful
1209 08:58:55.134420 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20251209 08:58:55.134618 28283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:58:55.124549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36978 (local address 127.20.49.125:44905)
1209 08:58:55.124681 (+ 132us) server_negotiation.cc:207] Beginning negotiation
1209 08:58:55.124685 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1209 08:58:55.125182 (+ 497us) server_negotiation.cc:408] Connection header received
1209 08:58:55.125361 (+ 179us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1209 08:58:55.125364 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1209 08:58:55.125416 (+ 52us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1209 08:58:55.125468 (+ 52us) server_negotiation.cc:227] Negotiated authn=SASL
1209 08:58:55.126601 (+ 1133us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:58:55.127387 (+ 786us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1209 08:58:55.128325 (+ 938us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:58:55.128526 (+ 201us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:58:55.130855 (+ 2329us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1209 08:58:55.130882 (+ 27us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1209 08:58:55.130887 (+ 5us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1209 08:58:55.130923 (+ 36us) server_negotiation.cc:923] calling sasl_server_start()
1209 08:58:55.132906 (+ 1983us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1209 08:58:55.133386 (+ 480us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1209 08:58:55.133389 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1209 08:58:55.133391 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1209 08:58:55.133437 (+ 46us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1209 08:58:55.133757 (+ 320us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1209 08:58:55.133761 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1209 08:58:55.133763 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1209 08:58:55.133923 (+ 160us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1209 08:58:55.134031 (+ 108us) server_negotiation.cc:1036] Waiting for connection context
1209 08:58:55.134364 (+ 333us) server_negotiation.cc:300] Negotiation successful
1209 08:58:55.134477 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20251209 08:58:56.097023 28749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:58:56.096600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.49.126:42449 (local address 127.20.49.67:38367)
1209 08:58:56.096852 (+ 252us) negotiation.cc:107] Waiting for socket to connect
1209 08:58:56.096923 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.49.126:42449: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":82,"threads_started":1}
W20251209 08:58:56.097302 28689 heartbeater.cc:646] Failed to heartbeat to 127.20.49.126:42449 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.20.49.126:42449: connect: Connection refused (error 111)
I20251209 08:58:56.099975 28750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:58:56.099545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.49.126:42449 (local address 127.20.49.65:34845)
1209 08:58:56.099791 (+ 246us) negotiation.cc:107] Waiting for socket to connect
1209 08:58:56.099859 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.49.126:42449: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":94,"threads_started":1}
W20251209 08:58:56.100340 28417 heartbeater.cc:646] Failed to heartbeat to 127.20.49.126:42449 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.20.49.126:42449: connect: Connection refused (error 111)
I20251209 08:58:56.102625 28751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:58:56.102200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.49.126:42449 (local address 127.20.49.66:34259)
1209 08:58:56.102463 (+ 263us) negotiation.cc:107] Waiting for socket to connect
1209 08:58:56.102532 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.49.126:42449: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":73,"threads_started":1}
W20251209 08:58:56.102881 28553 heartbeater.cc:646] Failed to heartbeat to 127.20.49.126:42449 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.20.49.126:42449: connect: Connection refused (error 111)
I20251209 08:58:56.608353 28752 raft_consensus.cc:493] T 00000000000000000000000000000000 P 0364ed40f171452f9fa244fa50f6a6c3 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c0a0d121c6bb474dac7c98dc7ec52d14)
I20251209 08:58:56.608556 28752 raft_consensus.cc:515] T 00000000000000000000000000000000 P 0364ed40f171452f9fa244fa50f6a6c3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "c0a0d121c6bb474dac7c98dc7ec52d14" member_type: VOTER last_known_addr { host: "127.20.49.126" port: 42449 } } peers { permanent_uuid: "798ad87d8a5a4392815120c18c40bedf" member_type: VOTER last_known_addr { host: "127.20.49.125" port: 44905 } attrs { promote: false } } peers { permanent_uuid: "0364ed40f171452f9fa244fa50f6a6c3" member_type: VOTER last_known_addr { host: "127.20.49.124" port: 44451 } attrs { promote: false } }
I20251209 08:58:56.608850 28753 raft_consensus.cc:493] T 00000000000000000000000000000000 P 798ad87d8a5a4392815120c18c40bedf [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c0a0d121c6bb474dac7c98dc7ec52d14)
I20251209 08:58:56.608970 28753 raft_consensus.cc:515] T 00000000000000000000000000000000 P 798ad87d8a5a4392815120c18c40bedf [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "c0a0d121c6bb474dac7c98dc7ec52d14" member_type: VOTER last_known_addr { host: "127.20.49.126" port: 42449 } } peers { permanent_uuid: "798ad87d8a5a4392815120c18c40bedf" member_type: VOTER last_known_addr { host: "127.20.49.125" port: 44905 } attrs { promote: false } } peers { permanent_uuid: "0364ed40f171452f9fa244fa50f6a6c3" member_type: VOTER last_known_addr { host: "127.20.49.124" port: 44451 } attrs { promote: false } }
I20251209 08:58:56.609516 28753 leader_election.cc:290] T 00000000000000000000000000000000 P 798ad87d8a5a4392815120c18c40bedf [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c0a0d121c6bb474dac7c98dc7ec52d14 (127.20.49.126:42449), 0364ed40f171452f9fa244fa50f6a6c3 (127.20.49.124:44451)
I20251209 08:58:56.609825 28752 leader_election.cc:290] T 00000000000000000000000000000000 P 0364ed40f171452f9fa244fa50f6a6c3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c0a0d121c6bb474dac7c98dc7ec52d14 (127.20.49.126:42449), 798ad87d8a5a4392815120c18c40bedf (127.20.49.125:44905)
I20251209 08:58:56.610064 28232 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "798ad87d8a5a4392815120c18c40bedf" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "0364ed40f171452f9fa244fa50f6a6c3" is_pre_election: true
I20251209 08:58:56.610226 28232 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 0364ed40f171452f9fa244fa50f6a6c3 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 798ad87d8a5a4392815120c18c40bedf in term 1.
I20251209 08:58:56.610268 28049 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0364ed40f171452f9fa244fa50f6a6c3" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "798ad87d8a5a4392815120c18c40bedf" is_pre_election: true
I20251209 08:58:56.610327 28754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:58:56.609748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.49.126:42449 (local address 127.0.0.1:59014)
1209 08:58:56.610133 (+ 385us) negotiation.cc:107] Waiting for socket to connect
1209 08:58:56.610215 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.49.126:42449: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":299,"thread_start_us":59,"threads_started":1}
I20251209 08:58:56.610391 28755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:58:56.609929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.49.126:42449 (local address 127.0.0.1:59020)
1209 08:58:56.610201 (+ 272us) negotiation.cc:107] Waiting for socket to connect
1209 08:58:56.610285 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.49.126:42449: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":66,"threads_started":1}
I20251209 08:58:56.610430 28049 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 798ad87d8a5a4392815120c18c40bedf [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0364ed40f171452f9fa244fa50f6a6c3 in term 1.
W20251209 08:58:56.610522 28208 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.49.126:42449: connect: Connection refused (error 111)
I20251209 08:58:56.610657 28026 leader_election.cc:304] T 00000000000000000000000000000000 P 798ad87d8a5a4392815120c18c40bedf [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0364ed40f171452f9fa244fa50f6a6c3, 798ad87d8a5a4392815120c18c40bedf; no voters:
W20251209 08:58:56.610674 28025 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.49.126:42449: connect: Connection refused (error 111)
I20251209 08:58:56.610903 28753 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 798ad87d8a5a4392815120c18c40bedf [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251209 08:58:56.610885 28206 leader_election.cc:304] T 00000000000000000000000000000000 P 0364ed40f171452f9fa244fa50f6a6c3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0364ed40f171452f9fa244fa50f6a6c3, 798ad87d8a5a4392815120c18c40bedf; no voters:
I20251209 08:58:56.611037 28753 raft_consensus.cc:493] T 00000000000000000000000000000000 P 798ad87d8a5a4392815120c18c40bedf [term 1 FOLLOWER]: Starting leader election (detected failure of leader c0a0d121c6bb474dac7c98dc7ec52d14)
I20251209 08:58:56.611102 28752 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 0364ed40f171452f9fa244fa50f6a6c3 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251209 08:58:56.611169 28752 raft_consensus.cc:493] T 00000000000000000000000000000000 P 0364ed40f171452f9fa244fa50f6a6c3 [term 1 FOLLOWER]: Starting leader election (detected failure of leader c0a0d121c6bb474dac7c98dc7ec52d14)
I20251209 08:58:56.611227 28752 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 0364ed40f171452f9fa244fa50f6a6c3 [term 1 FOLLOWER]: Advancing to term 2
I20251209 08:58:56.611413 28753 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 798ad87d8a5a4392815120c18c40bedf [term 1 FOLLOWER]: Advancing to term 2
I20251209 08:58:56.611567 28755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:58:56.611302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.49.126:42449 (local address 127.0.0.1:59026)
1209 08:58:56.611444 (+ 142us) negotiation.cc:107] Waiting for socket to connect
1209 08:58:56.611499 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.49.126:42449: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20251209 08:58:56.611672 28754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:58:56.611377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.49.126:42449 (local address 127.0.0.1:59028)
1209 08:58:56.611524 (+ 147us) negotiation.cc:107] Waiting for socket to connect
1209 08:58:56.611574 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.49.126:42449: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
W20251209 08:58:56.611728 28208 leader_election.cc:336] T 00000000000000000000000000000000 P 0364ed40f171452f9fa244fa50f6a6c3 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c0a0d121c6bb474dac7c98dc7ec52d14 (127.20.49.126:42449): Network error: Client connection negotiation failed: client connection to 127.20.49.126:42449: connect: Connection refused (error 111)
W20251209 08:58:56.611879 28025 leader_election.cc:336] T 00000000000000000000000000000000 P 798ad87d8a5a4392815120c18c40bedf [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c0a0d121c6bb474dac7c98dc7ec52d14 (127.20.49.126:42449): Network error: Client connection negotiation failed: client connection to 127.20.49.126:42449: connect: Connection refused (error 111)
I20251209 08:58:56.612475 28752 raft_consensus.cc:515] T 00000000000000000000000000000000 P 0364ed40f171452f9fa244fa50f6a6c3 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "c0a0d121c6bb474dac7c98dc7ec52d14" member_type: VOTER last_known_addr { host: "127.20.49.126" port: 42449 } } peers { permanent_uuid: "798ad87d8a5a4392815120c18c40bedf" member_type: VOTER last_known_addr { host: "127.20.49.125" port: 44905 } attrs { promote: false } } peers { permanent_uuid: "0364ed40f171452f9fa244fa50f6a6c3" member_type: VOTER last_known_addr { host: "127.20.49.124" port: 44451 } attrs { promote: false } }
I20251209 08:58:56.612576 28753 raft_consensus.cc:515] T 00000000000000000000000000000000 P 798ad87d8a5a4392815120c18c40bedf [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "c0a0d121c6bb474dac7c98dc7ec52d14" member_type: VOTER last_known_addr { host: "127.20.49.126" port: 42449 } } peers { permanent_uuid: "798ad87d8a5a4392815120c18c40bedf" member_type: VOTER last_known_addr { host: "127.20.49.125" port: 44905 } attrs { promote: false } } peers { permanent_uuid: "0364ed40f171452f9fa244fa50f6a6c3" member_type: VOTER last_known_addr { host: "127.20.49.124" port: 44451 } attrs { promote: false } }
I20251209 08:58:56.612830 28752 leader_election.cc:290] T 00000000000000000000000000000000 P 0364ed40f171452f9fa244fa50f6a6c3 [CANDIDATE]: Term 2 election: Requested vote from peers c0a0d121c6bb474dac7c98dc7ec52d14 (127.20.49.126:42449), 798ad87d8a5a4392815120c18c40bedf (127.20.49.125:44905)
I20251209 08:58:56.613163 28049 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0364ed40f171452f9fa244fa50f6a6c3" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "798ad87d8a5a4392815120c18c40bedf"
I20251209 08:58:56.613194 28755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:58:56.612912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.49.126:42449 (local address 127.0.0.1:59040)
1209 08:58:56.613055 (+ 143us) negotiation.cc:107] Waiting for socket to connect
1209 08:58:56.613128 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.49.126:42449: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20251209 08:58:56.613176 28232 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "798ad87d8a5a4392815120c18c40bedf" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "0364ed40f171452f9fa244fa50f6a6c3"
I20251209 08:58:56.613315 28049 raft_consensus.cc:2393] T 00000000000000000000000000000000 P 798ad87d8a5a4392815120c18c40bedf [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 0364ed40f171452f9fa244fa50f6a6c3 in current term 2: Already voted for candidate 798ad87d8a5a4392815120c18c40bedf in this term.
I20251209 08:58:56.613360 28232 raft_consensus.cc:2393] T 00000000000000000000000000000000 P 0364ed40f171452f9fa244fa50f6a6c3 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 798ad87d8a5a4392815120c18c40bedf in current term 2: Already voted for candidate 0364ed40f171452f9fa244fa50f6a6c3 in this term.
I20251209 08:58:56.613484 28754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:58:56.613217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.49.126:42449 (local address 127.0.0.1:59050)
1209 08:58:56.613371 (+ 154us) negotiation.cc:107] Waiting for socket to connect
1209 08:58:56.613420 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.49.126:42449: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20251209 08:58:56.613786 28753 leader_election.cc:290] T 00000000000000000000000000000000 P 798ad87d8a5a4392815120c18c40bedf [CANDIDATE]: Term 2 election: Requested vote from peers c0a0d121c6bb474dac7c98dc7ec52d14 (127.20.49.126:42449), 0364ed40f171452f9fa244fa50f6a6c3 (127.20.49.124:44451)
I20251209 08:58:56.613945 28758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:58:56.613618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.49.126:42449 (local address 127.0.0.1:59060)
1209 08:58:56.613835 (+ 217us) negotiation.cc:107] Waiting for socket to connect
1209 08:58:56.613883 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.49.126:42449: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":97,"threads_started":1}
I20251209 08:58:56.614142 28754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:58:56.613938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.49.126:42449 (local address 127.0.0.1:59072)
1209 08:58:56.614048 (+ 110us) negotiation.cc:107] Waiting for socket to connect
1209 08:58:56.614097 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.49.126:42449: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20251209 08:58:56.614167 28208 leader_election.cc:336] T 00000000000000000000000000000000 P 0364ed40f171452f9fa244fa50f6a6c3 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer c0a0d121c6bb474dac7c98dc7ec52d14 (127.20.49.126:42449): Network error: Client connection negotiation failed: client connection to 127.20.49.126:42449: connect: Connection refused (error 111)
I20251209 08:58:56.614223 28208 leader_election.cc:304] T 00000000000000000000000000000000 P 0364ed40f171452f9fa244fa50f6a6c3 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0364ed40f171452f9fa244fa50f6a6c3; no voters: 798ad87d8a5a4392815120c18c40bedf, c0a0d121c6bb474dac7c98dc7ec52d14
W20251209 08:58:56.614248 28025 leader_election.cc:336] T 00000000000000000000000000000000 P 798ad87d8a5a4392815120c18c40bedf [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer c0a0d121c6bb474dac7c98dc7ec52d14 (127.20.49.126:42449): Network error: Client connection negotiation failed: client connection to 127.20.49.126:42449: connect: Connection refused (error 111)
I20251209 08:58:56.614303 28025 leader_election.cc:304] T 00000000000000000000000000000000 P 798ad87d8a5a4392815120c18c40bedf [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 798ad87d8a5a4392815120c18c40bedf; no voters: 0364ed40f171452f9fa244fa50f6a6c3, c0a0d121c6bb474dac7c98dc7ec52d14
I20251209 08:58:56.614528 28752 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 0364ed40f171452f9fa244fa50f6a6c3 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20251209 08:58:56.614506 28753 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 798ad87d8a5a4392815120c18c40bedf [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20251209 08:58:57.098490 28759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:58:57.098024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.49.126:42449 (local address 127.20.49.67:44477)
1209 08:58:57.098309 (+ 285us) negotiation.cc:107] Waiting for socket to connect
1209 08:58:57.098385 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.49.126:42449: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":79,"threads_started":1}
I20251209 08:58:57.101325 28760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:58:57.100866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.49.126:42449 (local address 127.20.49.65:48373)
1209 08:58:57.101164 (+ 298us) negotiation.cc:107] Waiting for socket to connect
1209 08:58:57.101217 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.49.126:42449: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":77,"threads_started":1}
I20251209 08:58:57.103693 28761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:58:57.103327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.49.126:42449 (local address 127.20.49.66:50963)
1209 08:58:57.103558 (+ 231us) negotiation.cc:107] Waiting for socket to connect
1209 08:58:57.103608 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.49.126:42449: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":66,"threads_started":1}
I20251209 08:58:58.099620 28762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:58:58.099196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.49.126:42449 (local address 127.20.49.67:55617)
1209 08:58:58.099465 (+ 269us) negotiation.cc:107] Waiting for socket to connect
1209 08:58:58.099521 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.49.126:42449: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":98,"threads_started":1}
I20251209 08:58:58.102517 28763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:58:58.102147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.49.126:42449 (local address 127.20.49.65:40031)
1209 08:58:58.102378 (+ 231us) negotiation.cc:107] Waiting for socket to connect
1209 08:58:58.102430 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.49.126:42449: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":75,"threads_started":1}
I20251209 08:58:58.104652 28764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:58:58.104274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.49.126:42449 (local address 127.20.49.66:33977)
1209 08:58:58.104520 (+ 246us) negotiation.cc:107] Waiting for socket to connect
1209 08:58:58.104568 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.49.126:42449: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":81,"threads_started":1}
I20251209 08:58:58.394122 28765 raft_consensus.cc:493] T 00000000000000000000000000000000 P 0364ed40f171452f9fa244fa50f6a6c3 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251209 08:58:58.394275 28765 raft_consensus.cc:515] T 00000000000000000000000000000000 P 0364ed40f171452f9fa244fa50f6a6c3 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "c0a0d121c6bb474dac7c98dc7ec52d14" member_type: VOTER last_known_addr { host: "127.20.49.126" port: 42449 } } peers { permanent_uuid: "798ad87d8a5a4392815120c18c40bedf" member_type: VOTER last_known_addr { host: "127.20.49.125" port: 44905 } attrs { promote: false } } peers { permanent_uuid: "0364ed40f171452f9fa244fa50f6a6c3" member_type: VOTER last_known_addr { host: "127.20.49.124" port: 44451 } attrs { promote: false } }
I20251209 08:58:58.394754 28765 leader_election.cc:290] T 00000000000000000000000000000000 P 0364ed40f171452f9fa244fa50f6a6c3 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c0a0d121c6bb474dac7c98dc7ec52d14 (127.20.49.126:42449), 798ad87d8a5a4392815120c18c40bedf (127.20.49.125:44905)
I20251209 08:58:58.395246 28049 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0364ed40f171452f9fa244fa50f6a6c3" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "798ad87d8a5a4392815120c18c40bedf" is_pre_election: true
I20251209 08:58:58.395380 28049 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 798ad87d8a5a4392815120c18c40bedf [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0364ed40f171452f9fa244fa50f6a6c3 in term 2.
I20251209 08:58:58.395422 28766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:58:58.394967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.49.126:42449 (local address 127.0.0.1:37106)
1209 08:58:58.395252 (+ 285us) negotiation.cc:107] Waiting for socket to connect
1209 08:58:58.395307 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.49.126:42449: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":64,"threads_started":1}
I20251209 08:58:58.395808 28206 leader_election.cc:304] T 00000000000000000000000000000000 P 0364ed40f171452f9fa244fa50f6a6c3 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0364ed40f171452f9fa244fa50f6a6c3, 798ad87d8a5a4392815120c18c40bedf; no voters:
I20251209 08:58:58.395991 28765 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 0364ed40f171452f9fa244fa50f6a6c3 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251209 08:58:58.396095 28765 raft_consensus.cc:493] T 00000000000000000000000000000000 P 0364ed40f171452f9fa244fa50f6a6c3 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251209 08:58:58.396159 28765 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 0364ed40f171452f9fa244fa50f6a6c3 [term 2 FOLLOWER]: Advancing to term 3
I20251209 08:58:58.396410 28766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:58:58.396092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.49.126:42449 (local address 127.0.0.1:37110)
1209 08:58:58.396295 (+ 203us) negotiation.cc:107] Waiting for socket to connect
1209 08:58:58.396344 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.49.126:42449: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118}
W20251209 08:58:58.396502 28208 leader_election.cc:336] T 00000000000000000000000000000000 P 0364ed40f171452f9fa244fa50f6a6c3 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c0a0d121c6bb474dac7c98dc7ec52d14 (127.20.49.126:42449): Network error: Client connection negotiation failed: client connection to 127.20.49.126:42449: connect: Connection refused (error 111)
I20251209 08:58:58.397550 28765 raft_consensus.cc:515] T 00000000000000000000000000000000 P 0364ed40f171452f9fa244fa50f6a6c3 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "c0a0d121c6bb474dac7c98dc7ec52d14" member_type: VOTER last_known_addr { host: "127.20.49.126" port: 42449 } } peers { permanent_uuid: "798ad87d8a5a4392815120c18c40bedf" member_type: VOTER last_known_addr { host: "127.20.49.125" port: 44905 } attrs { promote: false } } peers { permanent_uuid: "0364ed40f171452f9fa244fa50f6a6c3" member_type: VOTER last_known_addr { host: "127.20.49.124" port: 44451 } attrs { promote: false } }
I20251209 08:58:58.397939 28765 leader_election.cc:290] T 00000000000000000000000000000000 P 0364ed40f171452f9fa244fa50f6a6c3 [CANDIDATE]: Term 3 election: Requested vote from peers c0a0d121c6bb474dac7c98dc7ec52d14 (127.20.49.126:42449), 798ad87d8a5a4392815120c18c40bedf (127.20.49.125:44905)
I20251209 08:58:58.398308 28049 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0364ed40f171452f9fa244fa50f6a6c3" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "798ad87d8a5a4392815120c18c40bedf"
I20251209 08:58:58.398458 28766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:58:58.398115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.49.126:42449 (local address 127.0.0.1:37120)
1209 08:58:58.398323 (+ 208us) negotiation.cc:107] Waiting for socket to connect
1209 08:58:58.398387 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.49.126:42449: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138}
I20251209 08:58:58.398450 28049 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 798ad87d8a5a4392815120c18c40bedf [term 2 FOLLOWER]: Advancing to term 3
I20251209 08:58:58.399402 28766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:58:58.399123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.49.126:42449 (local address 127.0.0.1:37122)
1209 08:58:58.399289 (+ 166us) negotiation.cc:107] Waiting for socket to connect
1209 08:58:58.399340 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.49.126:42449: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
W20251209 08:58:58.399586 28208 leader_election.cc:336] T 00000000000000000000000000000000 P 0364ed40f171452f9fa244fa50f6a6c3 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer c0a0d121c6bb474dac7c98dc7ec52d14 (127.20.49.126:42449): Network error: Client connection negotiation failed: client connection to 127.20.49.126:42449: connect: Connection refused (error 111)
I20251209 08:58:58.399816 28049 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 798ad87d8a5a4392815120c18c40bedf [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0364ed40f171452f9fa244fa50f6a6c3 in term 3.
I20251209 08:58:58.400290 28206 leader_election.cc:304] T 00000000000000000000000000000000 P 0364ed40f171452f9fa244fa50f6a6c3 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0364ed40f171452f9fa244fa50f6a6c3, 798ad87d8a5a4392815120c18c40bedf; no voters: c0a0d121c6bb474dac7c98dc7ec52d14
I20251209 08:58:58.400516 28765 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 0364ed40f171452f9fa244fa50f6a6c3 [term 3 FOLLOWER]: Leader election won for term 3
I20251209 08:58:58.400674 28765 raft_consensus.cc:697] T 00000000000000000000000000000000 P 0364ed40f171452f9fa244fa50f6a6c3 [term 3 LEADER]: Becoming Leader. State: Replica: 0364ed40f171452f9fa244fa50f6a6c3, State: Running, Role: LEADER
I20251209 08:58:58.400913 28765 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0364ed40f171452f9fa244fa50f6a6c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "c0a0d121c6bb474dac7c98dc7ec52d14" member_type: VOTER last_known_addr { host: "127.20.49.126" port: 42449 } } peers { permanent_uuid: "798ad87d8a5a4392815120c18c40bedf" member_type: VOTER last_known_addr { host: "127.20.49.125" port: 44905 } attrs { promote: false } } peers { permanent_uuid: "0364ed40f171452f9fa244fa50f6a6c3" member_type: VOTER last_known_addr { host: "127.20.49.124" port: 44451 } attrs { promote: false } }
I20251209 08:58:58.402343 28768 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0364ed40f171452f9fa244fa50f6a6c3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0364ed40f171452f9fa244fa50f6a6c3. Latest consensus state: current_term: 3 leader_uuid: "0364ed40f171452f9fa244fa50f6a6c3" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "c0a0d121c6bb474dac7c98dc7ec52d14" member_type: VOTER last_known_addr { host: "127.20.49.126" port: 42449 } } peers { permanent_uuid: "798ad87d8a5a4392815120c18c40bedf" member_type: VOTER last_known_addr { host: "127.20.49.125" port: 44905 } attrs { promote: false } } peers { permanent_uuid: "0364ed40f171452f9fa244fa50f6a6c3" member_type: VOTER last_known_addr { host: "127.20.49.124" port: 44451 } attrs { promote: false } } }
I20251209 08:58:58.402478 28768 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0364ed40f171452f9fa244fa50f6a6c3 [sys.catalog]: This master's current role is: LEADER
I20251209 08:58:58.402817 28770 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251209 08:58:58.405018 28770 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251209 08:58:58.405404 28770 catalog_manager.cc:1269] Loaded cluster ID: 51c444202dc5408cb61915bc1a2c504c
I20251209 08:58:58.405467 28770 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251209 08:58:58.406276 28770 catalog_manager.cc:1514] Loading token signing keys...
I20251209 08:58:58.406817 28770 catalog_manager.cc:6038] T 00000000000000000000000000000000 P 0364ed40f171452f9fa244fa50f6a6c3: Loaded TSK: 0
I20251209 08:58:58.407843 28770 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251209 08:58:58.416823 28221 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56830:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251209 08:58:58.419317 28221 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251209 08:58:58.422655 28766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:58:58.422385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.49.126:42449 (local address 127.0.0.1:37130)
1209 08:58:58.422533 (+ 148us) negotiation.cc:107] Waiting for socket to connect
1209 08:58:58.422583 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.49.126:42449: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20251209 08:58:58.422683 28049 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 798ad87d8a5a4392815120c18c40bedf [term 3 FOLLOWER]: Refusing update from remote peer 0364ed40f171452f9fa244fa50f6a6c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20251209 08:58:58.423156 28765 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 0364ed40f171452f9fa244fa50f6a6c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "798ad87d8a5a4392815120c18c40bedf" member_type: VOTER last_known_addr { host: "127.20.49.125" port: 44905 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20251209 08:58:58.423391 28766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:58:58.423156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.49.126:42449 (local address 127.0.0.1:37134)
1209 08:58:58.423278 (+ 122us) negotiation.cc:107] Waiting for socket to connect
1209 08:58:58.423329 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.49.126:42449: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20251209 08:58:58.423516 28208 consensus_peers.cc:597] T 00000000000000000000000000000000 P 0364ed40f171452f9fa244fa50f6a6c3 -> Peer c0a0d121c6bb474dac7c98dc7ec52d14 (127.20.49.126:42449): Couldn't send request to peer c0a0d121c6bb474dac7c98dc7ec52d14. Status: Network error: Client connection negotiation failed: client connection to 127.20.49.126:42449: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20251209 08:58:58.424590 28772 sys_catalog.cc:455] T 00000000000000000000000000000000 P 798ad87d8a5a4392815120c18c40bedf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0364ed40f171452f9fa244fa50f6a6c3. Latest consensus state: current_term: 3 leader_uuid: "0364ed40f171452f9fa244fa50f6a6c3" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "c0a0d121c6bb474dac7c98dc7ec52d14" member_type: VOTER last_known_addr { host: "127.20.49.126" port: 42449 } } peers { permanent_uuid: "798ad87d8a5a4392815120c18c40bedf" member_type: VOTER last_known_addr { host: "127.20.49.125" port: 44905 } attrs { promote: false } } peers { permanent_uuid: "0364ed40f171452f9fa244fa50f6a6c3" member_type: VOTER last_known_addr { host: "127.20.49.124" port: 44451 } attrs { promote: false } } }
I20251209 08:58:58.424718 28772 sys_catalog.cc:458] T 00000000000000000000000000000000 P 798ad87d8a5a4392815120c18c40bedf [sys.catalog]: This master's current role is: FOLLOWER
I20251209 08:58:58.425644 28765 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0364ed40f171452f9fa244fa50f6a6c3 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "0364ed40f171452f9fa244fa50f6a6c3" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "c0a0d121c6bb474dac7c98dc7ec52d14" member_type: VOTER last_known_addr { host: "127.20.49.126" port: 42449 } } peers { permanent_uuid: "798ad87d8a5a4392815120c18c40bedf" member_type: VOTER last_known_addr { host: "127.20.49.125" port: 44905 } attrs { promote: false } } peers { permanent_uuid: "0364ed40f171452f9fa244fa50f6a6c3" member_type: VOTER last_known_addr { host: "127.20.49.124" port: 44451 } attrs { promote: false } } }
I20251209 08:58:58.425793 28765 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0364ed40f171452f9fa244fa50f6a6c3 [sys.catalog]: This master's current role is: LEADER
I20251209 08:58:58.426205 28773 mvcc.cc:204] Tried to move back new op lower bound from 7230548944574402560 to 7230548944492716032. Current Snapshot: MvccSnapshot[applied={T|T < 7230548944574402560}]
I20251209 08:58:58.426386 28772 sys_catalog.cc:455] T 00000000000000000000000000000000 P 798ad87d8a5a4392815120c18c40bedf [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "0364ed40f171452f9fa244fa50f6a6c3" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "c0a0d121c6bb474dac7c98dc7ec52d14" member_type: VOTER last_known_addr { host: "127.20.49.126" port: 42449 } } peers { permanent_uuid: "798ad87d8a5a4392815120c18c40bedf" member_type: VOTER last_known_addr { host: "127.20.49.125" port: 44905 } attrs { promote: false } } peers { permanent_uuid: "0364ed40f171452f9fa244fa50f6a6c3" member_type: VOTER last_known_addr { host: "127.20.49.124" port: 44451 } attrs { promote: false } } }
I20251209 08:58:58.426648 28772 sys_catalog.cc:458] T 00000000000000000000000000000000 P 798ad87d8a5a4392815120c18c40bedf [sys.catalog]: This master's current role is: FOLLOWER
I20251209 08:58:58.435175 28766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:58:58.431784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.49.66:35715 (local address 127.0.0.1:43576)
1209 08:58:58.432047 (+ 263us) negotiation.cc:107] Waiting for socket to connect
1209 08:58:58.432062 (+ 15us) client_negotiation.cc:174] Beginning negotiation
1209 08:58:58.432161 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1209 08:58:58.432786 (+ 625us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1209 08:58:58.432791 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
1209 08:58:58.432811 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1209 08:58:58.433070 (+ 259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:58:58.433077 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:58:58.434045 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1209 08:58:58.434048 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1209 08:58:58.434767 (+ 719us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:58:58.434774 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:58:58.434874 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:58:58.434895 (+ 21us) client_negotiation.cc:769] Sending connection context
1209 08:58:58.434951 (+ 56us) client_negotiation.cc:240] Negotiation successful
1209 08:58:58.435034 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":176}
I20251209 08:58:58.435850 28781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:58:58.432528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.49.65:38709 (local address 127.0.0.1:49508)
1209 08:58:58.432797 (+ 269us) negotiation.cc:107] Waiting for socket to connect
1209 08:58:58.432810 (+ 13us) client_negotiation.cc:174] Beginning negotiation
1209 08:58:58.432888 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1209 08:58:58.433397 (+ 509us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1209 08:58:58.433400 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1209 08:58:58.433418 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1209 08:58:58.433646 (+ 228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:58:58.433654 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:58:58.434656 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1209 08:58:58.434659 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1209 08:58:58.435496 (+ 837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:58:58.435504 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:58:58.435602 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:58:58.435618 (+ 16us) client_negotiation.cc:769] Sending connection context
1209 08:58:58.435666 (+ 48us) client_negotiation.cc:240] Negotiation successful
1209 08:58:58.435723 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":105,"threads_started":1}
I20251209 08:58:58.436167 28778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:58:58.431901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43576 (local address 127.20.49.66:35715)
1209 08:58:58.432561 (+ 660us) server_negotiation.cc:207] Beginning negotiation
1209 08:58:58.432565 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1209 08:58:58.432577 (+ 12us) server_negotiation.cc:408] Connection header received
1209 08:58:58.432621 (+ 44us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1209 08:58:58.432623 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1209 08:58:58.432675 (+ 52us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1209 08:58:58.432777 (+ 102us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1209 08:58:58.433190 (+ 413us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:58:58.433945 (+ 755us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1209 08:58:58.435380 (+ 1435us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:58:58.435876 (+ 496us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:58:58.435913 (+ 37us) server_negotiation.cc:1036] Waiting for connection context
1209 08:58:58.435973 (+ 60us) server_negotiation.cc:300] Negotiation successful
1209 08:58:58.436043 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":571,"thread_start_us":98,"threads_started":1}
I20251209 08:58:58.436667 28782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:58:58.432884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49508 (local address 127.20.49.65:38709)
1209 08:58:58.433130 (+ 246us) server_negotiation.cc:207] Beginning negotiation
1209 08:58:58.433135 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1209 08:58:58.433148 (+ 13us) server_negotiation.cc:408] Connection header received
1209 08:58:58.433201 (+ 53us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1209 08:58:58.433207 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
1209 08:58:58.433257 (+ 50us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1209 08:58:58.433375 (+ 118us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1209 08:58:58.433815 (+ 440us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:58:58.434538 (+ 723us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1209 08:58:58.435606 (+ 1068us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:58:58.436201 (+ 595us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:58:58.436236 (+ 35us) server_negotiation.cc:1036] Waiting for connection context
1209 08:58:58.436507 (+ 271us) server_negotiation.cc:300] Negotiation successful
1209 08:58:58.436571 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":170,"thread_start_us":77,"threads_started":1}
I20251209 08:58:58.436713 28487 tablet_service.cc:1505] Processing CreateTablet for tablet d269cea41bf3481db8bc6ab9b79dc48a (DEFAULT_TABLE table=test-table [id=f954b4bef5e1492192bc0e2f04343010]), partition=RANGE (key) PARTITION UNBOUNDED
I20251209 08:58:58.436887 28780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:58:58.432395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38586 (local address 127.20.49.67:33177)
1209 08:58:58.433026 (+ 631us) server_negotiation.cc:207] Beginning negotiation
1209 08:58:58.433031 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1209 08:58:58.433045 (+ 14us) server_negotiation.cc:408] Connection header received
1209 08:58:58.433097 (+ 52us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1209 08:58:58.433100 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1209 08:58:58.433152 (+ 52us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1209 08:58:58.433245 (+ 93us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1209 08:58:58.433639 (+ 394us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:58:58.434443 (+ 804us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1209 08:58:58.436026 (+ 1583us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:58:58.436586 (+ 560us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:58:58.436623 (+ 37us) server_negotiation.cc:1036] Waiting for connection context
1209 08:58:58.436682 (+ 59us) server_negotiation.cc:300] Negotiation successful
1209 08:58:58.436757 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":544,"thread_start_us":61,"threads_started":1}
I20251209 08:58:58.437031 28487 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d269cea41bf3481db8bc6ab9b79dc48a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251209 08:58:58.437165 28779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:58:58.432218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.49.67:33177 (local address 127.0.0.1:38586)
1209 08:58:58.432517 (+ 299us) negotiation.cc:107] Waiting for socket to connect
1209 08:58:58.432531 (+ 14us) client_negotiation.cc:174] Beginning negotiation
1209 08:58:58.432619 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1209 08:58:58.433256 (+ 637us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1209 08:58:58.433259 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1209 08:58:58.433279 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1209 08:58:58.433510 (+ 231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:58:58.433516 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:58:58.434582 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1209 08:58:58.434588 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1209 08:58:58.435687 (+ 1099us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:58:58.435697 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:58:58.435804 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:58:58.435821 (+ 17us) client_negotiation.cc:769] Sending connection context
1209 08:58:58.435871 (+ 50us) client_negotiation.cc:240] Negotiation successful
1209 08:58:58.437057 (+ 1186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":109,"threads_started":1}
I20251209 08:58:58.437142 28351 tablet_service.cc:1505] Processing CreateTablet for tablet d269cea41bf3481db8bc6ab9b79dc48a (DEFAULT_TABLE table=test-table [id=f954b4bef5e1492192bc0e2f04343010]), partition=RANGE (key) PARTITION UNBOUNDED
I20251209 08:58:58.437414 28351 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d269cea41bf3481db8bc6ab9b79dc48a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251209 08:58:58.438472 28623 tablet_service.cc:1505] Processing CreateTablet for tablet d269cea41bf3481db8bc6ab9b79dc48a (DEFAULT_TABLE table=test-table [id=f954b4bef5e1492192bc0e2f04343010]), partition=RANGE (key) PARTITION UNBOUNDED
I20251209 08:58:58.438748 28623 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d269cea41bf3481db8bc6ab9b79dc48a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251209 08:58:58.439519 28783 tablet_bootstrap.cc:492] T d269cea41bf3481db8bc6ab9b79dc48a P c7b2b642c9144d568877db72b8553ac5: Bootstrap starting.
I20251209 08:58:58.439749 28784 tablet_bootstrap.cc:492] T d269cea41bf3481db8bc6ab9b79dc48a P 7182d3e0faba4ceaa1acc9b8b3674cbb: Bootstrap starting.
I20251209 08:58:58.440569 28784 tablet_bootstrap.cc:654] T d269cea41bf3481db8bc6ab9b79dc48a P 7182d3e0faba4ceaa1acc9b8b3674cbb: Neither blocks nor log segments found. Creating new log.
I20251209 08:58:58.440764 28785 tablet_bootstrap.cc:492] T d269cea41bf3481db8bc6ab9b79dc48a P 9787869327f04a04a23ea007848ae36b: Bootstrap starting.
I20251209 08:58:58.441735 28785 tablet_bootstrap.cc:654] T d269cea41bf3481db8bc6ab9b79dc48a P 9787869327f04a04a23ea007848ae36b: Neither blocks nor log segments found. Creating new log.
I20251209 08:58:58.441737 28783 tablet_bootstrap.cc:654] T d269cea41bf3481db8bc6ab9b79dc48a P c7b2b642c9144d568877db72b8553ac5: Neither blocks nor log segments found. Creating new log.
I20251209 08:58:58.442967 28784 tablet_bootstrap.cc:492] T d269cea41bf3481db8bc6ab9b79dc48a P 7182d3e0faba4ceaa1acc9b8b3674cbb: No bootstrap required, opened a new log
I20251209 08:58:58.443037 28785 tablet_bootstrap.cc:492] T d269cea41bf3481db8bc6ab9b79dc48a P 9787869327f04a04a23ea007848ae36b: No bootstrap required, opened a new log
I20251209 08:58:58.443079 28784 ts_tablet_manager.cc:1403] T d269cea41bf3481db8bc6ab9b79dc48a P 7182d3e0faba4ceaa1acc9b8b3674cbb: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20251209 08:58:58.443135 28785 ts_tablet_manager.cc:1403] T d269cea41bf3481db8bc6ab9b79dc48a P 9787869327f04a04a23ea007848ae36b: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20251209 08:58:58.443354 28783 tablet_bootstrap.cc:492] T d269cea41bf3481db8bc6ab9b79dc48a P c7b2b642c9144d568877db72b8553ac5: No bootstrap required, opened a new log
I20251209 08:58:58.443455 28783 ts_tablet_manager.cc:1403] T d269cea41bf3481db8bc6ab9b79dc48a P c7b2b642c9144d568877db72b8553ac5: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.001s
I20251209 08:58:58.443542 28785 raft_consensus.cc:359] T d269cea41bf3481db8bc6ab9b79dc48a P 9787869327f04a04a23ea007848ae36b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7182d3e0faba4ceaa1acc9b8b3674cbb" member_type: VOTER last_known_addr { host: "127.20.49.66" port: 35715 } } peers { permanent_uuid: "9787869327f04a04a23ea007848ae36b" member_type: VOTER last_known_addr { host: "127.20.49.67" port: 33177 } } peers { permanent_uuid: "c7b2b642c9144d568877db72b8553ac5" member_type: VOTER last_known_addr { host: "127.20.49.65" port: 38709 } }
I20251209 08:58:58.443645 28785 raft_consensus.cc:385] T d269cea41bf3481db8bc6ab9b79dc48a P 9787869327f04a04a23ea007848ae36b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251209 08:58:58.443656 28784 raft_consensus.cc:359] T d269cea41bf3481db8bc6ab9b79dc48a P 7182d3e0faba4ceaa1acc9b8b3674cbb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7182d3e0faba4ceaa1acc9b8b3674cbb" member_type: VOTER last_known_addr { host: "127.20.49.66" port: 35715 } } peers { permanent_uuid: "9787869327f04a04a23ea007848ae36b" member_type: VOTER last_known_addr { host: "127.20.49.67" port: 33177 } } peers { permanent_uuid: "c7b2b642c9144d568877db72b8553ac5" member_type: VOTER last_known_addr { host: "127.20.49.65" port: 38709 } }
I20251209 08:58:58.443708 28785 raft_consensus.cc:740] T d269cea41bf3481db8bc6ab9b79dc48a P 9787869327f04a04a23ea007848ae36b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9787869327f04a04a23ea007848ae36b, State: Initialized, Role: FOLLOWER
I20251209 08:58:58.443769 28784 raft_consensus.cc:385] T d269cea41bf3481db8bc6ab9b79dc48a P 7182d3e0faba4ceaa1acc9b8b3674cbb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251209 08:58:58.443811 28784 raft_consensus.cc:740] T d269cea41bf3481db8bc6ab9b79dc48a P 7182d3e0faba4ceaa1acc9b8b3674cbb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7182d3e0faba4ceaa1acc9b8b3674cbb, State: Initialized, Role: FOLLOWER
I20251209 08:58:58.443898 28785 consensus_queue.cc:260] T d269cea41bf3481db8bc6ab9b79dc48a P 9787869327f04a04a23ea007848ae36b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7182d3e0faba4ceaa1acc9b8b3674cbb" member_type: VOTER last_known_addr { host: "127.20.49.66" port: 35715 } } peers { permanent_uuid: "9787869327f04a04a23ea007848ae36b" member_type: VOTER last_known_addr { host: "127.20.49.67" port: 33177 } } peers { permanent_uuid: "c7b2b642c9144d568877db72b8553ac5" member_type: VOTER last_known_addr { host: "127.20.49.65" port: 38709 } }
I20251209 08:58:58.443919 28783 raft_consensus.cc:359] T d269cea41bf3481db8bc6ab9b79dc48a P c7b2b642c9144d568877db72b8553ac5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7182d3e0faba4ceaa1acc9b8b3674cbb" member_type: VOTER last_known_addr { host: "127.20.49.66" port: 35715 } } peers { permanent_uuid: "9787869327f04a04a23ea007848ae36b" member_type: VOTER last_known_addr { host: "127.20.49.67" port: 33177 } } peers { permanent_uuid: "c7b2b642c9144d568877db72b8553ac5" member_type: VOTER last_known_addr { host: "127.20.49.65" port: 38709 } }
I20251209 08:58:58.443948 28784 consensus_queue.cc:260] T d269cea41bf3481db8bc6ab9b79dc48a P 7182d3e0faba4ceaa1acc9b8b3674cbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7182d3e0faba4ceaa1acc9b8b3674cbb" member_type: VOTER last_known_addr { host: "127.20.49.66" port: 35715 } } peers { permanent_uuid: "9787869327f04a04a23ea007848ae36b" member_type: VOTER last_known_addr { host: "127.20.49.67" port: 33177 } } peers { permanent_uuid: "c7b2b642c9144d568877db72b8553ac5" member_type: VOTER last_known_addr { host: "127.20.49.65" port: 38709 } }
I20251209 08:58:58.444010 28783 raft_consensus.cc:385] T d269cea41bf3481db8bc6ab9b79dc48a P c7b2b642c9144d568877db72b8553ac5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251209 08:58:58.444048 28783 raft_consensus.cc:740] T d269cea41bf3481db8bc6ab9b79dc48a P c7b2b642c9144d568877db72b8553ac5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c7b2b642c9144d568877db72b8553ac5, State: Initialized, Role: FOLLOWER
I20251209 08:58:58.444237 28783 consensus_queue.cc:260] T d269cea41bf3481db8bc6ab9b79dc48a P c7b2b642c9144d568877db72b8553ac5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7182d3e0faba4ceaa1acc9b8b3674cbb" member_type: VOTER last_known_addr { host: "127.20.49.66" port: 35715 } } peers { permanent_uuid: "9787869327f04a04a23ea007848ae36b" member_type: VOTER last_known_addr { host: "127.20.49.67" port: 33177 } } peers { permanent_uuid: "c7b2b642c9144d568877db72b8553ac5" member_type: VOTER last_known_addr { host: "127.20.49.65" port: 38709 } }
I20251209 08:58:58.444310 28784 ts_tablet_manager.cc:1434] T d269cea41bf3481db8bc6ab9b79dc48a P 7182d3e0faba4ceaa1acc9b8b3674cbb: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20251209 08:58:58.444298 28785 ts_tablet_manager.cc:1434] T d269cea41bf3481db8bc6ab9b79dc48a P 9787869327f04a04a23ea007848ae36b: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20251209 08:58:58.445160 28764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:58:58.444675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.49.126:42449 (local address 127.20.49.66:35403)
1209 08:58:58.445025 (+ 350us) negotiation.cc:107] Waiting for socket to connect
1209 08:58:58.445081 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.49.126:42449: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":256}
I20251209 08:58:58.445204 28762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:58:58.444889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.49.126:42449 (local address 127.20.49.67:58725)
1209 08:58:58.445064 (+ 175us) negotiation.cc:107] Waiting for socket to connect
1209 08:58:58.445124 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.49.126:42449: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
I20251209 08:58:58.445433 28783 ts_tablet_manager.cc:1434] T d269cea41bf3481db8bc6ab9b79dc48a P c7b2b642c9144d568877db72b8553ac5: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20251209 08:58:58.446527 28763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:58:58.446135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.49.126:42449 (local address 127.20.49.65:58555)
1209 08:58:58.446399 (+ 264us) negotiation.cc:107] Waiting for socket to connect
1209 08:58:58.446451 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.49.126:42449: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186}
I20251209 08:58:58.447922 28763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:58:58.447637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.49.126:42449 (local address 127.20.49.65:54503)
1209 08:58:58.447808 (+ 171us) negotiation.cc:107] Waiting for socket to connect
1209 08:58:58.447858 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.49.126:42449: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20251209 08:58:58.448091 28764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:58:58.447849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.49.126:42449 (local address 127.20.49.66:35565)
1209 08:58:58.447964 (+ 115us) negotiation.cc:107] Waiting for socket to connect
1209 08:58:58.448026 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.49.126:42449: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20251209 08:58:58.448123 28418 heartbeater.cc:499] Master 127.20.49.124:44451 was elected leader, sending a full tablet report...
I20251209 08:58:58.448841 28554 heartbeater.cc:499] Master 127.20.49.124:44451 was elected leader, sending a full tablet report...
I20251209 08:58:58.449023 28690 heartbeater.cc:499] Master 127.20.49.124:44451 was elected leader, sending a full tablet report...
I20251209 08:58:58.449131 28762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:58:58.448874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.49.126:42449 (local address 127.20.49.67:49751)
1209 08:58:58.448988 (+ 114us) negotiation.cc:107] Waiting for socket to connect
1209 08:58:58.449057 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.49.126:42449: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20251209 08:58:58.465884 28692 tablet.cc:2378] T d269cea41bf3481db8bc6ab9b79dc48a P 9787869327f04a04a23ea007848ae36b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20251209 08:58:58.501658 28556 tablet.cc:2378] T d269cea41bf3481db8bc6ab9b79dc48a P 7182d3e0faba4ceaa1acc9b8b3674cbb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20251209 08:58:58.542819 28790 raft_consensus.cc:493] T d269cea41bf3481db8bc6ab9b79dc48a P 7182d3e0faba4ceaa1acc9b8b3674cbb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251209 08:58:58.542966 28790 raft_consensus.cc:515] T d269cea41bf3481db8bc6ab9b79dc48a P 7182d3e0faba4ceaa1acc9b8b3674cbb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7182d3e0faba4ceaa1acc9b8b3674cbb" member_type: VOTER last_known_addr { host: "127.20.49.66" port: 35715 } } peers { permanent_uuid: "9787869327f04a04a23ea007848ae36b" member_type: VOTER last_known_addr { host: "127.20.49.67" port: 33177 } } peers { permanent_uuid: "c7b2b642c9144d568877db72b8553ac5" member_type: VOTER last_known_addr { host: "127.20.49.65" port: 38709 } }
I20251209 08:58:58.544160 28790 leader_election.cc:290] T d269cea41bf3481db8bc6ab9b79dc48a P 7182d3e0faba4ceaa1acc9b8b3674cbb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9787869327f04a04a23ea007848ae36b (127.20.49.67:33177), c7b2b642c9144d568877db72b8553ac5 (127.20.49.65:38709)
I20251209 08:58:58.547034 28764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:58:58.544447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.49.67:33177 (local address 127.20.49.66:34837)
1209 08:58:58.544610 (+ 163us) negotiation.cc:107] Waiting for socket to connect
1209 08:58:58.544635 (+ 25us) client_negotiation.cc:174] Beginning negotiation
1209 08:58:58.544769 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1209 08:58:58.545026 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1209 08:58:58.545029 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1209 08:58:58.545059 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1209 08:58:58.545298 (+ 239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:58:58.545304 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:58:58.546053 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1209 08:58:58.546056 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1209 08:58:58.546634 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:58:58.546641 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:58:58.546754 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:58:58.546777 (+ 23us) client_negotiation.cc:769] Sending connection context
1209 08:58:58.546830 (+ 53us) client_negotiation.cc:240] Negotiation successful
1209 08:58:58.546895 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54,"spinlock_wait_cycles":3456}
I20251209 08:58:58.547231 28792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:58:58.544447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.49.65:38709 (local address 127.20.49.66:54609)
1209 08:58:58.544929 (+ 482us) negotiation.cc:107] Waiting for socket to connect
1209 08:58:58.544941 (+ 12us) client_negotiation.cc:174] Beginning negotiation
1209 08:58:58.545051 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1209 08:58:58.545281 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1209 08:58:58.545284 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1209 08:58:58.545300 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1209 08:58:58.545546 (+ 246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:58:58.545552 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:58:58.546281 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1209 08:58:58.546285 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1209 08:58:58.546854 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:58:58.546861 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:58:58.546971 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:58:58.546990 (+ 19us) client_negotiation.cc:769] Sending connection context
1209 08:58:58.547043 (+ 53us) client_negotiation.cc:240] Negotiation successful
1209 08:58:58.547095 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":415,"mutex_wait_us":129,"thread_start_us":160,"threads_started":1}
I20251209 08:58:58.547494 28780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:58:58.544550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.49.66:34837 (local address 127.20.49.67:33177)
1209 08:58:58.544730 (+ 180us) server_negotiation.cc:207] Beginning negotiation
1209 08:58:58.544735 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1209 08:58:58.544752 (+ 17us) server_negotiation.cc:408] Connection header received
1209 08:58:58.544893 (+ 141us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1209 08:58:58.544896 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1209 08:58:58.544940 (+ 44us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1209 08:58:58.545005 (+ 65us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1209 08:58:58.545410 (+ 405us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:58:58.545935 (+ 525us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1209 08:58:58.546751 (+ 816us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:58:58.547224 (+ 473us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:58:58.547262 (+ 38us) server_negotiation.cc:1036] Waiting for connection context
1209 08:58:58.547312 (+ 50us) server_negotiation.cc:300] Negotiation successful
1209 08:58:58.547363 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20251209 08:58:58.547731 28782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:58:58.544559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.49.66:54609 (local address 127.20.49.65:38709)
1209 08:58:58.544723 (+ 164us) server_negotiation.cc:207] Beginning negotiation
1209 08:58:58.544729 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1209 08:58:58.545024 (+ 295us) server_negotiation.cc:408] Connection header received
1209 08:58:58.545152 (+ 128us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1209 08:58:58.545156 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1209 08:58:58.545197 (+ 41us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1209 08:58:58.545263 (+ 66us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1209 08:58:58.545673 (+ 410us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:58:58.546177 (+ 504us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1209 08:58:58.546968 (+ 791us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:58:58.547466 (+ 498us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:58:58.547507 (+ 41us) server_negotiation.cc:1036] Waiting for connection context
1209 08:58:58.547560 (+ 53us) server_negotiation.cc:300] Negotiation successful
1209 08:58:58.547632 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20251209 08:58:58.547971 28642 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d269cea41bf3481db8bc6ab9b79dc48a" candidate_uuid: "7182d3e0faba4ceaa1acc9b8b3674cbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9787869327f04a04a23ea007848ae36b" is_pre_election: true
I20251209 08:58:58.548096 28370 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d269cea41bf3481db8bc6ab9b79dc48a" candidate_uuid: "7182d3e0faba4ceaa1acc9b8b3674cbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c7b2b642c9144d568877db72b8553ac5" is_pre_election: true
I20251209 08:58:58.548185 28642 raft_consensus.cc:2468] T d269cea41bf3481db8bc6ab9b79dc48a P 9787869327f04a04a23ea007848ae36b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7182d3e0faba4ceaa1acc9b8b3674cbb in term 0.
I20251209 08:58:58.548235 28370 raft_consensus.cc:2468] T d269cea41bf3481db8bc6ab9b79dc48a P c7b2b642c9144d568877db72b8553ac5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7182d3e0faba4ceaa1acc9b8b3674cbb in term 0.
I20251209 08:58:58.548578 28440 leader_election.cc:304] T d269cea41bf3481db8bc6ab9b79dc48a P 7182d3e0faba4ceaa1acc9b8b3674cbb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7182d3e0faba4ceaa1acc9b8b3674cbb, 9787869327f04a04a23ea007848ae36b; no voters:
I20251209 08:58:58.548807 28790 raft_consensus.cc:2804] T d269cea41bf3481db8bc6ab9b79dc48a P 7182d3e0faba4ceaa1acc9b8b3674cbb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251209 08:58:58.548870 28790 raft_consensus.cc:493] T d269cea41bf3481db8bc6ab9b79dc48a P 7182d3e0faba4ceaa1acc9b8b3674cbb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251209 08:58:58.548914 28790 raft_consensus.cc:3060] T d269cea41bf3481db8bc6ab9b79dc48a P 7182d3e0faba4ceaa1acc9b8b3674cbb [term 0 FOLLOWER]: Advancing to term 1
I20251209 08:58:58.549896 28790 raft_consensus.cc:515] T d269cea41bf3481db8bc6ab9b79dc48a P 7182d3e0faba4ceaa1acc9b8b3674cbb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7182d3e0faba4ceaa1acc9b8b3674cbb" member_type: VOTER last_known_addr { host: "127.20.49.66" port: 35715 } } peers { permanent_uuid: "9787869327f04a04a23ea007848ae36b" member_type: VOTER last_known_addr { host: "127.20.49.67" port: 33177 } } peers { permanent_uuid: "c7b2b642c9144d568877db72b8553ac5" member_type: VOTER last_known_addr { host: "127.20.49.65" port: 38709 } }
I20251209 08:58:58.550235 28790 leader_election.cc:290] T d269cea41bf3481db8bc6ab9b79dc48a P 7182d3e0faba4ceaa1acc9b8b3674cbb [CANDIDATE]: Term 1 election: Requested vote from peers 9787869327f04a04a23ea007848ae36b (127.20.49.67:33177), c7b2b642c9144d568877db72b8553ac5 (127.20.49.65:38709)
I20251209 08:58:58.550618 28370 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d269cea41bf3481db8bc6ab9b79dc48a" candidate_uuid: "7182d3e0faba4ceaa1acc9b8b3674cbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c7b2b642c9144d568877db72b8553ac5"
I20251209 08:58:58.550621 28642 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d269cea41bf3481db8bc6ab9b79dc48a" candidate_uuid: "7182d3e0faba4ceaa1acc9b8b3674cbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9787869327f04a04a23ea007848ae36b"
I20251209 08:58:58.550717 28370 raft_consensus.cc:3060] T d269cea41bf3481db8bc6ab9b79dc48a P c7b2b642c9144d568877db72b8553ac5 [term 0 FOLLOWER]: Advancing to term 1
I20251209 08:58:58.550755 28642 raft_consensus.cc:3060] T d269cea41bf3481db8bc6ab9b79dc48a P 9787869327f04a04a23ea007848ae36b [term 0 FOLLOWER]: Advancing to term 1
I20251209 08:58:58.551512 28370 raft_consensus.cc:2468] T d269cea41bf3481db8bc6ab9b79dc48a P c7b2b642c9144d568877db72b8553ac5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7182d3e0faba4ceaa1acc9b8b3674cbb in term 1.
I20251209 08:58:58.551692 28642 raft_consensus.cc:2468] T d269cea41bf3481db8bc6ab9b79dc48a P 9787869327f04a04a23ea007848ae36b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7182d3e0faba4ceaa1acc9b8b3674cbb in term 1.
I20251209 08:58:58.551781 28443 leader_election.cc:304] T d269cea41bf3481db8bc6ab9b79dc48a P 7182d3e0faba4ceaa1acc9b8b3674cbb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7182d3e0faba4ceaa1acc9b8b3674cbb, c7b2b642c9144d568877db72b8553ac5; no voters:
I20251209 08:58:58.551968 28790 raft_consensus.cc:2804] T d269cea41bf3481db8bc6ab9b79dc48a P 7182d3e0faba4ceaa1acc9b8b3674cbb [term 1 FOLLOWER]: Leader election won for term 1
I20251209 08:58:58.552163 28790 raft_consensus.cc:697] T d269cea41bf3481db8bc6ab9b79dc48a P 7182d3e0faba4ceaa1acc9b8b3674cbb [term 1 LEADER]: Becoming Leader. State: Replica: 7182d3e0faba4ceaa1acc9b8b3674cbb, State: Running, Role: LEADER
I20251209 08:58:58.552392 28790 consensus_queue.cc:237] T d269cea41bf3481db8bc6ab9b79dc48a P 7182d3e0faba4ceaa1acc9b8b3674cbb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7182d3e0faba4ceaa1acc9b8b3674cbb" member_type: VOTER last_known_addr { host: "127.20.49.66" port: 35715 } } peers { permanent_uuid: "9787869327f04a04a23ea007848ae36b" member_type: VOTER last_known_addr { host: "127.20.49.67" port: 33177 } } peers { permanent_uuid: "c7b2b642c9144d568877db72b8553ac5" member_type: VOTER last_known_addr { host: "127.20.49.65" port: 38709 } }
I20251209 08:58:58.552933 28792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:58:58.552698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.49.126:42449 (local address 127.20.49.66:40317)
1209 08:58:58.552809 (+ 111us) negotiation.cc:107] Waiting for socket to connect
1209 08:58:58.552861 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.49.126:42449: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20251209 08:58:58.554924 28222 catalog_manager.cc:5654] T d269cea41bf3481db8bc6ab9b79dc48a P 7182d3e0faba4ceaa1acc9b8b3674cbb reported cstate change: term changed from 0 to 1, leader changed from <none> to 7182d3e0faba4ceaa1acc9b8b3674cbb (127.20.49.66). New cstate: current_term: 1 leader_uuid: "7182d3e0faba4ceaa1acc9b8b3674cbb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7182d3e0faba4ceaa1acc9b8b3674cbb" member_type: VOTER last_known_addr { host: "127.20.49.66" port: 35715 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9787869327f04a04a23ea007848ae36b" member_type: VOTER last_known_addr { host: "127.20.49.67" port: 33177 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c7b2b642c9144d568877db72b8553ac5" member_type: VOTER last_known_addr { host: "127.20.49.65" port: 38709 } health_report { overall_health: UNKNOWN } } }
I20251209 08:58:58.593429 28778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:58:58.589902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43580 (local address 127.20.49.66:35715)
1209 08:58:58.590043 (+ 141us) server_negotiation.cc:207] Beginning negotiation
1209 08:58:58.590047 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1209 08:58:58.590062 (+ 15us) server_negotiation.cc:408] Connection header received
1209 08:58:58.590206 (+ 144us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1209 08:58:58.590210 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1209 08:58:58.590279 (+ 69us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1209 08:58:58.590382 (+ 103us) server_negotiation.cc:227] Negotiated authn=TOKEN
1209 08:58:58.590835 (+ 453us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:58:58.591549 (+ 714us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1209 08:58:58.592223 (+ 674us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:58:58.592425 (+ 202us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:58:58.592485 (+ 60us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1209 08:58:58.593008 (+ 523us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1209 08:58:58.593100 (+ 92us) server_negotiation.cc:1036] Waiting for connection context
1209 08:58:58.593255 (+ 155us) server_negotiation.cc:300] Negotiation successful
1209 08:58:58.593316 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20251209 08:58:58.596318 28370 raft_consensus.cc:1275] T d269cea41bf3481db8bc6ab9b79dc48a P c7b2b642c9144d568877db72b8553ac5 [term 1 FOLLOWER]: Refusing update from remote peer 7182d3e0faba4ceaa1acc9b8b3674cbb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251209 08:58:58.596668 28642 raft_consensus.cc:1275] T d269cea41bf3481db8bc6ab9b79dc48a P 9787869327f04a04a23ea007848ae36b [term 1 FOLLOWER]: Refusing update from remote peer 7182d3e0faba4ceaa1acc9b8b3674cbb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251209 08:58:58.596832 28793 consensus_queue.cc:1048] T d269cea41bf3481db8bc6ab9b79dc48a P 7182d3e0faba4ceaa1acc9b8b3674cbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "c7b2b642c9144d568877db72b8553ac5" member_type: VOTER last_known_addr { host: "127.20.49.65" port: 38709 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251209 08:58:58.597168 28790 consensus_queue.cc:1048] T d269cea41bf3481db8bc6ab9b79dc48a P 7182d3e0faba4ceaa1acc9b8b3674cbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "9787869327f04a04a23ea007848ae36b" member_type: VOTER last_known_addr { host: "127.20.49.67" port: 33177 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251209 08:58:58.599031 28762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:58:58.598716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.49.126:42449 (local address 127.20.49.67:59569)
1209 08:58:58.598893 (+ 177us) negotiation.cc:107] Waiting for socket to connect
1209 08:58:58.598952 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.49.126:42449: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98}
I20251209 08:58:58.599782 28763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:58:58.599221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.49.126:42449 (local address 127.20.49.65:58889)
1209 08:58:58.599368 (+ 147us) negotiation.cc:107] Waiting for socket to connect
1209 08:58:58.599421 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.49.126:42449: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20251209 08:58:58.603394 28792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:58:58.602827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.49.126:42449 (local address 127.20.49.66:34541)
1209 08:58:58.603266 (+ 439us) negotiation.cc:107] Waiting for socket to connect
1209 08:58:58.603320 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.49.126:42449: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":369}
I20251209 08:58:58.604200 28762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:58:58.603913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.49.126:42449 (local address 127.20.49.67:48947)
1209 08:58:58.604072 (+ 159us) negotiation.cc:107] Waiting for socket to connect
1209 08:58:58.604122 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.49.126:42449: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
I20251209 08:58:58.604346 28763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:58:58.604027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.49.126:42449 (local address 127.20.49.65:47513)
1209 08:58:58.604223 (+ 196us) negotiation.cc:107] Waiting for socket to connect
1209 08:58:58.604275 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.49.126:42449: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127}
I20251209 08:58:58.605811 28792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:58:58.605501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.49.126:42449 (local address 127.20.49.66:35257)
1209 08:58:58.605658 (+ 157us) negotiation.cc:107] Waiting for socket to connect
1209 08:58:58.605742 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.49.126:42449: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20251209 08:58:58.611382 28222 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56830:
table { table_name: "test-table" } modify_external_catalogs: true
I20251209 08:58:58.611526 28222 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56830:
table { table_name: "test-table" } modify_external_catalogs: true
I20251209 08:58:58.615037 28222 catalog_manager.cc:5941] T 00000000000000000000000000000000 P 0364ed40f171452f9fa244fa50f6a6c3: Sending DeleteTablet for 3 replicas of tablet d269cea41bf3481db8bc6ab9b79dc48a
I20251209 08:58:58.615723 28623 tablet_service.cc:1552] Processing DeleteTablet for tablet d269cea41bf3481db8bc6ab9b79dc48a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-09 08:58:58 UTC) from {username='oryx', principal='oryx/127.20.49.124@KRBTEST.COM'} at 127.0.0.1:38586
I20251209 08:58:58.615959 28351 tablet_service.cc:1552] Processing DeleteTablet for tablet d269cea41bf3481db8bc6ab9b79dc48a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-09 08:58:58 UTC) from {username='oryx', principal='oryx/127.20.49.124@KRBTEST.COM'} at 127.0.0.1:49508
I20251209 08:58:58.616171 28807 tablet_replica.cc:333] T d269cea41bf3481db8bc6ab9b79dc48a P 9787869327f04a04a23ea007848ae36b: stopping tablet replica
I20251209 08:58:58.616344 28807 raft_consensus.cc:2243] T d269cea41bf3481db8bc6ab9b79dc48a P 9787869327f04a04a23ea007848ae36b [term 1 FOLLOWER]: Raft consensus shutting down.
I20251209 08:58:58.616385 28808 tablet_replica.cc:333] T d269cea41bf3481db8bc6ab9b79dc48a P c7b2b642c9144d568877db72b8553ac5: stopping tablet replica
I20251209 08:58:58.616418 28487 tablet_service.cc:1552] Processing DeleteTablet for tablet d269cea41bf3481db8bc6ab9b79dc48a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-09 08:58:58 UTC) from {username='oryx', principal='oryx/127.20.49.124@KRBTEST.COM'} at 127.0.0.1:43576
I20251209 08:58:58.616490 28808 raft_consensus.cc:2243] T d269cea41bf3481db8bc6ab9b79dc48a P c7b2b642c9144d568877db72b8553ac5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251209 08:58:58.616636 28808 raft_consensus.cc:2272] T d269cea41bf3481db8bc6ab9b79dc48a P c7b2b642c9144d568877db72b8553ac5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251209 08:58:58.616644 28807 raft_consensus.cc:2272] T d269cea41bf3481db8bc6ab9b79dc48a P 9787869327f04a04a23ea007848ae36b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251209 08:58:58.617419 20677 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task8XuDZV/build/debug/bin/kudu with pid 28288
I20251209 08:58:58.617426 28807 ts_tablet_manager.cc:1916] T d269cea41bf3481db8bc6ab9b79dc48a P 9787869327f04a04a23ea007848ae36b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251209 08:58:58.618191 28809 tablet_replica.cc:333] T d269cea41bf3481db8bc6ab9b79dc48a P 7182d3e0faba4ceaa1acc9b8b3674cbb: stopping tablet replica
I20251209 08:58:58.618320 28809 raft_consensus.cc:2243] T d269cea41bf3481db8bc6ab9b79dc48a P 7182d3e0faba4ceaa1acc9b8b3674cbb [term 1 LEADER]: Raft consensus shutting down.
I20251209 08:58:58.618665 28809 raft_consensus.cc:2272] T d269cea41bf3481db8bc6ab9b79dc48a P 7182d3e0faba4ceaa1acc9b8b3674cbb [term 1 FOLLOWER]: Raft consensus is shut down!
I20251209 08:58:58.619369 28809 ts_tablet_manager.cc:1916] T d269cea41bf3481db8bc6ab9b79dc48a P 7182d3e0faba4ceaa1acc9b8b3674cbb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251209 08:58:58.621271 28807 ts_tablet_manager.cc:1929] T d269cea41bf3481db8bc6ab9b79dc48a P 9787869327f04a04a23ea007848ae36b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251209 08:58:58.621403 28807 log.cc:1199] T d269cea41bf3481db8bc6ab9b79dc48a P 9787869327f04a04a23ea007848ae36b: Deleting WAL directory at /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765270668420570-20677-0/minicluster-data/ts-2/wal/wals/d269cea41bf3481db8bc6ab9b79dc48a
I20251209 08:58:58.621824 28807 ts_tablet_manager.cc:1950] T d269cea41bf3481db8bc6ab9b79dc48a P 9787869327f04a04a23ea007848ae36b: Deleting consensus metadata
I20251209 08:58:58.621922 28809 ts_tablet_manager.cc:1929] T d269cea41bf3481db8bc6ab9b79dc48a P 7182d3e0faba4ceaa1acc9b8b3674cbb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251209 08:58:58.621997 28809 log.cc:1199] T d269cea41bf3481db8bc6ab9b79dc48a P 7182d3e0faba4ceaa1acc9b8b3674cbb: Deleting WAL directory at /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1765270668420570-20677-0/minicluster-data/ts-1/wal/wals/d269cea41bf3481db8bc6ab9b79dc48a
I20251209 08:58:58.622298 28809 ts_tablet_manager.cc:1950] T d269cea41bf3481db8bc6ab9b79dc48a P 7182d3e0faba4ceaa1acc9b8b3674cbb: Deleting consensus metadata
I20251209 08:58:58.622925 28209 catalog_manager.cc:4985] TS 7182d3e0faba4ceaa1acc9b8b3674cbb (127.20.49.66:35715): tablet d269cea41bf3481db8bc6ab9b79dc48a (table test-table [id=f954b4bef5e1492192bc0e2f04343010]) successfully deleted
I20251209 08:58:58.622891 28206 catalog_manager.cc:4985] TS 9787869327f04a04a23ea007848ae36b (127.20.49.67:33177): tablet d269cea41bf3481db8bc6ab9b79dc48a (table test-table [id=f954b4bef5e1492192bc0e2f04343010]) successfully deleted
I20251209 08:58:58.628212 20677 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task8XuDZV/build/debug/bin/kudu with pid 28424
I20251209 08:58:58.628242 28779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:58:58.627996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.49.65:38709 (local address 127.0.0.1:49510)
1209 08:58:58.628117 (+ 121us) negotiation.cc:107] Waiting for socket to connect
1209 08:58:58.628166 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.49.65:38709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20251209 08:58:58.629696 28209 catalog_manager.cc:4712] TS c7b2b642c9144d568877db72b8553ac5 (127.20.49.65:38709): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d269cea41bf3481db8bc6ab9b79dc48a: Network error: Client connection negotiation failed: client connection to 127.20.49.65:38709: connect: Connection refused (error 111)
I20251209 08:58:58.634058 20677 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task8XuDZV/build/debug/bin/kudu with pid 28560
I20251209 08:58:58.641399 20677 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task8XuDZV/build/debug/bin/kudu with pid 27899
I20251209 08:58:58.647804 20677 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task8XuDZV/build/debug/bin/kudu with pid 28020
2025-12-09T08:58:58Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (9282 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765270668420570-20677-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Dec 09 08:58:58 dist-test-slave-8w6c krb5kdc[28812](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Dec 09 08:58:58 dist-test-slave-8w6c krb5kdc[28812](info): set up 2 sockets
Dec 09 08:58:58 dist-test-slave-8w6c krb5kdc[28812](info): commencing operation
krb5kdc: starting...
W20251209 08:59:00.650810 20677 mini_kdc.cc:121] Time spent starting KDC: real 1.978s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765270668420570-20677-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765270668420570-20677-0/krb5kdc/test-user.keytab.
Dec 09 08:59:00 dist-test-slave-8w6c krb5kdc[28812](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765270740, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-12-09T08:59:00Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-12-09T08:59:00Z Disabled control of system clock
WARNING: no policy specified for oryx/127.20.49.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.20.49.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.20.49.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765270668420570-20677-0/krb5kdc/oryx.keytab.
Entry for principal oryx/127.20.49.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765270668420570-20677-0/krb5kdc/oryx.keytab.
WARNING: no policy specified for HTTP/127.20.49.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.20.49.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.20.49.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765270668420570-20677-0/krb5kdc/oryx.keytab.
Entry for principal HTTP/127.20.49.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765270668420570-20677-0/krb5kdc/oryx.keytab.
I20251209 08:59:00.797816 20677 external_mini_cluster.cc:1366] Running /tmp/dist-test-task8XuDZV/build/debug/bin/kudu
/tmp/dist-test-task8XuDZV/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765270668420570-20677-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765270668420570-20677-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765270668420570-20677-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765270668420570-20677-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.49.126:41745
--webserver_interface=127.20.49.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765270668420570-20677-0/krb5kdc/oryx.keytab
--principal=oryx/127.20.49.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.20.49.84:44369
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.49.126:41745
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765270668420570-20677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765270668420570-20677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765270668420570-20677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251209 08:59:00.908627 28828 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251209 08:59:00.908883 28828 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251209 08:59:00.908952 28828 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251209 08:59:00.912335 28828 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251209 08:59:00.912410 28828 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20251209 08:59:00.912429 28828 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251209 08:59:00.912451 28828 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251209 08:59:00.912478 28828 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251209 08:59:00.912518 28828 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251209 08:59:00.916894 28828 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.49.84:44369
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765270668420570-20677-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765270668420570-20677-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.49.126:41745
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765270668420570-20677-0/krb5kdc/oryx.keytab
--principal=oryx/127.20.49.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.49.126:41745
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765270668420570-20677-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.49.126
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28828
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765270668420570-20677-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 468897755969d686fb08386f42f7835685e7953a
build type DEBUG
built by None at 09 Dec 2025 08:43:15 UTC on 24a791456cd2
build id 9426
I20251209 08:59:00.918046 28828 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251209 08:59:00.918939 28828 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251209 08:59:00.924922 28834 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251209 08:59:00.924924 28833 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251209 08:59:00.924922 28836 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251209 08:59:00.925117 28828 server_base.cc:1047] running on GCE node
I20251209 08:59:00.925751 28828 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251209 08:59:00.926644 28828 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251209 08:59:00.927810 28828 hybrid_clock.cc:648] HybridClock initialized: now 1765270740927792 us; error 39 us; skew 500 ppm
Dec 09 08:59:00 dist-test-slave-8w6c krb5kdc[28812](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765270740, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.49.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251209 08:59:00.930531 28828 init.cc:377] Logged in from keytab as oryx/127.20.49.126@KRBTEST.COM (short username oryx)
I20251209 08:59:00.931632 28828 webserver.cc:492] Webserver started at http://127.20.49.126:44723/ using document root <none> and password file <none>
I20251209 08:59:00.932173 28828 fs_manager.cc:362] Metadata directory not provided
I20251209 08:59:00.932227 28828 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251209 08:59:00.932410 28828 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251209 08:59:00.934180 28828 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765270668420570-20677-0/minicluster-data/master-0/data/instance:
uuid: "7f2aac7cb38e4c609b71560ad54a20b1"
format_stamp: "Formatted at 2025-12-09 08:59:00 on dist-test-slave-8w6c"
server_key: "11015ecad542a720d0e9b8d1f7480a0a"
server_key_iv: "6dba302a8f81f4c0e925afed6c6ce5f0"
server_key_version: "encryptionkey@0"
I20251209 08:59:00.934669 28828 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765270668420570-20677-0/minicluster-data/master-0/wal/instance:
uuid: "7f2aac7cb38e4c609b71560ad54a20b1"
format_stamp: "Formatted at 2025-12-09 08:59:00 on dist-test-slave-8w6c"
server_key: "11015ecad542a720d0e9b8d1f7480a0a"
server_key_iv: "6dba302a8f81f4c0e925afed6c6ce5f0"
server_key_version: "encryptionkey@0"
I20251209 08:59:00.938045 28828 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20251209 08:59:00.940327 28843 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251209 08:59:00.941293 28828 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20251209 08:59:00.941402 28828 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765270668420570-20677-0/minicluster-data/master-0/data,/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765270668420570-20677-0/minicluster-data/master-0/wal
uuid: "7f2aac7cb38e4c609b71560ad54a20b1"
format_stamp: "Formatted at 2025-12-09 08:59:00 on dist-test-slave-8w6c"
server_key: "11015ecad542a720d0e9b8d1f7480a0a"
server_key_iv: "6dba302a8f81f4c0e925afed6c6ce5f0"
server_key_version: "encryptionkey@0"
I20251209 08:59:00.941496 28828 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765270668420570-20677-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765270668420570-20677-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765270668420570-20677-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251209 08:59:00.961057 28828 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251209 08:59:00.964139 28828 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251209 08:59:00.964337 28828 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251209 08:59:00.972678 28828 rpc_server.cc:307] RPC server started. Bound to: 127.20.49.126:41745
I20251209 08:59:00.972999 28905 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.49.126:41745 every 8 connection(s)
I20251209 08:59:00.973793 28828 server_base.cc:1179] Dumped server information to /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765270668420570-20677-0/minicluster-data/master-0/data/info.pb
I20251209 08:59:00.974174 20677 external_mini_cluster.cc:1428] Started /tmp/dist-test-task8XuDZV/build/debug/bin/kudu as pid 28828
I20251209 08:59:00.974305 20677 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765270668420570-20677-0/minicluster-data/master-0/wal/instance
I20251209 08:59:00.974563 20677 external_mini_cluster.cc:1468] Setting key 3b2b74e0ff688d0afac392fbdd622020
I20251209 08:59:00.977202 28906 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Dec 09 08:59:00 dist-test-slave-8w6c krb5kdc[28812](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765270740, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.20.49.126@KRBTEST.COM
I20251209 08:59:00.983247 28906 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7f2aac7cb38e4c609b71560ad54a20b1: Bootstrap starting.
I20251209 08:59:00.985911 28906 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7f2aac7cb38e4c609b71560ad54a20b1: Neither blocks nor log segments found. Creating new log.
I20251209 08:59:00.986742 28906 log.cc:826] T 00000000000000000000000000000000 P 7f2aac7cb38e4c609b71560ad54a20b1: Log is configured to *not* fsync() on all Append() calls
I20251209 08:59:00.988217 28909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:59:00.975818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59154 (local address 127.20.49.126:41745)
1209 08:59:00.976228 (+ 410us) server_negotiation.cc:207] Beginning negotiation
1209 08:59:00.976240 (+ 12us) server_negotiation.cc:400] Waiting for connection header
1209 08:59:00.976281 (+ 41us) server_negotiation.cc:408] Connection header received
1209 08:59:00.977111 (+ 830us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1209 08:59:00.977136 (+ 25us) server_negotiation.cc:462] Received NEGOTIATE request from client
1209 08:59:00.977505 (+ 369us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1209 08:59:00.977913 (+ 408us) server_negotiation.cc:227] Negotiated authn=SASL
1209 08:59:00.978827 (+ 914us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:59:00.980015 (+ 1188us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1209 08:59:00.980758 (+ 743us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:59:00.981037 (+ 279us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:59:00.983417 (+ 2380us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1209 08:59:00.983467 (+ 50us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1209 08:59:00.983480 (+ 13us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1209 08:59:00.983520 (+ 40us) server_negotiation.cc:923] calling sasl_server_start()
1209 08:59:00.985607 (+ 2087us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1209 08:59:00.986089 (+ 482us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1209 08:59:00.986098 (+ 9us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1209 08:59:00.986106 (+ 8us) server_negotiation.cc:968] calling sasl_server_step()
1209 08:59:00.986203 (+ 97us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1209 08:59:00.986400 (+ 197us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1209 08:59:00.986404 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1209 08:59:00.986406 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1209 08:59:00.986835 (+ 429us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1209 08:59:00.987037 (+ 202us) server_negotiation.cc:1036] Waiting for connection context
1209 08:59:00.987233 (+ 196us) server_negotiation.cc:300] Negotiation successful
1209 08:59:00.987487 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":267,"thread_start_us":115,"threads_started":1}
I20251209 08:59:00.988806 28906 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7f2aac7cb38e4c609b71560ad54a20b1: No bootstrap required, opened a new log
I20251209 08:59:00.991694 28906 raft_consensus.cc:359] T 00000000000000000000000000000000 P 7f2aac7cb38e4c609b71560ad54a20b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f2aac7cb38e4c609b71560ad54a20b1" member_type: VOTER last_known_addr { host: "127.20.49.126" port: 41745 } }
I20251209 08:59:00.991875 28906 raft_consensus.cc:385] T 00000000000000000000000000000000 P 7f2aac7cb38e4c609b71560ad54a20b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251209 08:59:00.991919 28906 raft_consensus.cc:740] T 00000000000000000000000000000000 P 7f2aac7cb38e4c609b71560ad54a20b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f2aac7cb38e4c609b71560ad54a20b1, State: Initialized, Role: FOLLOWER
I20251209 08:59:00.992287 28906 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7f2aac7cb38e4c609b71560ad54a20b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f2aac7cb38e4c609b71560ad54a20b1" member_type: VOTER last_known_addr { host: "127.20.49.126" port: 41745 } }
I20251209 08:59:00.992405 28906 raft_consensus.cc:399] T 00000000000000000000000000000000 P 7f2aac7cb38e4c609b71560ad54a20b1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251209 08:59:00.992451 28906 raft_consensus.cc:493] T 00000000000000000000000000000000 P 7f2aac7cb38e4c609b71560ad54a20b1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251209 08:59:00.992518 28906 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 7f2aac7cb38e4c609b71560ad54a20b1 [term 0 FOLLOWER]: Advancing to term 1
I20251209 08:59:00.993315 28906 raft_consensus.cc:515] T 00000000000000000000000000000000 P 7f2aac7cb38e4c609b71560ad54a20b1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f2aac7cb38e4c609b71560ad54a20b1" member_type: VOTER last_known_addr { host: "127.20.49.126" port: 41745 } }
I20251209 08:59:00.993624 28906 leader_election.cc:304] T 00000000000000000000000000000000 P 7f2aac7cb38e4c609b71560ad54a20b1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7f2aac7cb38e4c609b71560ad54a20b1; no voters:
I20251209 08:59:00.993925 28906 leader_election.cc:290] T 00000000000000000000000000000000 P 7f2aac7cb38e4c609b71560ad54a20b1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20251209 08:59:00.994035 28911 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 7f2aac7cb38e4c609b71560ad54a20b1 [term 1 FOLLOWER]: Leader election won for term 1
I20251209 08:59:00.994239 28911 raft_consensus.cc:697] T 00000000000000000000000000000000 P 7f2aac7cb38e4c609b71560ad54a20b1 [term 1 LEADER]: Becoming Leader. State: Replica: 7f2aac7cb38e4c609b71560ad54a20b1, State: Running, Role: LEADER
I20251209 08:59:00.994545 28911 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7f2aac7cb38e4c609b71560ad54a20b1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f2aac7cb38e4c609b71560ad54a20b1" member_type: VOTER last_known_addr { host: "127.20.49.126" port: 41745 } }
I20251209 08:59:00.995055 28906 sys_catalog.cc:565] T 00000000000000000000000000000000 P 7f2aac7cb38e4c609b71560ad54a20b1 [sys.catalog]: configured and running, proceeding with master startup.
I20251209 08:59:00.996192 28912 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f2aac7cb38e4c609b71560ad54a20b1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7f2aac7cb38e4c609b71560ad54a20b1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f2aac7cb38e4c609b71560ad54a20b1" member_type: VOTER last_known_addr { host: "127.20.49.126" port: 41745 } } }
I20251209 08:59:00.996315 28912 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f2aac7cb38e4c609b71560ad54a20b1 [sys.catalog]: This master's current role is: LEADER
I20251209 08:59:00.996578 28913 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f2aac7cb38e4c609b71560ad54a20b1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7f2aac7cb38e4c609b71560ad54a20b1. Latest consensus state: current_term: 1 leader_uuid: "7f2aac7cb38e4c609b71560ad54a20b1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f2aac7cb38e4c609b71560ad54a20b1" member_type: VOTER last_known_addr { host: "127.20.49.126" port: 41745 } } }
I20251209 08:59:00.996663 28913 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f2aac7cb38e4c609b71560ad54a20b1 [sys.catalog]: This master's current role is: LEADER
I20251209 08:59:00.996748 28920 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251209 08:59:00.999583 28920 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251209 08:59:01.004536 28920 catalog_manager.cc:1357] Generated new cluster ID: 9ff9b7a780dc47f7b2884c68b057454a
I20251209 08:59:01.004613 28920 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251209 08:59:01.021533 28920 catalog_manager.cc:1380] Generated new certificate authority record
I20251209 08:59:01.022943 28920 catalog_manager.cc:1514] Loading token signing keys...
I20251209 08:59:01.033481 28920 catalog_manager.cc:6027] T 00000000000000000000000000000000 P 7f2aac7cb38e4c609b71560ad54a20b1: Generated new TSK 0
I20251209 08:59:01.034409 28920 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.20.49.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.20.49.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.20.49.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765270668420570-20677-0/krb5kdc/oryx.keytab.
Entry for principal oryx/127.20.49.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765270668420570-20677-0/krb5kdc/oryx.keytab.
WARNING: no policy specified for HTTP/127.20.49.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.20.49.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.20.49.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765270668420570-20677-0/krb5kdc/oryx.keytab.
Entry for principal HTTP/127.20.49.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765270668420570-20677-0/krb5kdc/oryx.keytab.
I20251209 08:59:01.096647 20677 external_mini_cluster.cc:1366] Running /tmp/dist-test-task8XuDZV/build/debug/bin/kudu
/tmp/dist-test-task8XuDZV/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765270668420570-20677-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765270668420570-20677-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765270668420570-20677-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765270668420570-20677-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.49.65:0
--local_ip_for_outbound_sockets=127.20.49.65
--webserver_interface=127.20.49.65
--webserver_port=0
--tserver_master_addrs=127.20.49.126:41745
--keytab_file=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765270668420570-20677-0/krb5kdc/oryx.keytab
--principal=oryx/127.20.49.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.20.49.84:44369
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765270668420570-20677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765270668420570-20677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765270668420570-20677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251209 08:59:01.208438 28934 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251209 08:59:01.208664 28934 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251209 08:59:01.208726 28934 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251209 08:59:01.212004 28934 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251209 08:59:01.212098 28934 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251209 08:59:01.212180 28934 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20251209 08:59:01.212241 28934 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.49.65
I20251209 08:59:01.216584 28934 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.49.84:44369
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765270668420570-20677-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765270668420570-20677-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765270668420570-20677-0/krb5kdc/oryx.keytab
--principal=oryx/127.20.49.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.49.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765270668420570-20677-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.49.65
--webserver_port=0
--webserver_require_spnego=true
--enable_txn_system_client_init=true
--tserver_master_addrs=127.20.49.126:41745
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28934
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.49.65
--log_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765270668420570-20677-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 468897755969d686fb08386f42f7835685e7953a
build type DEBUG
built by None at 09 Dec 2025 08:43:15 UTC on 24a791456cd2
build id 9426
I20251209 08:59:01.217700 28934 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251209 08:59:01.218583 28934 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251209 08:59:01.225427 28939 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251209 08:59:01.225423 28940 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251209 08:59:01.225427 28942 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251209 08:59:01.225787 28934 server_base.cc:1047] running on GCE node
I20251209 08:59:01.226148 28934 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251209 08:59:01.226689 28934 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251209 08:59:01.227828 28934 hybrid_clock.cc:648] HybridClock initialized: now 1765270741227807 us; error 39 us; skew 500 ppm
Dec 09 08:59:01 dist-test-slave-8w6c krb5kdc[28812](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765270741, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.49.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251209 08:59:01.230479 28934 init.cc:377] Logged in from keytab as oryx/127.20.49.65@KRBTEST.COM (short username oryx)
I20251209 08:59:01.231521 28934 webserver.cc:492] Webserver started at http://127.20.49.65:36307/ using document root <none> and password file <none>
I20251209 08:59:01.232079 28934 fs_manager.cc:362] Metadata directory not provided
I20251209 08:59:01.232134 28934 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251209 08:59:01.232333 28934 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251209 08:59:01.234110 28934 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765270668420570-20677-0/minicluster-data/ts-0/data/instance:
uuid: "3cbbcfcfb7634dc588875d9c22e2328e"
format_stamp: "Formatted at 2025-12-09 08:59:01 on dist-test-slave-8w6c"
server_key: "e6a6883879bae62e1c014139c6a73cd7"
server_key_iv: "50222873ef363f931a39fc349701b883"
server_key_version: "encryptionkey@0"
I20251209 08:59:01.234575 28934 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765270668420570-20677-0/minicluster-data/ts-0/wal/instance:
uuid: "3cbbcfcfb7634dc588875d9c22e2328e"
format_stamp: "Formatted at 2025-12-09 08:59:01 on dist-test-slave-8w6c"
server_key: "e6a6883879bae62e1c014139c6a73cd7"
server_key_iv: "50222873ef363f931a39fc349701b883"
server_key_version: "encryptionkey@0"
I20251209 08:59:01.237990 28934 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20251209 08:59:01.240206 28949 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251209 08:59:01.241189 28934 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20251209 08:59:01.241293 28934 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765270668420570-20677-0/minicluster-data/ts-0/data,/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765270668420570-20677-0/minicluster-data/ts-0/wal
uuid: "3cbbcfcfb7634dc588875d9c22e2328e"
format_stamp: "Formatted at 2025-12-09 08:59:01 on dist-test-slave-8w6c"
server_key: "e6a6883879bae62e1c014139c6a73cd7"
server_key_iv: "50222873ef363f931a39fc349701b883"
server_key_version: "encryptionkey@0"
I20251209 08:59:01.241380 28934 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765270668420570-20677-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765270668420570-20677-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765270668420570-20677-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251209 08:59:01.256883 28934 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251209 08:59:01.259955 28934 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251209 08:59:01.260140 28934 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251209 08:59:01.261577 28934 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251209 08:59:01.261631 28934 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251209 08:59:01.261714 28934 ts_tablet_manager.cc:616] Registered 0 tablets
I20251209 08:59:01.261749 28934 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Dec 09 08:59:01 dist-test-slave-8w6c krb5kdc[28812](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765270741, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.49.65@KRBTEST.COM for oryx/127.20.49.126@KRBTEST.COM
I20251209 08:59:01.273430 28934 rpc_server.cc:307] RPC server started. Bound to: 127.20.49.65:39793
I20251209 08:59:01.273459 29064 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.49.65:39793 every 8 connection(s)
I20251209 08:59:01.274432 28934 server_base.cc:1179] Dumped server information to /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765270668420570-20677-0/minicluster-data/ts-0/data/info.pb
I20251209 08:59:01.274766 28909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:59:01.262305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.49.65:50589 (local address 127.20.49.126:41745)
1209 08:59:01.262423 (+ 118us) server_negotiation.cc:207] Beginning negotiation
1209 08:59:01.262428 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1209 08:59:01.263335 (+ 907us) server_negotiation.cc:408] Connection header received
1209 08:59:01.264392 (+ 1057us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1209 08:59:01.264397 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1209 08:59:01.264471 (+ 74us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1209 08:59:01.264579 (+ 108us) server_negotiation.cc:227] Negotiated authn=SASL
1209 08:59:01.266580 (+ 2001us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:59:01.267133 (+ 553us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1209 08:59:01.267870 (+ 737us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:59:01.268059 (+ 189us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:59:01.270705 (+ 2646us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1209 08:59:01.270727 (+ 22us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1209 08:59:01.270729 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1209 08:59:01.270757 (+ 28us) server_negotiation.cc:923] calling sasl_server_start()
1209 08:59:01.272697 (+ 1940us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1209 08:59:01.273270 (+ 573us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1209 08:59:01.273275 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1209 08:59:01.273279 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
1209 08:59:01.273341 (+ 62us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1209 08:59:01.273645 (+ 304us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1209 08:59:01.273649 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1209 08:59:01.273651 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1209 08:59:01.273848 (+ 197us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1209 08:59:01.273946 (+ 98us) server_negotiation.cc:1036] Waiting for connection context
1209 08:59:01.274428 (+ 482us) server_negotiation.cc:300] Negotiation successful
1209 08:59:01.274605 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20251209 08:59:01.275386 28958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:59:01.262672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.49.126:41745 (local address 127.20.49.65:50589)
1209 08:59:01.263183 (+ 511us) negotiation.cc:107] Waiting for socket to connect
1209 08:59:01.263220 (+ 37us) client_negotiation.cc:174] Beginning negotiation
1209 08:59:01.264131 (+ 911us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1209 08:59:01.264950 (+ 819us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1209 08:59:01.264963 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
1209 08:59:01.265588 (+ 625us) client_negotiation.cc:189] Negotiated authn=SASL
1209 08:59:01.266401 (+ 813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:59:01.266426 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:59:01.267263 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1209 08:59:01.267267 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1209 08:59:01.267747 (+ 480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:59:01.267761 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:59:01.268002 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:59:01.268666 (+ 664us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1209 08:59:01.268699 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
1209 08:59:01.270540 (+ 1841us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1209 08:59:01.272840 (+ 2300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1209 08:59:01.272845 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1209 08:59:01.272854 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
1209 08:59:01.273132 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1209 08:59:01.273452 (+ 320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1209 08:59:01.273454 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1209 08:59:01.273456 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1209 08:59:01.273559 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1209 08:59:01.273950 (+ 391us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1209 08:59:01.273960 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1209 08:59:01.274189 (+ 229us) client_negotiation.cc:769] Sending connection context
1209 08:59:01.274376 (+ 187us) client_negotiation.cc:240] Negotiation successful
1209 08:59:01.274651 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":328,"thread_start_us":140,"threads_started":1}
I20251209 08:59:01.276558 29065 heartbeater.cc:344] Connected to a master server at 127.20.49.126:41745
I20251209 08:59:01.276715 29065 heartbeater.cc:461] Registering TS with master...
I20251209 08:59:01.277303 29065 heartbeater.cc:507] Master 127.20.49.126:41745 requested a full tablet report, sending...
I20251209 08:59:01.278885 28859 ts_manager.cc:194] Registered new tserver with Master: 3cbbcfcfb7634dc588875d9c22e2328e (127.20.49.65:39793)
I20251209 08:59:01.280093 28859 master_service.cc:502] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.20.49.65@KRBTEST.COM'} at 127.20.49.65:50589
I20251209 08:59:01.283285 20677 external_mini_cluster.cc:1428] Started /tmp/dist-test-task8XuDZV/build/debug/bin/kudu as pid 28934
I20251209 08:59:01.283412 20677 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765270668420570-20677-0/minicluster-data/ts-0/wal/instance
I20251209 08:59:01.283663 20677 external_mini_cluster.cc:1468] Setting key cc8ca2125390cc04362b6b13ec8d16fd
I20251209 08:59:01.286885 28909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:59:01.278813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.49.65:57609 (local address 127.20.49.126:41745)
1209 08:59:01.278919 (+ 106us) server_negotiation.cc:207] Beginning negotiation
1209 08:59:01.278922 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1209 08:59:01.280765 (+ 1843us) server_negotiation.cc:408] Connection header received
1209 08:59:01.280904 (+ 139us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1209 08:59:01.280907 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1209 08:59:01.280955 (+ 48us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1209 08:59:01.281059 (+ 104us) server_negotiation.cc:227] Negotiated authn=SASL
1209 08:59:01.281632 (+ 573us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:59:01.282350 (+ 718us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1209 08:59:01.282966 (+ 616us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:59:01.283118 (+ 152us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:59:01.284032 (+ 914us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1209 08:59:01.284048 (+ 16us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1209 08:59:01.284050 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1209 08:59:01.284072 (+ 22us) server_negotiation.cc:923] calling sasl_server_start()
1209 08:59:01.285480 (+ 1408us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1209 08:59:01.285911 (+ 431us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1209 08:59:01.285915 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1209 08:59:01.285917 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1209 08:59:01.285959 (+ 42us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1209 08:59:01.286169 (+ 210us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1209 08:59:01.286173 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1209 08:59:01.286175 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1209 08:59:01.286308 (+ 133us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1209 08:59:01.286396 (+ 88us) server_negotiation.cc:1036] Waiting for connection context
1209 08:59:01.286634 (+ 238us) server_negotiation.cc:300] Negotiation successful
1209 08:59:01.286741 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20251209 08:59:01.286903 29071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:59:01.278741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.49.126:41745 (local address 127.20.49.65:57609)
1209 08:59:01.280683 (+ 1942us) negotiation.cc:107] Waiting for socket to connect
1209 08:59:01.280699 (+ 16us) client_negotiation.cc:174] Beginning negotiation
1209 08:59:01.280804 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1209 08:59:01.281052 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1209 08:59:01.281058 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
1209 08:59:01.281282 (+ 224us) client_negotiation.cc:189] Negotiated authn=SASL
1209 08:59:01.281518 (+ 236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:59:01.281526 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:59:01.282466 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1209 08:59:01.282469 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1209 08:59:01.282867 (+ 398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:59:01.282875 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:59:01.282967 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:59:01.283580 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1209 08:59:01.283594 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
1209 08:59:01.283922 (+ 328us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1209 08:59:01.285590 (+ 1668us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1209 08:59:01.285594 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1209 08:59:01.285596 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1209 08:59:01.285827 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1209 08:59:01.286041 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1209 08:59:01.286044 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1209 08:59:01.286046 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1209 08:59:01.286096 (+ 50us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1209 08:59:01.286397 (+ 301us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1209 08:59:01.286400 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1209 08:59:01.286485 (+ 85us) client_negotiation.cc:769] Sending connection context
1209 08:59:01.286574 (+ 89us) client_negotiation.cc:240] Negotiation successful
1209 08:59:01.286712 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1861,"thread_start_us":95,"threads_started":1}
WARNING: no policy specified for oryx/127.20.49.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.20.49.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.20.49.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765270668420570-20677-0/krb5kdc/oryx.keytab.
Entry for principal oryx/127.20.49.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765270668420570-20677-0/krb5kdc/oryx.keytab.
WARNING: no policy specified for HTTP/127.20.49.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.20.49.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.20.49.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765270668420570-20677-0/krb5kdc/oryx.keytab.
Entry for principal HTTP/127.20.49.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765270668420570-20677-0/krb5kdc/oryx.keytab.
I20251209 08:59:01.332379 20677 external_mini_cluster.cc:1366] Running /tmp/dist-test-task8XuDZV/build/debug/bin/kudu
/tmp/dist-test-task8XuDZV/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765270668420570-20677-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765270668420570-20677-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765270668420570-20677-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765270668420570-20677-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.49.66:0
--local_ip_for_outbound_sockets=127.20.49.66
--webserver_interface=127.20.49.66
--webserver_port=0
--tserver_master_addrs=127.20.49.126:41745
--keytab_file=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765270668420570-20677-0/krb5kdc/oryx.keytab
--principal=oryx/127.20.49.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.20.49.84:44369
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765270668420570-20677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765270668420570-20677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765270668420570-20677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251209 08:59:01.441982 29076 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251209 08:59:01.442215 29076 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251209 08:59:01.442301 29076 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251209 08:59:01.445731 29076 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251209 08:59:01.445796 29076 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251209 08:59:01.445870 29076 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20251209 08:59:01.445946 29076 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.49.66
I20251209 08:59:01.450234 29076 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.49.84:44369
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765270668420570-20677-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765270668420570-20677-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765270668420570-20677-0/krb5kdc/oryx.keytab
--principal=oryx/127.20.49.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.49.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765270668420570-20677-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.49.66
--webserver_port=0
--webserver_require_spnego=true
--enable_txn_system_client_init=true
--tserver_master_addrs=127.20.49.126:41745
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29076
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.49.66
--log_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765270668420570-20677-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 468897755969d686fb08386f42f7835685e7953a
build type DEBUG
built by None at 09 Dec 2025 08:43:15 UTC on 24a791456cd2
build id 9426
I20251209 08:59:01.451321 29076 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251209 08:59:01.452183 29076 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251209 08:59:01.458711 29082 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251209 08:59:01.458678 29081 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251209 08:59:01.458681 29084 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251209 08:59:01.459117 29076 server_base.cc:1047] running on GCE node
I20251209 08:59:01.459477 29076 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251209 08:59:01.460047 29076 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251209 08:59:01.461229 29076 hybrid_clock.cc:648] HybridClock initialized: now 1765270741461216 us; error 31 us; skew 500 ppm
Dec 09 08:59:01 dist-test-slave-8w6c krb5kdc[28812](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765270741, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.49.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251209 08:59:01.463974 29076 init.cc:377] Logged in from keytab as oryx/127.20.49.66@KRBTEST.COM (short username oryx)
I20251209 08:59:01.465080 29076 webserver.cc:492] Webserver started at http://127.20.49.66:35045/ using document root <none> and password file <none>
I20251209 08:59:01.465656 29076 fs_manager.cc:362] Metadata directory not provided
I20251209 08:59:01.465752 29076 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251209 08:59:01.465957 29076 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251209 08:59:01.467799 29076 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765270668420570-20677-0/minicluster-data/ts-1/data/instance:
uuid: "84d473e778bd4553a757338c4cb05fb4"
format_stamp: "Formatted at 2025-12-09 08:59:01 on dist-test-slave-8w6c"
server_key: "5447c6b60309ed87d8409e3b4c273e92"
server_key_iv: "4193e655cf210360cb2e010a2f53c545"
server_key_version: "encryptionkey@0"
I20251209 08:59:01.468308 29076 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765270668420570-20677-0/minicluster-data/ts-1/wal/instance:
uuid: "84d473e778bd4553a757338c4cb05fb4"
format_stamp: "Formatted at 2025-12-09 08:59:01 on dist-test-slave-8w6c"
server_key: "5447c6b60309ed87d8409e3b4c273e92"
server_key_iv: "4193e655cf210360cb2e010a2f53c545"
server_key_version: "encryptionkey@0"
I20251209 08:59:01.471863 29076 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20251209 08:59:01.474174 29091 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251209 08:59:01.475188 29076 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20251209 08:59:01.475296 29076 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765270668420570-20677-0/minicluster-data/ts-1/data,/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765270668420570-20677-0/minicluster-data/ts-1/wal
uuid: "84d473e778bd4553a757338c4cb05fb4"
format_stamp: "Formatted at 2025-12-09 08:59:01 on dist-test-slave-8w6c"
server_key: "5447c6b60309ed87d8409e3b4c273e92"
server_key_iv: "4193e655cf210360cb2e010a2f53c545"
server_key_version: "encryptionkey@0"
I20251209 08:59:01.475385 29076 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765270668420570-20677-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765270668420570-20677-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765270668420570-20677-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251209 08:59:01.497864 29076 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251209 08:59:01.501111 29076 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251209 08:59:01.501296 29076 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251209 08:59:01.502794 29076 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251209 08:59:01.502851 29076 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251209 08:59:01.502911 29076 ts_tablet_manager.cc:616] Registered 0 tablets
I20251209 08:59:01.502943 29076 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Dec 09 08:59:01 dist-test-slave-8w6c krb5kdc[28812](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765270741, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.49.66@KRBTEST.COM for oryx/127.20.49.126@KRBTEST.COM
I20251209 08:59:01.514523 29076 rpc_server.cc:307] RPC server started. Bound to: 127.20.49.66:44029
I20251209 08:59:01.514595 29206 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.49.66:44029 every 8 connection(s)
I20251209 08:59:01.515156 28909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:59:01.503558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.49.66:60423 (local address 127.20.49.126:41745)
1209 08:59:01.503723 (+ 165us) server_negotiation.cc:207] Beginning negotiation
1209 08:59:01.503729 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1209 08:59:01.504448 (+ 719us) server_negotiation.cc:408] Connection header received
1209 08:59:01.505443 (+ 995us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1209 08:59:01.505447 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1209 08:59:01.505517 (+ 70us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1209 08:59:01.505627 (+ 110us) server_negotiation.cc:227] Negotiated authn=SASL
1209 08:59:01.507048 (+ 1421us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:59:01.507761 (+ 713us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1209 08:59:01.508489 (+ 728us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:59:01.508728 (+ 239us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:59:01.511255 (+ 2527us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1209 08:59:01.511275 (+ 20us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1209 08:59:01.511278 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1209 08:59:01.511306 (+ 28us) server_negotiation.cc:923] calling sasl_server_start()
1209 08:59:01.513007 (+ 1701us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1209 08:59:01.513589 (+ 582us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1209 08:59:01.513593 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1209 08:59:01.513594 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1209 08:59:01.513640 (+ 46us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1209 08:59:01.513985 (+ 345us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1209 08:59:01.513987 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1209 08:59:01.513989 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1209 08:59:01.514145 (+ 156us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1209 08:59:01.514238 (+ 93us) server_negotiation.cc:1036] Waiting for connection context
1209 08:59:01.514854 (+ 616us) server_negotiation.cc:300] Negotiation successful
1209 08:59:01.514987 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20251209 08:59:01.515604 29076 server_base.cc:1179] Dumped server information to /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765270668420570-20677-0/minicluster-data/ts-1/data/info.pb
I20251209 08:59:01.515805 29100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:59:01.503857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.49.126:41745 (local address 127.20.49.66:60423)
1209 08:59:01.504293 (+ 436us) negotiation.cc:107] Waiting for socket to connect
1209 08:59:01.504334 (+ 41us) client_negotiation.cc:174] Beginning negotiation
1209 08:59:01.505209 (+ 875us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1209 08:59:01.505891 (+ 682us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1209 08:59:01.505900 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
1209 08:59:01.506285 (+ 385us) client_negotiation.cc:189] Negotiated authn=SASL
1209 08:59:01.506846 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:59:01.506859 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:59:01.507945 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1209 08:59:01.507950 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1209 08:59:01.508367 (+ 417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:59:01.508374 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:59:01.508604 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:59:01.509327 (+ 723us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1209 08:59:01.509350 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
1209 08:59:01.511085 (+ 1735us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1209 08:59:01.513118 (+ 2033us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1209 08:59:01.513128 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1209 08:59:01.513143 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
1209 08:59:01.513491 (+ 348us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1209 08:59:01.513765 (+ 274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1209 08:59:01.513768 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1209 08:59:01.513771 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
1209 08:59:01.513888 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1209 08:59:01.514255 (+ 367us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1209 08:59:01.514263 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1209 08:59:01.514553 (+ 290us) client_negotiation.cc:769] Sending connection context
1209 08:59:01.514799 (+ 246us) client_negotiation.cc:240] Negotiation successful
1209 08:59:01.515086 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":115,"threads_started":1}
I20251209 08:59:01.517043 29207 heartbeater.cc:344] Connected to a master server at 127.20.49.126:41745
I20251209 08:59:01.517191 29207 heartbeater.cc:461] Registering TS with master...
I20251209 08:59:01.517750 29207 heartbeater.cc:507] Master 127.20.49.126:41745 requested a full tablet report, sending...
I20251209 08:59:01.518724 28859 ts_manager.cc:194] Registered new tserver with Master: 84d473e778bd4553a757338c4cb05fb4 (127.20.49.66:44029)
I20251209 08:59:01.519029 20677 external_mini_cluster.cc:1428] Started /tmp/dist-test-task8XuDZV/build/debug/bin/kudu as pid 29076
I20251209 08:59:01.519141 20677 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765270668420570-20677-0/minicluster-data/ts-1/wal/instance
I20251209 08:59:01.519429 20677 external_mini_cluster.cc:1468] Setting key 7e6dec9c2923c7adf26ab411660d14b8
I20251209 08:59:01.519493 28859 master_service.cc:502] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.20.49.66@KRBTEST.COM'} at 127.20.49.66:60423
I20251209 08:59:01.525784 28909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:59:01.519058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.49.66:41549 (local address 127.20.49.126:41745)
1209 08:59:01.519236 (+ 178us) server_negotiation.cc:207] Beginning negotiation
1209 08:59:01.519241 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1209 08:59:01.519362 (+ 121us) server_negotiation.cc:408] Connection header received
1209 08:59:01.519468 (+ 106us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1209 08:59:01.519472 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1209 08:59:01.519530 (+ 58us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1209 08:59:01.519620 (+ 90us) server_negotiation.cc:227] Negotiated authn=SASL
1209 08:59:01.520246 (+ 626us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:59:01.520956 (+ 710us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1209 08:59:01.521782 (+ 826us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:59:01.522013 (+ 231us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:59:01.522742 (+ 729us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1209 08:59:01.522763 (+ 21us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1209 08:59:01.522768 (+ 5us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1209 08:59:01.522798 (+ 30us) server_negotiation.cc:923] calling sasl_server_start()
1209 08:59:01.524298 (+ 1500us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1209 08:59:01.524741 (+ 443us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1209 08:59:01.524744 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1209 08:59:01.524745 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1209 08:59:01.524786 (+ 41us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1209 08:59:01.524992 (+ 206us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1209 08:59:01.524995 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1209 08:59:01.524997 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1209 08:59:01.525158 (+ 161us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1209 08:59:01.525246 (+ 88us) server_negotiation.cc:1036] Waiting for connection context
1209 08:59:01.525474 (+ 228us) server_negotiation.cc:300] Negotiation successful
1209 08:59:01.525603 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
I20251209 08:59:01.525801 29213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:59:01.518990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.49.126:41745 (local address 127.20.49.66:41549)
1209 08:59:01.519277 (+ 287us) negotiation.cc:107] Waiting for socket to connect
1209 08:59:01.519292 (+ 15us) client_negotiation.cc:174] Beginning negotiation
1209 08:59:01.519387 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1209 08:59:01.519648 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1209 08:59:01.519651 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1209 08:59:01.519861 (+ 210us) client_negotiation.cc:189] Negotiated authn=SASL
1209 08:59:01.520132 (+ 271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:59:01.520141 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:59:01.521093 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1209 08:59:01.521097 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1209 08:59:01.521620 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:59:01.521631 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:59:01.521795 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:59:01.522331 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1209 08:59:01.522343 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
1209 08:59:01.522623 (+ 280us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1209 08:59:01.524428 (+ 1805us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1209 08:59:01.524432 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1209 08:59:01.524434 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1209 08:59:01.524631 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1209 08:59:01.524868 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1209 08:59:01.524871 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1209 08:59:01.524873 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1209 08:59:01.524915 (+ 42us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1209 08:59:01.525245 (+ 330us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1209 08:59:01.525249 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1209 08:59:01.525328 (+ 79us) client_negotiation.cc:769] Sending connection context
1209 08:59:01.525415 (+ 87us) client_negotiation.cc:240] Negotiation successful
1209 08:59:01.525567 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":80,"threads_started":1}
WARNING: no policy specified for oryx/127.20.49.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.20.49.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.20.49.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765270668420570-20677-0/krb5kdc/oryx.keytab.
Entry for principal oryx/127.20.49.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765270668420570-20677-0/krb5kdc/oryx.keytab.
WARNING: no policy specified for HTTP/127.20.49.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.20.49.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.20.49.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765270668420570-20677-0/krb5kdc/oryx.keytab.
Entry for principal HTTP/127.20.49.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765270668420570-20677-0/krb5kdc/oryx.keytab.
I20251209 08:59:01.570888 20677 external_mini_cluster.cc:1366] Running /tmp/dist-test-task8XuDZV/build/debug/bin/kudu
/tmp/dist-test-task8XuDZV/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765270668420570-20677-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765270668420570-20677-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765270668420570-20677-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765270668420570-20677-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.49.67:0
--local_ip_for_outbound_sockets=127.20.49.67
--webserver_interface=127.20.49.67
--webserver_port=0
--tserver_master_addrs=127.20.49.126:41745
--keytab_file=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765270668420570-20677-0/krb5kdc/oryx.keytab
--principal=oryx/127.20.49.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.20.49.84:44369
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765270668420570-20677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765270668420570-20677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765270668420570-20677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251209 08:59:01.683972 29218 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251209 08:59:01.684228 29218 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251209 08:59:01.684290 29218 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251209 08:59:01.687654 29218 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251209 08:59:01.687728 29218 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251209 08:59:01.687781 29218 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20251209 08:59:01.687831 29218 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.49.67
I20251209 08:59:01.692139 29218 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.49.84:44369
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765270668420570-20677-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765270668420570-20677-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765270668420570-20677-0/krb5kdc/oryx.keytab
--principal=oryx/127.20.49.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.49.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765270668420570-20677-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.49.67
--webserver_port=0
--webserver_require_spnego=true
--enable_txn_system_client_init=true
--tserver_master_addrs=127.20.49.126:41745
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29218
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.49.67
--log_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765270668420570-20677-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 468897755969d686fb08386f42f7835685e7953a
build type DEBUG
built by None at 09 Dec 2025 08:43:15 UTC on 24a791456cd2
build id 9426
I20251209 08:59:01.693220 29218 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251209 08:59:01.694105 29218 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251209 08:59:01.701056 29224 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251209 08:59:01.701067 29223 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251209 08:59:01.701056 29226 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251209 08:59:01.701272 29218 server_base.cc:1047] running on GCE node
I20251209 08:59:01.701673 29218 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251209 08:59:01.702277 29218 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251209 08:59:01.703439 29218 hybrid_clock.cc:648] HybridClock initialized: now 1765270741703418 us; error 36 us; skew 500 ppm
Dec 09 08:59:01 dist-test-slave-8w6c krb5kdc[28812](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765270741, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.49.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251209 08:59:01.706355 29218 init.cc:377] Logged in from keytab as oryx/127.20.49.67@KRBTEST.COM (short username oryx)
I20251209 08:59:01.707477 29218 webserver.cc:492] Webserver started at http://127.20.49.67:45923/ using document root <none> and password file <none>
I20251209 08:59:01.708128 29218 fs_manager.cc:362] Metadata directory not provided
I20251209 08:59:01.708186 29218 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251209 08:59:01.708377 29218 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251209 08:59:01.710220 29218 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765270668420570-20677-0/minicluster-data/ts-2/data/instance:
uuid: "7a440d3ea874483980ad5a12a47ce1d9"
format_stamp: "Formatted at 2025-12-09 08:59:01 on dist-test-slave-8w6c"
server_key: "83da05978d3c0bfde8a835bbe2f41c3b"
server_key_iv: "6f969e0ea07dbcda781f1b3eb04fba2f"
server_key_version: "encryptionkey@0"
I20251209 08:59:01.710717 29218 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765270668420570-20677-0/minicluster-data/ts-2/wal/instance:
uuid: "7a440d3ea874483980ad5a12a47ce1d9"
format_stamp: "Formatted at 2025-12-09 08:59:01 on dist-test-slave-8w6c"
server_key: "83da05978d3c0bfde8a835bbe2f41c3b"
server_key_iv: "6f969e0ea07dbcda781f1b3eb04fba2f"
server_key_version: "encryptionkey@0"
I20251209 08:59:01.714321 29218 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20251209 08:59:01.716572 29233 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251209 08:59:01.717715 29218 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20251209 08:59:01.717834 29218 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765270668420570-20677-0/minicluster-data/ts-2/data,/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765270668420570-20677-0/minicluster-data/ts-2/wal
uuid: "7a440d3ea874483980ad5a12a47ce1d9"
format_stamp: "Formatted at 2025-12-09 08:59:01 on dist-test-slave-8w6c"
server_key: "83da05978d3c0bfde8a835bbe2f41c3b"
server_key_iv: "6f969e0ea07dbcda781f1b3eb04fba2f"
server_key_version: "encryptionkey@0"
I20251209 08:59:01.717931 29218 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765270668420570-20677-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765270668420570-20677-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765270668420570-20677-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251209 08:59:01.734647 29218 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251209 08:59:01.737516 29218 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251209 08:59:01.737756 29218 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251209 08:59:01.739280 29218 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251209 08:59:01.739331 29218 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251209 08:59:01.739374 29218 ts_tablet_manager.cc:616] Registered 0 tablets
I20251209 08:59:01.739409 29218 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Dec 09 08:59:01 dist-test-slave-8w6c krb5kdc[28812](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765270741, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.49.67@KRBTEST.COM for oryx/127.20.49.126@KRBTEST.COM
I20251209 08:59:01.750157 29218 rpc_server.cc:307] RPC server started. Bound to: 127.20.49.67:38547
I20251209 08:59:01.750200 29348 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.49.67:38547 every 8 connection(s)
I20251209 08:59:01.751410 29218 server_base.cc:1179] Dumped server information to /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765270668420570-20677-0/minicluster-data/ts-2/data/info.pb
I20251209 08:59:01.752723 28909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:59:01.739976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.49.67:44649 (local address 127.20.49.126:41745)
1209 08:59:01.740135 (+ 159us) server_negotiation.cc:207] Beginning negotiation
1209 08:59:01.740140 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1209 08:59:01.740860 (+ 720us) server_negotiation.cc:408] Connection header received
1209 08:59:01.741797 (+ 937us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1209 08:59:01.741804 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
1209 08:59:01.741876 (+ 72us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1209 08:59:01.741994 (+ 118us) server_negotiation.cc:227] Negotiated authn=SASL
1209 08:59:01.743574 (+ 1580us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:59:01.744351 (+ 777us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1209 08:59:01.745370 (+ 1019us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:59:01.745596 (+ 226us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:59:01.748469 (+ 2873us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1209 08:59:01.748496 (+ 27us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1209 08:59:01.748500 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1209 08:59:01.748539 (+ 39us) server_negotiation.cc:923] calling sasl_server_start()
1209 08:59:01.750325 (+ 1786us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1209 08:59:01.750888 (+ 563us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1209 08:59:01.750891 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1209 08:59:01.750893 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1209 08:59:01.750943 (+ 50us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1209 08:59:01.751230 (+ 287us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1209 08:59:01.751232 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1209 08:59:01.751234 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1209 08:59:01.751386 (+ 152us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1209 08:59:01.751458 (+ 72us) server_negotiation.cc:1036] Waiting for connection context
1209 08:59:01.752426 (+ 968us) server_negotiation.cc:300] Negotiation successful
1209 08:59:01.752597 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20251209 08:59:01.752972 29242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:59:01.740258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.49.126:41745 (local address 127.20.49.67:44649)
1209 08:59:01.740694 (+ 436us) negotiation.cc:107] Waiting for socket to connect
1209 08:59:01.740738 (+ 44us) client_negotiation.cc:174] Beginning negotiation
1209 08:59:01.741527 (+ 789us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1209 08:59:01.742140 (+ 613us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1209 08:59:01.742148 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
1209 08:59:01.742653 (+ 505us) client_negotiation.cc:189] Negotiated authn=SASL
1209 08:59:01.743398 (+ 745us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:59:01.743421 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:59:01.744492 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1209 08:59:01.744500 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1209 08:59:01.745238 (+ 738us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:59:01.745251 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:59:01.745515 (+ 264us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:59:01.746328 (+ 813us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1209 08:59:01.746372 (+ 44us) client_negotiation.cc:656] calling sasl_client_start()
1209 08:59:01.748272 (+ 1900us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1209 08:59:01.750434 (+ 2162us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1209 08:59:01.750440 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1209 08:59:01.750450 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
1209 08:59:01.750780 (+ 330us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1209 08:59:01.751031 (+ 251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1209 08:59:01.751034 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1209 08:59:01.751036 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1209 08:59:01.751149 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1209 08:59:01.751481 (+ 332us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1209 08:59:01.751487 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1209 08:59:01.751796 (+ 309us) client_negotiation.cc:769] Sending connection context
1209 08:59:01.752042 (+ 246us) client_negotiation.cc:240] Negotiation successful
1209 08:59:01.752287 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":119,"threads_started":1}
I20251209 08:59:01.754148 29349 heartbeater.cc:344] Connected to a master server at 127.20.49.126:41745
I20251209 08:59:01.754292 29349 heartbeater.cc:461] Registering TS with master...
I20251209 08:59:01.754763 29349 heartbeater.cc:507] Master 127.20.49.126:41745 requested a full tablet report, sending...
I20251209 08:59:01.755825 28860 ts_manager.cc:194] Registered new tserver with Master: 7a440d3ea874483980ad5a12a47ce1d9 (127.20.49.67:38547)
I20251209 08:59:01.756500 28860 master_service.cc:502] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.20.49.67@KRBTEST.COM'} at 127.20.49.67:44649
I20251209 08:59:01.757346 20677 external_mini_cluster.cc:1428] Started /tmp/dist-test-task8XuDZV/build/debug/bin/kudu as pid 29218
I20251209 08:59:01.757462 20677 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765270668420570-20677-0/minicluster-data/ts-2/wal/instance
I20251209 08:59:01.757755 20677 external_mini_cluster.cc:1468] Setting key a9f02fbda71621d7c2821f91c8de3611
I20251209 08:59:01.760083 20677 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20251209 08:59:01.762846 28909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:59:01.756106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.49.67:54679 (local address 127.20.49.126:41745)
1209 08:59:01.756243 (+ 137us) server_negotiation.cc:207] Beginning negotiation
1209 08:59:01.756247 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1209 08:59:01.756436 (+ 189us) server_negotiation.cc:408] Connection header received
1209 08:59:01.756554 (+ 118us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1209 08:59:01.756560 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
1209 08:59:01.756610 (+ 50us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1209 08:59:01.756715 (+ 105us) server_negotiation.cc:227] Negotiated authn=SASL
1209 08:59:01.757316 (+ 601us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:59:01.758058 (+ 742us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1209 08:59:01.758817 (+ 759us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:59:01.759007 (+ 190us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:59:01.759781 (+ 774us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1209 08:59:01.759799 (+ 18us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1209 08:59:01.759802 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1209 08:59:01.759829 (+ 27us) server_negotiation.cc:923] calling sasl_server_start()
1209 08:59:01.761379 (+ 1550us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1209 08:59:01.761869 (+ 490us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1209 08:59:01.761871 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1209 08:59:01.761873 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1209 08:59:01.761915 (+ 42us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1209 08:59:01.762136 (+ 221us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1209 08:59:01.762138 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1209 08:59:01.762140 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1209 08:59:01.762268 (+ 128us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1209 08:59:01.762352 (+ 84us) server_negotiation.cc:1036] Waiting for connection context
1209 08:59:01.762600 (+ 248us) server_negotiation.cc:300] Negotiation successful
1209 08:59:01.762701 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20251209 08:59:01.762863 29355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:59:01.756055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.49.126:41745 (local address 127.20.49.67:54679)
1209 08:59:01.756360 (+ 305us) negotiation.cc:107] Waiting for socket to connect
1209 08:59:01.756371 (+ 11us) client_negotiation.cc:174] Beginning negotiation
1209 08:59:01.756457 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1209 08:59:01.756725 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1209 08:59:01.756728 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1209 08:59:01.756925 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
1209 08:59:01.757162 (+ 237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:59:01.757173 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:59:01.758176 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1209 08:59:01.758179 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1209 08:59:01.758705 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:59:01.758715 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:59:01.758829 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:59:01.759370 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1209 08:59:01.759383 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
1209 08:59:01.759669 (+ 286us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1209 08:59:01.761486 (+ 1817us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1209 08:59:01.761493 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1209 08:59:01.761498 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
1209 08:59:01.761776 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1209 08:59:01.761991 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1209 08:59:01.761996 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1209 08:59:01.762001 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
1209 08:59:01.762064 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1209 08:59:01.762362 (+ 298us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1209 08:59:01.762369 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1209 08:59:01.762445 (+ 76us) client_negotiation.cc:769] Sending connection context
1209 08:59:01.762538 (+ 93us) client_negotiation.cc:240] Negotiation successful
1209 08:59:01.762670 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":76,"threads_started":1}
Dec 09 08:59:01 dist-test-slave-8w6c krb5kdc[28812](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765270741, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Dec 09 08:59:01 dist-test-slave-8w6c krb5kdc[28812](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.20.49.126@KRBTEST.COM, Server not found in Kerberos database
Dec 09 08:59:01 dist-test-slave-8w6c krb5kdc[28812](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.20.49.126@KRBTEST.COM, Server not found in Kerberos database
W20251209 08:59:01.786213 28909 negotiation.cc:337] Failed RPC negotiation. Trace:
1209 08:59:01.780581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59158 (local address 127.20.49.126:41745)
1209 08:59:01.780711 (+ 130us) server_negotiation.cc:207] Beginning negotiation
1209 08:59:01.780714 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1209 08:59:01.780727 (+ 13us) server_negotiation.cc:408] Connection header received
1209 08:59:01.780769 (+ 42us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1209 08:59:01.780771 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1209 08:59:01.780815 (+ 44us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1209 08:59:01.780889 (+ 74us) server_negotiation.cc:227] Negotiated authn=SASL
1209 08:59:01.781704 (+ 815us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:59:01.782194 (+ 490us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1209 08:59:01.782951 (+ 757us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:59:01.783104 (+ 153us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:59:01.786064 (+ 2960us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59158: BlockingRecv error: recv got EOF from 127.0.0.1:59158 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Dec 09 08:59:01 dist-test-slave-8w6c krb5kdc[28812](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765270741, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.20.49.126@KRBTEST.COM
I20251209 08:59:01.795570 28909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:59:01.787903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59162 (local address 127.20.49.126:41745)
1209 08:59:01.788062 (+ 159us) server_negotiation.cc:207] Beginning negotiation
1209 08:59:01.788066 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1209 08:59:01.788081 (+ 15us) server_negotiation.cc:408] Connection header received
1209 08:59:01.788203 (+ 122us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1209 08:59:01.788205 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1209 08:59:01.788249 (+ 44us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1209 08:59:01.788309 (+ 60us) server_negotiation.cc:227] Negotiated authn=SASL
1209 08:59:01.789056 (+ 747us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:59:01.789584 (+ 528us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1209 08:59:01.790273 (+ 689us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:59:01.790412 (+ 139us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:59:01.792536 (+ 2124us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1209 08:59:01.792554 (+ 18us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1209 08:59:01.792556 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1209 08:59:01.792583 (+ 27us) server_negotiation.cc:923] calling sasl_server_start()
1209 08:59:01.794063 (+ 1480us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1209 08:59:01.794444 (+ 381us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1209 08:59:01.794448 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1209 08:59:01.794449 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1209 08:59:01.794511 (+ 62us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1209 08:59:01.794811 (+ 300us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1209 08:59:01.794814 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1209 08:59:01.794816 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1209 08:59:01.794972 (+ 156us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1209 08:59:01.795100 (+ 128us) server_negotiation.cc:1036] Waiting for connection context
1209 08:59:01.795267 (+ 167us) server_negotiation.cc:300] Negotiation successful
1209 08:59:01.795361 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20251209 08:59:01.798425 28860 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59162:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251209 08:59:01.800846 28860 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251209 08:59:01.813961 29373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:59:01.809040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.49.67:38547 (local address 127.0.0.1:35670)
1209 08:59:01.809918 (+ 878us) negotiation.cc:107] Waiting for socket to connect
1209 08:59:01.809958 (+ 40us) client_negotiation.cc:174] Beginning negotiation
1209 08:59:01.810084 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1209 08:59:01.810817 (+ 733us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1209 08:59:01.810823 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
1209 08:59:01.810847 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1209 08:59:01.811167 (+ 320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:59:01.811177 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:59:01.812505 (+ 1328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1209 08:59:01.812510 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1209 08:59:01.813339 (+ 829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:59:01.813347 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:59:01.813455 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:59:01.813518 (+ 63us) client_negotiation.cc:769] Sending connection context
1209 08:59:01.813614 (+ 96us) client_negotiation.cc:240] Negotiation successful
1209 08:59:01.813731 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":700,"thread_start_us":83,"threads_started":1}
I20251209 08:59:01.813961 29374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:59:01.809220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.49.66:44029 (local address 127.0.0.1:52844)
1209 08:59:01.810374 (+ 1154us) negotiation.cc:107] Waiting for socket to connect
1209 08:59:01.810388 (+ 14us) client_negotiation.cc:174] Beginning negotiation
1209 08:59:01.810469 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1209 08:59:01.811118 (+ 649us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1209 08:59:01.811121 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1209 08:59:01.811140 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1209 08:59:01.811364 (+ 224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:59:01.811370 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:59:01.812579 (+ 1209us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1209 08:59:01.812583 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1209 08:59:01.813530 (+ 947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:59:01.813538 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:59:01.813641 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:59:01.813659 (+ 18us) client_negotiation.cc:769] Sending connection context
1209 08:59:01.813738 (+ 79us) client_negotiation.cc:240] Negotiation successful
1209 08:59:01.813799 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1073,"thread_start_us":124,"threads_started":1}
I20251209 08:59:01.814507 29372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:59:01.809208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35670 (local address 127.20.49.67:38547)
1209 08:59:01.810372 (+ 1164us) server_negotiation.cc:207] Beginning negotiation
1209 08:59:01.810380 (+ 8us) server_negotiation.cc:400] Waiting for connection header
1209 08:59:01.810404 (+ 24us) server_negotiation.cc:408] Connection header received
1209 08:59:01.810499 (+ 95us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1209 08:59:01.810505 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
1209 08:59:01.810663 (+ 158us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1209 08:59:01.810785 (+ 122us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1209 08:59:01.811284 (+ 499us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:59:01.812373 (+ 1089us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1209 08:59:01.813474 (+ 1101us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:59:01.814049 (+ 575us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:59:01.814174 (+ 125us) server_negotiation.cc:1036] Waiting for connection context
1209 08:59:01.814260 (+ 86us) server_negotiation.cc:300] Negotiation successful
1209 08:59:01.814347 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1058,"thread_start_us":118,"threads_started":1}
I20251209 08:59:01.814594 29371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:59:01.809771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.49.65:39793 (local address 127.0.0.1:60562)
1209 08:59:01.810994 (+ 1223us) negotiation.cc:107] Waiting for socket to connect
1209 08:59:01.811005 (+ 11us) client_negotiation.cc:174] Beginning negotiation
1209 08:59:01.811093 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1209 08:59:01.811519 (+ 426us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1209 08:59:01.811524 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
1209 08:59:01.811542 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1209 08:59:01.811773 (+ 231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:59:01.811782 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:59:01.813297 (+ 1515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1209 08:59:01.813301 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1209 08:59:01.814240 (+ 939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:59:01.814250 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:59:01.814358 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:59:01.814375 (+ 17us) client_negotiation.cc:769] Sending connection context
1209 08:59:01.814429 (+ 54us) client_negotiation.cc:240] Negotiation successful
1209 08:59:01.814487 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1166,"thread_start_us":101,"threads_started":1}
I20251209 08:59:01.815714 29376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:59:01.810187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60562 (local address 127.20.49.65:39793)
1209 08:59:01.810609 (+ 422us) server_negotiation.cc:207] Beginning negotiation
1209 08:59:01.810615 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1209 08:59:01.811089 (+ 474us) server_negotiation.cc:408] Connection header received
1209 08:59:01.811216 (+ 127us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1209 08:59:01.811220 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1209 08:59:01.811376 (+ 156us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1209 08:59:01.811525 (+ 149us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1209 08:59:01.811899 (+ 374us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:59:01.813156 (+ 1257us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1209 08:59:01.814767 (+ 1611us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:59:01.815271 (+ 504us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:59:01.815393 (+ 122us) server_negotiation.cc:1036] Waiting for connection context
1209 08:59:01.815456 (+ 63us) server_negotiation.cc:300] Negotiation successful
1209 08:59:01.815547 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":316,"thread_start_us":81,"threads_started":1}
I20251209 08:59:01.816149 29375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:59:01.809757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52844 (local address 127.20.49.66:44029)
1209 08:59:01.810708 (+ 951us) server_negotiation.cc:207] Beginning negotiation
1209 08:59:01.810713 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1209 08:59:01.810729 (+ 16us) server_negotiation.cc:408] Connection header received
1209 08:59:01.810811 (+ 82us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1209 08:59:01.810816 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1209 08:59:01.810968 (+ 152us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1209 08:59:01.811470 (+ 502us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1209 08:59:01.811641 (+ 171us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:59:01.812408 (+ 767us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1209 08:59:01.814440 (+ 2032us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:59:01.815756 (+ 1316us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:59:01.815865 (+ 109us) server_negotiation.cc:1036] Waiting for connection context
1209 08:59:01.815928 (+ 63us) server_negotiation.cc:300] Negotiation successful
1209 08:59:01.816014 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":873,"thread_start_us":106,"threads_started":1}
I20251209 08:59:01.817461 29283 tablet_service.cc:1505] Processing CreateTablet for tablet 32849c0a934c42868d51faf6d6a7a293 (DEFAULT_TABLE table=test-table [id=938e48322161461d89813bcf0665bc0b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251209 08:59:01.818470 29283 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 32849c0a934c42868d51faf6d6a7a293. 1 dirs total, 0 dirs full, 0 dirs failed
I20251209 08:59:01.818504 29141 tablet_service.cc:1505] Processing CreateTablet for tablet 32849c0a934c42868d51faf6d6a7a293 (DEFAULT_TABLE table=test-table [id=938e48322161461d89813bcf0665bc0b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251209 08:59:01.818490 28999 tablet_service.cc:1505] Processing CreateTablet for tablet 32849c0a934c42868d51faf6d6a7a293 (DEFAULT_TABLE table=test-table [id=938e48322161461d89813bcf0665bc0b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251209 08:59:01.819480 28999 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 32849c0a934c42868d51faf6d6a7a293. 1 dirs total, 0 dirs full, 0 dirs failed
I20251209 08:59:01.819483 29141 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 32849c0a934c42868d51faf6d6a7a293. 1 dirs total, 0 dirs full, 0 dirs failed
I20251209 08:59:01.826043 29378 tablet_bootstrap.cc:492] T 32849c0a934c42868d51faf6d6a7a293 P 3cbbcfcfb7634dc588875d9c22e2328e: Bootstrap starting.
I20251209 08:59:01.826305 29377 tablet_bootstrap.cc:492] T 32849c0a934c42868d51faf6d6a7a293 P 7a440d3ea874483980ad5a12a47ce1d9: Bootstrap starting.
I20251209 08:59:01.826884 29379 tablet_bootstrap.cc:492] T 32849c0a934c42868d51faf6d6a7a293 P 84d473e778bd4553a757338c4cb05fb4: Bootstrap starting.
I20251209 08:59:01.828456 29378 tablet_bootstrap.cc:654] T 32849c0a934c42868d51faf6d6a7a293 P 3cbbcfcfb7634dc588875d9c22e2328e: Neither blocks nor log segments found. Creating new log.
I20251209 08:59:01.828456 29377 tablet_bootstrap.cc:654] T 32849c0a934c42868d51faf6d6a7a293 P 7a440d3ea874483980ad5a12a47ce1d9: Neither blocks nor log segments found. Creating new log.
I20251209 08:59:01.828588 29379 tablet_bootstrap.cc:654] T 32849c0a934c42868d51faf6d6a7a293 P 84d473e778bd4553a757338c4cb05fb4: Neither blocks nor log segments found. Creating new log.
I20251209 08:59:01.829414 29378 log.cc:826] T 32849c0a934c42868d51faf6d6a7a293 P 3cbbcfcfb7634dc588875d9c22e2328e: Log is configured to *not* fsync() on all Append() calls
I20251209 08:59:01.829586 29379 log.cc:826] T 32849c0a934c42868d51faf6d6a7a293 P 84d473e778bd4553a757338c4cb05fb4: Log is configured to *not* fsync() on all Append() calls
I20251209 08:59:01.829639 29377 log.cc:826] T 32849c0a934c42868d51faf6d6a7a293 P 7a440d3ea874483980ad5a12a47ce1d9: Log is configured to *not* fsync() on all Append() calls
I20251209 08:59:01.831532 29379 tablet_bootstrap.cc:492] T 32849c0a934c42868d51faf6d6a7a293 P 84d473e778bd4553a757338c4cb05fb4: No bootstrap required, opened a new log
I20251209 08:59:01.831529 29377 tablet_bootstrap.cc:492] T 32849c0a934c42868d51faf6d6a7a293 P 7a440d3ea874483980ad5a12a47ce1d9: No bootstrap required, opened a new log
I20251209 08:59:01.831529 29378 tablet_bootstrap.cc:492] T 32849c0a934c42868d51faf6d6a7a293 P 3cbbcfcfb7634dc588875d9c22e2328e: No bootstrap required, opened a new log
I20251209 08:59:01.831735 29378 ts_tablet_manager.cc:1403] T 32849c0a934c42868d51faf6d6a7a293 P 3cbbcfcfb7634dc588875d9c22e2328e: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20251209 08:59:01.831735 29377 ts_tablet_manager.cc:1403] T 32849c0a934c42868d51faf6d6a7a293 P 7a440d3ea874483980ad5a12a47ce1d9: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20251209 08:59:01.831751 29379 ts_tablet_manager.cc:1403] T 32849c0a934c42868d51faf6d6a7a293 P 84d473e778bd4553a757338c4cb05fb4: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20251209 08:59:01.834350 29379 raft_consensus.cc:359] T 32849c0a934c42868d51faf6d6a7a293 P 84d473e778bd4553a757338c4cb05fb4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a440d3ea874483980ad5a12a47ce1d9" member_type: VOTER last_known_addr { host: "127.20.49.67" port: 38547 } } peers { permanent_uuid: "84d473e778bd4553a757338c4cb05fb4" member_type: VOTER last_known_addr { host: "127.20.49.66" port: 44029 } } peers { permanent_uuid: "3cbbcfcfb7634dc588875d9c22e2328e" member_type: VOTER last_known_addr { host: "127.20.49.65" port: 39793 } }
I20251209 08:59:01.834566 29379 raft_consensus.cc:385] T 32849c0a934c42868d51faf6d6a7a293 P 84d473e778bd4553a757338c4cb05fb4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251209 08:59:01.834627 29379 raft_consensus.cc:740] T 32849c0a934c42868d51faf6d6a7a293 P 84d473e778bd4553a757338c4cb05fb4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84d473e778bd4553a757338c4cb05fb4, State: Initialized, Role: FOLLOWER
I20251209 08:59:01.835059 29379 consensus_queue.cc:260] T 32849c0a934c42868d51faf6d6a7a293 P 84d473e778bd4553a757338c4cb05fb4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a440d3ea874483980ad5a12a47ce1d9" member_type: VOTER last_known_addr { host: "127.20.49.67" port: 38547 } } peers { permanent_uuid: "84d473e778bd4553a757338c4cb05fb4" member_type: VOTER last_known_addr { host: "127.20.49.66" port: 44029 } } peers { permanent_uuid: "3cbbcfcfb7634dc588875d9c22e2328e" member_type: VOTER last_known_addr { host: "127.20.49.65" port: 39793 } }
I20251209 08:59:01.834964 29378 raft_consensus.cc:359] T 32849c0a934c42868d51faf6d6a7a293 P 3cbbcfcfb7634dc588875d9c22e2328e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a440d3ea874483980ad5a12a47ce1d9" member_type: VOTER last_known_addr { host: "127.20.49.67" port: 38547 } } peers { permanent_uuid: "84d473e778bd4553a757338c4cb05fb4" member_type: VOTER last_known_addr { host: "127.20.49.66" port: 44029 } } peers { permanent_uuid: "3cbbcfcfb7634dc588875d9c22e2328e" member_type: VOTER last_known_addr { host: "127.20.49.65" port: 39793 } }
I20251209 08:59:01.834970 29377 raft_consensus.cc:359] T 32849c0a934c42868d51faf6d6a7a293 P 7a440d3ea874483980ad5a12a47ce1d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a440d3ea874483980ad5a12a47ce1d9" member_type: VOTER last_known_addr { host: "127.20.49.67" port: 38547 } } peers { permanent_uuid: "84d473e778bd4553a757338c4cb05fb4" member_type: VOTER last_known_addr { host: "127.20.49.66" port: 44029 } } peers { permanent_uuid: "3cbbcfcfb7634dc588875d9c22e2328e" member_type: VOTER last_known_addr { host: "127.20.49.65" port: 39793 } }
I20251209 08:59:01.835176 29377 raft_consensus.cc:385] T 32849c0a934c42868d51faf6d6a7a293 P 7a440d3ea874483980ad5a12a47ce1d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251209 08:59:01.835176 29378 raft_consensus.cc:385] T 32849c0a934c42868d51faf6d6a7a293 P 3cbbcfcfb7634dc588875d9c22e2328e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251209 08:59:01.835238 29378 raft_consensus.cc:740] T 32849c0a934c42868d51faf6d6a7a293 P 3cbbcfcfb7634dc588875d9c22e2328e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3cbbcfcfb7634dc588875d9c22e2328e, State: Initialized, Role: FOLLOWER
I20251209 08:59:01.835247 29377 raft_consensus.cc:740] T 32849c0a934c42868d51faf6d6a7a293 P 7a440d3ea874483980ad5a12a47ce1d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a440d3ea874483980ad5a12a47ce1d9, State: Initialized, Role: FOLLOWER
I20251209 08:59:01.835670 29377 consensus_queue.cc:260] T 32849c0a934c42868d51faf6d6a7a293 P 7a440d3ea874483980ad5a12a47ce1d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a440d3ea874483980ad5a12a47ce1d9" member_type: VOTER last_known_addr { host: "127.20.49.67" port: 38547 } } peers { permanent_uuid: "84d473e778bd4553a757338c4cb05fb4" member_type: VOTER last_known_addr { host: "127.20.49.66" port: 44029 } } peers { permanent_uuid: "3cbbcfcfb7634dc588875d9c22e2328e" member_type: VOTER last_known_addr { host: "127.20.49.65" port: 39793 } }
I20251209 08:59:01.835670 29378 consensus_queue.cc:260] T 32849c0a934c42868d51faf6d6a7a293 P 3cbbcfcfb7634dc588875d9c22e2328e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a440d3ea874483980ad5a12a47ce1d9" member_type: VOTER last_known_addr { host: "127.20.49.67" port: 38547 } } peers { permanent_uuid: "84d473e778bd4553a757338c4cb05fb4" member_type: VOTER last_known_addr { host: "127.20.49.66" port: 44029 } } peers { permanent_uuid: "3cbbcfcfb7634dc588875d9c22e2328e" member_type: VOTER last_known_addr { host: "127.20.49.65" port: 39793 } }
I20251209 08:59:01.835819 29207 heartbeater.cc:499] Master 127.20.49.126:41745 was elected leader, sending a full tablet report...
I20251209 08:59:01.836017 29379 ts_tablet_manager.cc:1434] T 32849c0a934c42868d51faf6d6a7a293 P 84d473e778bd4553a757338c4cb05fb4: Time spent starting tablet: real 0.004s user 0.000s sys 0.004s
I20251209 08:59:01.836483 29378 ts_tablet_manager.cc:1434] T 32849c0a934c42868d51faf6d6a7a293 P 3cbbcfcfb7634dc588875d9c22e2328e: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20251209 08:59:01.836483 29377 ts_tablet_manager.cc:1434] T 32849c0a934c42868d51faf6d6a7a293 P 7a440d3ea874483980ad5a12a47ce1d9: Time spent starting tablet: real 0.005s user 0.001s sys 0.005s
I20251209 08:59:01.836885 29349 heartbeater.cc:499] Master 127.20.49.126:41745 was elected leader, sending a full tablet report...
I20251209 08:59:01.837878 29065 heartbeater.cc:499] Master 127.20.49.126:41745 was elected leader, sending a full tablet report...
I20251209 08:59:01.977973 29385 raft_consensus.cc:493] T 32849c0a934c42868d51faf6d6a7a293 P 7a440d3ea874483980ad5a12a47ce1d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251209 08:59:01.978164 29385 raft_consensus.cc:515] T 32849c0a934c42868d51faf6d6a7a293 P 7a440d3ea874483980ad5a12a47ce1d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a440d3ea874483980ad5a12a47ce1d9" member_type: VOTER last_known_addr { host: "127.20.49.67" port: 38547 } } peers { permanent_uuid: "84d473e778bd4553a757338c4cb05fb4" member_type: VOTER last_known_addr { host: "127.20.49.66" port: 44029 } } peers { permanent_uuid: "3cbbcfcfb7634dc588875d9c22e2328e" member_type: VOTER last_known_addr { host: "127.20.49.65" port: 39793 } }
I20251209 08:59:01.979367 29385 leader_election.cc:290] T 32849c0a934c42868d51faf6d6a7a293 P 7a440d3ea874483980ad5a12a47ce1d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 84d473e778bd4553a757338c4cb05fb4 (127.20.49.66:44029), 3cbbcfcfb7634dc588875d9c22e2328e (127.20.49.65:39793)
I20251209 08:59:01.982805 29242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:59:01.979570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.49.66:44029 (local address 127.20.49.67:42181)
1209 08:59:01.979717 (+ 147us) negotiation.cc:107] Waiting for socket to connect
1209 08:59:01.979734 (+ 17us) client_negotiation.cc:174] Beginning negotiation
1209 08:59:01.979857 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1209 08:59:01.980382 (+ 525us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1209 08:59:01.980385 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1209 08:59:01.980407 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1209 08:59:01.980728 (+ 321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:59:01.980737 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:59:01.981511 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1209 08:59:01.981515 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1209 08:59:01.982369 (+ 854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:59:01.982378 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:59:01.982497 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:59:01.982521 (+ 24us) client_negotiation.cc:769] Sending connection context
1209 08:59:01.982587 (+ 66us) client_negotiation.cc:240] Negotiation successful
1209 08:59:01.982664 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51,"spinlock_wait_cycles":48000}
I20251209 08:59:01.983279 29375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:59:01.979789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.49.67:42181 (local address 127.20.49.66:44029)
1209 08:59:01.980060 (+ 271us) server_negotiation.cc:207] Beginning negotiation
1209 08:59:01.980065 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1209 08:59:01.980078 (+ 13us) server_negotiation.cc:408] Connection header received
1209 08:59:01.980138 (+ 60us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1209 08:59:01.980141 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1209 08:59:01.980200 (+ 59us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1209 08:59:01.980271 (+ 71us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1209 08:59:01.980895 (+ 624us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:59:01.981402 (+ 507us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1209 08:59:01.982499 (+ 1097us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:59:01.983003 (+ 504us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:59:01.983040 (+ 37us) server_negotiation.cc:1036] Waiting for connection context
1209 08:59:01.983091 (+ 51us) server_negotiation.cc:300] Negotiation successful
1209 08:59:01.983166 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":171}
I20251209 08:59:01.983631 29376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:59:01.980080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.49.67:51593 (local address 127.20.49.65:39793)
1209 08:59:01.980199 (+ 119us) server_negotiation.cc:207] Beginning negotiation
1209 08:59:01.980204 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1209 08:59:01.980218 (+ 14us) server_negotiation.cc:408] Connection header received
1209 08:59:01.980270 (+ 52us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1209 08:59:01.980273 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1209 08:59:01.980321 (+ 48us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1209 08:59:01.980410 (+ 89us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1209 08:59:01.980874 (+ 464us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:59:01.981595 (+ 721us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1209 08:59:01.982661 (+ 1066us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:59:01.983198 (+ 537us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:59:01.983234 (+ 36us) server_negotiation.cc:1036] Waiting for connection context
1209 08:59:01.983446 (+ 212us) server_negotiation.cc:300] Negotiation successful
1209 08:59:01.983523 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20251209 08:59:01.983913 29161 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "32849c0a934c42868d51faf6d6a7a293" candidate_uuid: "7a440d3ea874483980ad5a12a47ce1d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84d473e778bd4553a757338c4cb05fb4" is_pre_election: true
I20251209 08:59:01.983991 29386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:59:01.979621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.49.65:39793 (local address 127.20.49.67:51593)
1209 08:59:01.979996 (+ 375us) negotiation.cc:107] Waiting for socket to connect
1209 08:59:01.980009 (+ 13us) client_negotiation.cc:174] Beginning negotiation
1209 08:59:01.980079 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1209 08:59:01.980424 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1209 08:59:01.980427 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1209 08:59:01.980472 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1209 08:59:01.980745 (+ 273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:59:01.980753 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:59:01.981729 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1209 08:59:01.981732 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1209 08:59:01.982522 (+ 790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:59:01.982529 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:59:01.983295 (+ 766us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:59:01.983308 (+ 13us) client_negotiation.cc:769] Sending connection context
1209 08:59:01.983780 (+ 472us) client_negotiation.cc:240] Negotiation successful
1209 08:59:01.983855 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":305,"mutex_wait_us":12,"spinlock_wait_cycles":3072,"thread_start_us":173,"threads_started":1}
I20251209 08:59:01.984215 29161 raft_consensus.cc:2468] T 32849c0a934c42868d51faf6d6a7a293 P 84d473e778bd4553a757338c4cb05fb4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a440d3ea874483980ad5a12a47ce1d9 in term 0.
I20251209 08:59:01.984711 29234 leader_election.cc:304] T 32849c0a934c42868d51faf6d6a7a293 P 7a440d3ea874483980ad5a12a47ce1d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a440d3ea874483980ad5a12a47ce1d9, 84d473e778bd4553a757338c4cb05fb4; no voters:
I20251209 08:59:01.984885 29019 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "32849c0a934c42868d51faf6d6a7a293" candidate_uuid: "7a440d3ea874483980ad5a12a47ce1d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3cbbcfcfb7634dc588875d9c22e2328e" is_pre_election: true
I20251209 08:59:01.985069 29385 raft_consensus.cc:2804] T 32849c0a934c42868d51faf6d6a7a293 P 7a440d3ea874483980ad5a12a47ce1d9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251209 08:59:01.985132 29019 raft_consensus.cc:2468] T 32849c0a934c42868d51faf6d6a7a293 P 3cbbcfcfb7634dc588875d9c22e2328e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a440d3ea874483980ad5a12a47ce1d9 in term 0.
I20251209 08:59:01.985162 29385 raft_consensus.cc:493] T 32849c0a934c42868d51faf6d6a7a293 P 7a440d3ea874483980ad5a12a47ce1d9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251209 08:59:01.985206 29385 raft_consensus.cc:3060] T 32849c0a934c42868d51faf6d6a7a293 P 7a440d3ea874483980ad5a12a47ce1d9 [term 0 FOLLOWER]: Advancing to term 1
I20251209 08:59:01.986414 29385 raft_consensus.cc:515] T 32849c0a934c42868d51faf6d6a7a293 P 7a440d3ea874483980ad5a12a47ce1d9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a440d3ea874483980ad5a12a47ce1d9" member_type: VOTER last_known_addr { host: "127.20.49.67" port: 38547 } } peers { permanent_uuid: "84d473e778bd4553a757338c4cb05fb4" member_type: VOTER last_known_addr { host: "127.20.49.66" port: 44029 } } peers { permanent_uuid: "3cbbcfcfb7634dc588875d9c22e2328e" member_type: VOTER last_known_addr { host: "127.20.49.65" port: 39793 } }
I20251209 08:59:01.986747 29385 leader_election.cc:290] T 32849c0a934c42868d51faf6d6a7a293 P 7a440d3ea874483980ad5a12a47ce1d9 [CANDIDATE]: Term 1 election: Requested vote from peers 84d473e778bd4553a757338c4cb05fb4 (127.20.49.66:44029), 3cbbcfcfb7634dc588875d9c22e2328e (127.20.49.65:39793)
I20251209 08:59:01.987083 29161 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "32849c0a934c42868d51faf6d6a7a293" candidate_uuid: "7a440d3ea874483980ad5a12a47ce1d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84d473e778bd4553a757338c4cb05fb4"
I20251209 08:59:01.987083 29019 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "32849c0a934c42868d51faf6d6a7a293" candidate_uuid: "7a440d3ea874483980ad5a12a47ce1d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3cbbcfcfb7634dc588875d9c22e2328e"
I20251209 08:59:01.987205 29019 raft_consensus.cc:3060] T 32849c0a934c42868d51faf6d6a7a293 P 3cbbcfcfb7634dc588875d9c22e2328e [term 0 FOLLOWER]: Advancing to term 1
I20251209 08:59:01.987205 29161 raft_consensus.cc:3060] T 32849c0a934c42868d51faf6d6a7a293 P 84d473e778bd4553a757338c4cb05fb4 [term 0 FOLLOWER]: Advancing to term 1
I20251209 08:59:01.988458 29019 raft_consensus.cc:2468] T 32849c0a934c42868d51faf6d6a7a293 P 3cbbcfcfb7634dc588875d9c22e2328e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a440d3ea874483980ad5a12a47ce1d9 in term 1.
I20251209 08:59:01.988458 29161 raft_consensus.cc:2468] T 32849c0a934c42868d51faf6d6a7a293 P 84d473e778bd4553a757338c4cb05fb4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a440d3ea874483980ad5a12a47ce1d9 in term 1.
I20251209 08:59:01.988761 29234 leader_election.cc:304] T 32849c0a934c42868d51faf6d6a7a293 P 7a440d3ea874483980ad5a12a47ce1d9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a440d3ea874483980ad5a12a47ce1d9, 84d473e778bd4553a757338c4cb05fb4; no voters:
I20251209 08:59:01.988935 29385 raft_consensus.cc:2804] T 32849c0a934c42868d51faf6d6a7a293 P 7a440d3ea874483980ad5a12a47ce1d9 [term 1 FOLLOWER]: Leader election won for term 1
I20251209 08:59:01.989167 29385 raft_consensus.cc:697] T 32849c0a934c42868d51faf6d6a7a293 P 7a440d3ea874483980ad5a12a47ce1d9 [term 1 LEADER]: Becoming Leader. State: Replica: 7a440d3ea874483980ad5a12a47ce1d9, State: Running, Role: LEADER
I20251209 08:59:01.989486 29385 consensus_queue.cc:237] T 32849c0a934c42868d51faf6d6a7a293 P 7a440d3ea874483980ad5a12a47ce1d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a440d3ea874483980ad5a12a47ce1d9" member_type: VOTER last_known_addr { host: "127.20.49.67" port: 38547 } } peers { permanent_uuid: "84d473e778bd4553a757338c4cb05fb4" member_type: VOTER last_known_addr { host: "127.20.49.66" port: 44029 } } peers { permanent_uuid: "3cbbcfcfb7634dc588875d9c22e2328e" member_type: VOTER last_known_addr { host: "127.20.49.65" port: 39793 } }
I20251209 08:59:01.992883 28857 catalog_manager.cc:5654] T 32849c0a934c42868d51faf6d6a7a293 P 7a440d3ea874483980ad5a12a47ce1d9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7a440d3ea874483980ad5a12a47ce1d9 (127.20.49.67). New cstate: current_term: 1 leader_uuid: "7a440d3ea874483980ad5a12a47ce1d9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a440d3ea874483980ad5a12a47ce1d9" member_type: VOTER last_known_addr { host: "127.20.49.67" port: 38547 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "84d473e778bd4553a757338c4cb05fb4" member_type: VOTER last_known_addr { host: "127.20.49.66" port: 44029 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3cbbcfcfb7634dc588875d9c22e2328e" member_type: VOTER last_known_addr { host: "127.20.49.65" port: 39793 } health_report { overall_health: UNKNOWN } } }
W20251209 08:59:02.002347 29350 tablet.cc:2378] T 32849c0a934c42868d51faf6d6a7a293 P 7a440d3ea874483980ad5a12a47ce1d9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20251209 08:59:02.005087 29372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:59:02.002230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35680 (local address 127.20.49.67:38547)
1209 08:59:02.002373 (+ 143us) server_negotiation.cc:207] Beginning negotiation
1209 08:59:02.002380 (+ 7us) server_negotiation.cc:400] Waiting for connection header
1209 08:59:02.002398 (+ 18us) server_negotiation.cc:408] Connection header received
1209 08:59:02.002481 (+ 83us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1209 08:59:02.002484 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1209 08:59:02.002565 (+ 81us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1209 08:59:02.002670 (+ 105us) server_negotiation.cc:227] Negotiated authn=TOKEN
1209 08:59:02.002980 (+ 310us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:59:02.003516 (+ 536us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1209 08:59:02.004134 (+ 618us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:59:02.004284 (+ 150us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:59:02.004335 (+ 51us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1209 08:59:02.004792 (+ 457us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1209 08:59:02.004884 (+ 92us) server_negotiation.cc:1036] Waiting for connection context
1209 08:59:02.004934 (+ 50us) server_negotiation.cc:300] Negotiation successful
1209 08:59:02.004982 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20251209 08:59:02.010427 29161 raft_consensus.cc:1275] T 32849c0a934c42868d51faf6d6a7a293 P 84d473e778bd4553a757338c4cb05fb4 [term 1 FOLLOWER]: Refusing update from remote peer 7a440d3ea874483980ad5a12a47ce1d9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251209 08:59:02.010427 29019 raft_consensus.cc:1275] T 32849c0a934c42868d51faf6d6a7a293 P 3cbbcfcfb7634dc588875d9c22e2328e [term 1 FOLLOWER]: Refusing update from remote peer 7a440d3ea874483980ad5a12a47ce1d9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251209 08:59:02.011042 29385 consensus_queue.cc:1048] T 32849c0a934c42868d51faf6d6a7a293 P 7a440d3ea874483980ad5a12a47ce1d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84d473e778bd4553a757338c4cb05fb4" member_type: VOTER last_known_addr { host: "127.20.49.66" port: 44029 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251209 08:59:02.011214 29387 consensus_queue.cc:1048] T 32849c0a934c42868d51faf6d6a7a293 P 7a440d3ea874483980ad5a12a47ce1d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3cbbcfcfb7634dc588875d9c22e2328e" member_type: VOTER last_known_addr { host: "127.20.49.65" port: 39793 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20251209 08:59:02.017001 29208 tablet.cc:2378] T 32849c0a934c42868d51faf6d6a7a293 P 84d473e778bd4553a757338c4cb05fb4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20251209 08:59:02.025475 28859 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59162:
table { table_name: "test-table" } modify_external_catalogs: true
I20251209 08:59:02.025646 28859 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59162:
table { table_name: "test-table" } modify_external_catalogs: true
I20251209 08:59:02.027910 28859 catalog_manager.cc:5941] T 00000000000000000000000000000000 P 7f2aac7cb38e4c609b71560ad54a20b1: Sending DeleteTablet for 3 replicas of tablet 32849c0a934c42868d51faf6d6a7a293
I20251209 08:59:02.028551 29283 tablet_service.cc:1552] Processing DeleteTablet for tablet 32849c0a934c42868d51faf6d6a7a293 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-09 08:59:02 UTC) from {username='oryx', principal='oryx/127.20.49.126@KRBTEST.COM'} at 127.0.0.1:35670
I20251209 08:59:02.028605 29141 tablet_service.cc:1552] Processing DeleteTablet for tablet 32849c0a934c42868d51faf6d6a7a293 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-09 08:59:02 UTC) from {username='oryx', principal='oryx/127.20.49.126@KRBTEST.COM'} at 127.0.0.1:52844
I20251209 08:59:02.028995 29400 tablet_replica.cc:333] T 32849c0a934c42868d51faf6d6a7a293 P 7a440d3ea874483980ad5a12a47ce1d9: stopping tablet replica
I20251209 08:59:02.029289 29400 raft_consensus.cc:2243] T 32849c0a934c42868d51faf6d6a7a293 P 7a440d3ea874483980ad5a12a47ce1d9 [term 1 LEADER]: Raft consensus shutting down.
I20251209 08:59:02.029485 28999 tablet_service.cc:1552] Processing DeleteTablet for tablet 32849c0a934c42868d51faf6d6a7a293 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-09 08:59:02 UTC) from {username='oryx', principal='oryx/127.20.49.126@KRBTEST.COM'} at 127.0.0.1:60562
I20251209 08:59:02.029568 28859 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59162:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251209 08:59:02.029995 29401 tablet_replica.cc:333] T 32849c0a934c42868d51faf6d6a7a293 P 84d473e778bd4553a757338c4cb05fb4: stopping tablet replica
W20251209 08:59:02.030058 28859 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251209 08:59:02.030020 29400 raft_consensus.cc:2272] T 32849c0a934c42868d51faf6d6a7a293 P 7a440d3ea874483980ad5a12a47ce1d9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251209 08:59:02.030256 29401 raft_consensus.cc:2243] T 32849c0a934c42868d51faf6d6a7a293 P 84d473e778bd4553a757338c4cb05fb4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251209 08:59:02.030563 29401 raft_consensus.cc:2272] T 32849c0a934c42868d51faf6d6a7a293 P 84d473e778bd4553a757338c4cb05fb4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251209 08:59:02.031566 29400 ts_tablet_manager.cc:1916] T 32849c0a934c42868d51faf6d6a7a293 P 7a440d3ea874483980ad5a12a47ce1d9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251209 08:59:02.034236 29400 ts_tablet_manager.cc:1929] T 32849c0a934c42868d51faf6d6a7a293 P 7a440d3ea874483980ad5a12a47ce1d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251209 08:59:02.034334 29400 log.cc:1199] T 32849c0a934c42868d51faf6d6a7a293 P 7a440d3ea874483980ad5a12a47ce1d9: Deleting WAL directory at /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765270668420570-20677-0/minicluster-data/ts-2/wal/wals/32849c0a934c42868d51faf6d6a7a293
I20251209 08:59:02.034638 29400 ts_tablet_manager.cc:1950] T 32849c0a934c42868d51faf6d6a7a293 P 7a440d3ea874483980ad5a12a47ce1d9: Deleting consensus metadata
I20251209 08:59:02.035709 29283 tablet_service.cc:1505] Processing CreateTablet for tablet c7c0d2789b5c4aceb7a1f436559d634b (DEFAULT_TABLE table=test-table [id=a8a0991b38364a7bbd9fdbfb0b9fe13a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251209 08:59:02.035959 29283 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c7c0d2789b5c4aceb7a1f436559d634b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251209 08:59:02.035943 29141 tablet_service.cc:1505] Processing CreateTablet for tablet c7c0d2789b5c4aceb7a1f436559d634b (DEFAULT_TABLE table=test-table [id=a8a0991b38364a7bbd9fdbfb0b9fe13a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251209 08:59:02.035979 28999 tablet_service.cc:1505] Processing CreateTablet for tablet c7c0d2789b5c4aceb7a1f436559d634b (DEFAULT_TABLE table=test-table [id=a8a0991b38364a7bbd9fdbfb0b9fe13a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251209 08:59:02.036173 29141 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c7c0d2789b5c4aceb7a1f436559d634b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251209 08:59:02.036245 28999 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c7c0d2789b5c4aceb7a1f436559d634b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251209 08:59:02.038209 29377 tablet_bootstrap.cc:492] T c7c0d2789b5c4aceb7a1f436559d634b P 7a440d3ea874483980ad5a12a47ce1d9: Bootstrap starting.
I20251209 08:59:02.038582 29378 tablet_bootstrap.cc:492] T c7c0d2789b5c4aceb7a1f436559d634b P 3cbbcfcfb7634dc588875d9c22e2328e: Bootstrap starting.
I20251209 08:59:02.038679 29379 tablet_bootstrap.cc:492] T c7c0d2789b5c4aceb7a1f436559d634b P 84d473e778bd4553a757338c4cb05fb4: Bootstrap starting.
I20251209 08:59:02.039305 29377 tablet_bootstrap.cc:654] T c7c0d2789b5c4aceb7a1f436559d634b P 7a440d3ea874483980ad5a12a47ce1d9: Neither blocks nor log segments found. Creating new log.
I20251209 08:59:02.039674 29379 tablet_bootstrap.cc:654] T c7c0d2789b5c4aceb7a1f436559d634b P 84d473e778bd4553a757338c4cb05fb4: Neither blocks nor log segments found. Creating new log.
I20251209 08:59:02.039682 29378 tablet_bootstrap.cc:654] T c7c0d2789b5c4aceb7a1f436559d634b P 3cbbcfcfb7634dc588875d9c22e2328e: Neither blocks nor log segments found. Creating new log.
I20251209 08:59:02.040586 28846 catalog_manager.cc:4985] TS 7a440d3ea874483980ad5a12a47ce1d9 (127.20.49.67:38547): tablet 32849c0a934c42868d51faf6d6a7a293 (table test-table [id=938e48322161461d89813bcf0665bc0b]) successfully deleted
I20251209 08:59:02.040906 29401 ts_tablet_manager.cc:1916] T 32849c0a934c42868d51faf6d6a7a293 P 84d473e778bd4553a757338c4cb05fb4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251209 08:59:02.041333 29379 tablet_bootstrap.cc:492] T c7c0d2789b5c4aceb7a1f436559d634b P 84d473e778bd4553a757338c4cb05fb4: No bootstrap required, opened a new log
I20251209 08:59:02.041407 29379 ts_tablet_manager.cc:1403] T c7c0d2789b5c4aceb7a1f436559d634b P 84d473e778bd4553a757338c4cb05fb4: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20251209 08:59:02.041848 29402 tablet_replica.cc:333] T 32849c0a934c42868d51faf6d6a7a293 P 3cbbcfcfb7634dc588875d9c22e2328e: stopping tablet replica
I20251209 08:59:02.041916 29379 raft_consensus.cc:359] T c7c0d2789b5c4aceb7a1f436559d634b P 84d473e778bd4553a757338c4cb05fb4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cbbcfcfb7634dc588875d9c22e2328e" member_type: VOTER last_known_addr { host: "127.20.49.65" port: 39793 } } peers { permanent_uuid: "84d473e778bd4553a757338c4cb05fb4" member_type: VOTER last_known_addr { host: "127.20.49.66" port: 44029 } } peers { permanent_uuid: "7a440d3ea874483980ad5a12a47ce1d9" member_type: VOTER last_known_addr { host: "127.20.49.67" port: 38547 } }
I20251209 08:59:02.042030 29379 raft_consensus.cc:385] T c7c0d2789b5c4aceb7a1f436559d634b P 84d473e778bd4553a757338c4cb05fb4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251209 08:59:02.042084 29379 raft_consensus.cc:740] T c7c0d2789b5c4aceb7a1f436559d634b P 84d473e778bd4553a757338c4cb05fb4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84d473e778bd4553a757338c4cb05fb4, State: Initialized, Role: FOLLOWER
I20251209 08:59:02.042086 29402 raft_consensus.cc:2243] T 32849c0a934c42868d51faf6d6a7a293 P 3cbbcfcfb7634dc588875d9c22e2328e [term 1 FOLLOWER]: Raft consensus shutting down.
I20251209 08:59:02.042219 29379 consensus_queue.cc:260] T c7c0d2789b5c4aceb7a1f436559d634b P 84d473e778bd4553a757338c4cb05fb4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cbbcfcfb7634dc588875d9c22e2328e" member_type: VOTER last_known_addr { host: "127.20.49.65" port: 39793 } } peers { permanent_uuid: "84d473e778bd4553a757338c4cb05fb4" member_type: VOTER last_known_addr { host: "127.20.49.66" port: 44029 } } peers { permanent_uuid: "7a440d3ea874483980ad5a12a47ce1d9" member_type: VOTER last_known_addr { host: "127.20.49.67" port: 38547 } }
I20251209 08:59:02.042375 29402 raft_consensus.cc:2272] T 32849c0a934c42868d51faf6d6a7a293 P 3cbbcfcfb7634dc588875d9c22e2328e [term 1 FOLLOWER]: Raft consensus is shut down!
I20251209 08:59:02.042502 29379 ts_tablet_manager.cc:1434] T c7c0d2789b5c4aceb7a1f436559d634b P 84d473e778bd4553a757338c4cb05fb4: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20251209 08:59:02.043737 29401 ts_tablet_manager.cc:1929] T 32849c0a934c42868d51faf6d6a7a293 P 84d473e778bd4553a757338c4cb05fb4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251209 08:59:02.043820 29401 log.cc:1199] T 32849c0a934c42868d51faf6d6a7a293 P 84d473e778bd4553a757338c4cb05fb4: Deleting WAL directory at /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765270668420570-20677-0/minicluster-data/ts-1/wal/wals/32849c0a934c42868d51faf6d6a7a293
I20251209 08:59:02.044086 29401 ts_tablet_manager.cc:1950] T 32849c0a934c42868d51faf6d6a7a293 P 84d473e778bd4553a757338c4cb05fb4: Deleting consensus metadata
I20251209 08:59:02.045632 29377 tablet_bootstrap.cc:492] T c7c0d2789b5c4aceb7a1f436559d634b P 7a440d3ea874483980ad5a12a47ce1d9: No bootstrap required, opened a new log
I20251209 08:59:02.045603 29402 ts_tablet_manager.cc:1916] T 32849c0a934c42868d51faf6d6a7a293 P 3cbbcfcfb7634dc588875d9c22e2328e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251209 08:59:02.045758 29377 ts_tablet_manager.cc:1403] T c7c0d2789b5c4aceb7a1f436559d634b P 7a440d3ea874483980ad5a12a47ce1d9: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.001s
I20251209 08:59:02.046286 29377 raft_consensus.cc:359] T c7c0d2789b5c4aceb7a1f436559d634b P 7a440d3ea874483980ad5a12a47ce1d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cbbcfcfb7634dc588875d9c22e2328e" member_type: VOTER last_known_addr { host: "127.20.49.65" port: 39793 } } peers { permanent_uuid: "84d473e778bd4553a757338c4cb05fb4" member_type: VOTER last_known_addr { host: "127.20.49.66" port: 44029 } } peers { permanent_uuid: "7a440d3ea874483980ad5a12a47ce1d9" member_type: VOTER last_known_addr { host: "127.20.49.67" port: 38547 } }
I20251209 08:59:02.046386 29377 raft_consensus.cc:385] T c7c0d2789b5c4aceb7a1f436559d634b P 7a440d3ea874483980ad5a12a47ce1d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251209 08:59:02.046425 29377 raft_consensus.cc:740] T c7c0d2789b5c4aceb7a1f436559d634b P 7a440d3ea874483980ad5a12a47ce1d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a440d3ea874483980ad5a12a47ce1d9, State: Initialized, Role: FOLLOWER
I20251209 08:59:02.046546 29377 consensus_queue.cc:260] T c7c0d2789b5c4aceb7a1f436559d634b P 7a440d3ea874483980ad5a12a47ce1d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cbbcfcfb7634dc588875d9c22e2328e" member_type: VOTER last_known_addr { host: "127.20.49.65" port: 39793 } } peers { permanent_uuid: "84d473e778bd4553a757338c4cb05fb4" member_type: VOTER last_known_addr { host: "127.20.49.66" port: 44029 } } peers { permanent_uuid: "7a440d3ea874483980ad5a12a47ce1d9" member_type: VOTER last_known_addr { host: "127.20.49.67" port: 38547 } }
I20251209 08:59:02.046823 29377 ts_tablet_manager.cc:1434] T c7c0d2789b5c4aceb7a1f436559d634b P 7a440d3ea874483980ad5a12a47ce1d9: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20251209 08:59:02.048362 28844 catalog_manager.cc:4985] TS 84d473e778bd4553a757338c4cb05fb4 (127.20.49.66:44029): tablet 32849c0a934c42868d51faf6d6a7a293 (table test-table [id=938e48322161461d89813bcf0665bc0b]) successfully deleted
I20251209 08:59:02.049309 29378 tablet_bootstrap.cc:492] T c7c0d2789b5c4aceb7a1f436559d634b P 3cbbcfcfb7634dc588875d9c22e2328e: No bootstrap required, opened a new log
I20251209 08:59:02.049407 29378 ts_tablet_manager.cc:1403] T c7c0d2789b5c4aceb7a1f436559d634b P 3cbbcfcfb7634dc588875d9c22e2328e: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.000s
I20251209 08:59:02.049659 29402 ts_tablet_manager.cc:1929] T 32849c0a934c42868d51faf6d6a7a293 P 3cbbcfcfb7634dc588875d9c22e2328e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251209 08:59:02.049803 29402 log.cc:1199] T 32849c0a934c42868d51faf6d6a7a293 P 3cbbcfcfb7634dc588875d9c22e2328e: Deleting WAL directory at /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765270668420570-20677-0/minicluster-data/ts-0/wal/wals/32849c0a934c42868d51faf6d6a7a293
I20251209 08:59:02.049973 29378 raft_consensus.cc:359] T c7c0d2789b5c4aceb7a1f436559d634b P 3cbbcfcfb7634dc588875d9c22e2328e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cbbcfcfb7634dc588875d9c22e2328e" member_type: VOTER last_known_addr { host: "127.20.49.65" port: 39793 } } peers { permanent_uuid: "84d473e778bd4553a757338c4cb05fb4" member_type: VOTER last_known_addr { host: "127.20.49.66" port: 44029 } } peers { permanent_uuid: "7a440d3ea874483980ad5a12a47ce1d9" member_type: VOTER last_known_addr { host: "127.20.49.67" port: 38547 } }
I20251209 08:59:02.050078 29378 raft_consensus.cc:385] T c7c0d2789b5c4aceb7a1f436559d634b P 3cbbcfcfb7634dc588875d9c22e2328e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251209 08:59:02.050119 29402 ts_tablet_manager.cc:1950] T 32849c0a934c42868d51faf6d6a7a293 P 3cbbcfcfb7634dc588875d9c22e2328e: Deleting consensus metadata
I20251209 08:59:02.050118 29378 raft_consensus.cc:740] T c7c0d2789b5c4aceb7a1f436559d634b P 3cbbcfcfb7634dc588875d9c22e2328e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3cbbcfcfb7634dc588875d9c22e2328e, State: Initialized, Role: FOLLOWER
I20251209 08:59:02.050333 29378 consensus_queue.cc:260] T c7c0d2789b5c4aceb7a1f436559d634b P 3cbbcfcfb7634dc588875d9c22e2328e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cbbcfcfb7634dc588875d9c22e2328e" member_type: VOTER last_known_addr { host: "127.20.49.65" port: 39793 } } peers { permanent_uuid: "84d473e778bd4553a757338c4cb05fb4" member_type: VOTER last_known_addr { host: "127.20.49.66" port: 44029 } } peers { permanent_uuid: "7a440d3ea874483980ad5a12a47ce1d9" member_type: VOTER last_known_addr { host: "127.20.49.67" port: 38547 } }
I20251209 08:59:02.050587 29378 ts_tablet_manager.cc:1434] T c7c0d2789b5c4aceb7a1f436559d634b P 3cbbcfcfb7634dc588875d9c22e2328e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20251209 08:59:02.051367 28847 catalog_manager.cc:4985] TS 3cbbcfcfb7634dc588875d9c22e2328e (127.20.49.65:39793): tablet 32849c0a934c42868d51faf6d6a7a293 (table test-table [id=938e48322161461d89813bcf0665bc0b]) successfully deleted
I20251209 08:59:02.064232 29396 raft_consensus.cc:493] T c7c0d2789b5c4aceb7a1f436559d634b P 7a440d3ea874483980ad5a12a47ce1d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251209 08:59:02.064356 29396 raft_consensus.cc:515] T c7c0d2789b5c4aceb7a1f436559d634b P 7a440d3ea874483980ad5a12a47ce1d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cbbcfcfb7634dc588875d9c22e2328e" member_type: VOTER last_known_addr { host: "127.20.49.65" port: 39793 } } peers { permanent_uuid: "84d473e778bd4553a757338c4cb05fb4" member_type: VOTER last_known_addr { host: "127.20.49.66" port: 44029 } } peers { permanent_uuid: "7a440d3ea874483980ad5a12a47ce1d9" member_type: VOTER last_known_addr { host: "127.20.49.67" port: 38547 } }
I20251209 08:59:02.064798 29396 leader_election.cc:290] T c7c0d2789b5c4aceb7a1f436559d634b P 7a440d3ea874483980ad5a12a47ce1d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3cbbcfcfb7634dc588875d9c22e2328e (127.20.49.65:39793), 84d473e778bd4553a757338c4cb05fb4 (127.20.49.66:44029)
I20251209 08:59:02.065213 29019 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c7c0d2789b5c4aceb7a1f436559d634b" candidate_uuid: "7a440d3ea874483980ad5a12a47ce1d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3cbbcfcfb7634dc588875d9c22e2328e" is_pre_election: true
I20251209 08:59:02.065272 29161 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c7c0d2789b5c4aceb7a1f436559d634b" candidate_uuid: "7a440d3ea874483980ad5a12a47ce1d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84d473e778bd4553a757338c4cb05fb4" is_pre_election: true
I20251209 08:59:02.065373 29019 raft_consensus.cc:2468] T c7c0d2789b5c4aceb7a1f436559d634b P 3cbbcfcfb7634dc588875d9c22e2328e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a440d3ea874483980ad5a12a47ce1d9 in term 0.
I20251209 08:59:02.065393 29161 raft_consensus.cc:2468] T c7c0d2789b5c4aceb7a1f436559d634b P 84d473e778bd4553a757338c4cb05fb4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a440d3ea874483980ad5a12a47ce1d9 in term 0.
I20251209 08:59:02.065730 29234 leader_election.cc:304] T c7c0d2789b5c4aceb7a1f436559d634b P 7a440d3ea874483980ad5a12a47ce1d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a440d3ea874483980ad5a12a47ce1d9, 84d473e778bd4553a757338c4cb05fb4; no voters:
I20251209 08:59:02.066005 29396 raft_consensus.cc:2804] T c7c0d2789b5c4aceb7a1f436559d634b P 7a440d3ea874483980ad5a12a47ce1d9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251209 08:59:02.066076 29396 raft_consensus.cc:493] T c7c0d2789b5c4aceb7a1f436559d634b P 7a440d3ea874483980ad5a12a47ce1d9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251209 08:59:02.066118 29396 raft_consensus.cc:3060] T c7c0d2789b5c4aceb7a1f436559d634b P 7a440d3ea874483980ad5a12a47ce1d9 [term 0 FOLLOWER]: Advancing to term 1
I20251209 08:59:02.067451 29396 raft_consensus.cc:515] T c7c0d2789b5c4aceb7a1f436559d634b P 7a440d3ea874483980ad5a12a47ce1d9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cbbcfcfb7634dc588875d9c22e2328e" member_type: VOTER last_known_addr { host: "127.20.49.65" port: 39793 } } peers { permanent_uuid: "84d473e778bd4553a757338c4cb05fb4" member_type: VOTER last_known_addr { host: "127.20.49.66" port: 44029 } } peers { permanent_uuid: "7a440d3ea874483980ad5a12a47ce1d9" member_type: VOTER last_known_addr { host: "127.20.49.67" port: 38547 } }
I20251209 08:59:02.067844 29396 leader_election.cc:290] T c7c0d2789b5c4aceb7a1f436559d634b P 7a440d3ea874483980ad5a12a47ce1d9 [CANDIDATE]: Term 1 election: Requested vote from peers 3cbbcfcfb7634dc588875d9c22e2328e (127.20.49.65:39793), 84d473e778bd4553a757338c4cb05fb4 (127.20.49.66:44029)
I20251209 08:59:02.068166 29161 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c7c0d2789b5c4aceb7a1f436559d634b" candidate_uuid: "7a440d3ea874483980ad5a12a47ce1d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84d473e778bd4553a757338c4cb05fb4"
I20251209 08:59:02.068166 29019 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c7c0d2789b5c4aceb7a1f436559d634b" candidate_uuid: "7a440d3ea874483980ad5a12a47ce1d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3cbbcfcfb7634dc588875d9c22e2328e"
I20251209 08:59:02.068292 29019 raft_consensus.cc:3060] T c7c0d2789b5c4aceb7a1f436559d634b P 3cbbcfcfb7634dc588875d9c22e2328e [term 0 FOLLOWER]: Advancing to term 1
I20251209 08:59:02.068292 29161 raft_consensus.cc:3060] T c7c0d2789b5c4aceb7a1f436559d634b P 84d473e778bd4553a757338c4cb05fb4 [term 0 FOLLOWER]: Advancing to term 1
I20251209 08:59:02.069352 29161 raft_consensus.cc:2468] T c7c0d2789b5c4aceb7a1f436559d634b P 84d473e778bd4553a757338c4cb05fb4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a440d3ea874483980ad5a12a47ce1d9 in term 1.
I20251209 08:59:02.069352 29019 raft_consensus.cc:2468] T c7c0d2789b5c4aceb7a1f436559d634b P 3cbbcfcfb7634dc588875d9c22e2328e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a440d3ea874483980ad5a12a47ce1d9 in term 1.
I20251209 08:59:02.069741 29237 leader_election.cc:304] T c7c0d2789b5c4aceb7a1f436559d634b P 7a440d3ea874483980ad5a12a47ce1d9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3cbbcfcfb7634dc588875d9c22e2328e, 7a440d3ea874483980ad5a12a47ce1d9; no voters:
I20251209 08:59:02.069955 29396 raft_consensus.cc:2804] T c7c0d2789b5c4aceb7a1f436559d634b P 7a440d3ea874483980ad5a12a47ce1d9 [term 1 FOLLOWER]: Leader election won for term 1
I20251209 08:59:02.070048 29396 raft_consensus.cc:697] T c7c0d2789b5c4aceb7a1f436559d634b P 7a440d3ea874483980ad5a12a47ce1d9 [term 1 LEADER]: Becoming Leader. State: Replica: 7a440d3ea874483980ad5a12a47ce1d9, State: Running, Role: LEADER
I20251209 08:59:02.070192 29396 consensus_queue.cc:237] T c7c0d2789b5c4aceb7a1f436559d634b P 7a440d3ea874483980ad5a12a47ce1d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cbbcfcfb7634dc588875d9c22e2328e" member_type: VOTER last_known_addr { host: "127.20.49.65" port: 39793 } } peers { permanent_uuid: "84d473e778bd4553a757338c4cb05fb4" member_type: VOTER last_known_addr { host: "127.20.49.66" port: 44029 } } peers { permanent_uuid: "7a440d3ea874483980ad5a12a47ce1d9" member_type: VOTER last_known_addr { host: "127.20.49.67" port: 38547 } }
I20251209 08:59:02.071703 28857 catalog_manager.cc:5654] T c7c0d2789b5c4aceb7a1f436559d634b P 7a440d3ea874483980ad5a12a47ce1d9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7a440d3ea874483980ad5a12a47ce1d9 (127.20.49.67). New cstate: current_term: 1 leader_uuid: "7a440d3ea874483980ad5a12a47ce1d9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cbbcfcfb7634dc588875d9c22e2328e" member_type: VOTER last_known_addr { host: "127.20.49.65" port: 39793 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "84d473e778bd4553a757338c4cb05fb4" member_type: VOTER last_known_addr { host: "127.20.49.66" port: 44029 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a440d3ea874483980ad5a12a47ce1d9" member_type: VOTER last_known_addr { host: "127.20.49.67" port: 38547 } health_report { overall_health: HEALTHY } } }
Dec 09 08:59:02 dist-test-slave-8w6c krb5kdc[28812](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765270740, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.49.126@KRBTEST.COM for oryx/127.20.49.126@KRBTEST.COM
I20251209 08:59:02.115567 28909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:59:02.107700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59164 (local address 127.20.49.126:41745)
1209 08:59:02.107841 (+ 141us) server_negotiation.cc:207] Beginning negotiation
1209 08:59:02.107846 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1209 08:59:02.108053 (+ 207us) server_negotiation.cc:408] Connection header received
1209 08:59:02.108195 (+ 142us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1209 08:59:02.108198 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1209 08:59:02.108234 (+ 36us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1209 08:59:02.108295 (+ 61us) server_negotiation.cc:227] Negotiated authn=SASL
1209 08:59:02.109214 (+ 919us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:59:02.109749 (+ 535us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1209 08:59:02.110329 (+ 580us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:59:02.110500 (+ 171us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:59:02.112582 (+ 2082us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1209 08:59:02.112600 (+ 18us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1209 08:59:02.112602 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1209 08:59:02.112623 (+ 21us) server_negotiation.cc:923] calling sasl_server_start()
1209 08:59:02.114069 (+ 1446us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1209 08:59:02.114501 (+ 432us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1209 08:59:02.114504 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1209 08:59:02.114506 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1209 08:59:02.114549 (+ 43us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1209 08:59:02.114746 (+ 197us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1209 08:59:02.114749 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1209 08:59:02.114751 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1209 08:59:02.114901 (+ 150us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1209 08:59:02.114995 (+ 94us) server_negotiation.cc:1036] Waiting for connection context
1209 08:59:02.115270 (+ 275us) server_negotiation.cc:300] Negotiation successful
1209 08:59:02.115444 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":69,"server-negotiator.queue_time_us":42}
I20251209 08:59:02.115567 29412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:59:02.107642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.49.126:41745 (local address 127.0.0.1:59164)
1209 08:59:02.107971 (+ 329us) negotiation.cc:107] Waiting for socket to connect
1209 08:59:02.107983 (+ 12us) client_negotiation.cc:174] Beginning negotiation
1209 08:59:02.108109 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1209 08:59:02.108338 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1209 08:59:02.108341 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1209 08:59:02.108883 (+ 542us) client_negotiation.cc:189] Negotiated authn=SASL
1209 08:59:02.109087 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:59:02.109094 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:59:02.109856 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1209 08:59:02.109859 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1209 08:59:02.110232 (+ 373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:59:02.110239 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:59:02.110350 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:59:02.110996 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1209 08:59:02.111010 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
1209 08:59:02.112457 (+ 1447us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1209 08:59:02.114180 (+ 1723us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1209 08:59:02.114189 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1209 08:59:02.114195 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
1209 08:59:02.114414 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1209 08:59:02.114628 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1209 08:59:02.114631 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1209 08:59:02.114633 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1209 08:59:02.114673 (+ 40us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1209 08:59:02.115002 (+ 329us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1209 08:59:02.115012 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1209 08:59:02.115116 (+ 104us) client_negotiation.cc:769] Sending connection context
1209 08:59:02.115217 (+ 101us) client_negotiation.cc:240] Negotiation successful
1209 08:59:02.115375 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":122,"threads_started":1}
I20251209 08:59:02.119704 28857 catalog_manager.cc:2257] Servicing CreateTable request from {username='oryx', principal='oryx/127.20.49.126@KRBTEST.COM'} at 127.0.0.1:59164:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20251209 08:59:02.120429 28857 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251209 08:59:02.125998 29141 tablet_service.cc:1505] Processing CreateTablet for tablet 0ea73d83465640babe82c28dfc0c369e (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=3ee08b784d1b4c29aaacf2138bc61536]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20251209 08:59:02.126008 28999 tablet_service.cc:1505] Processing CreateTablet for tablet 0ea73d83465640babe82c28dfc0c369e (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=3ee08b784d1b4c29aaacf2138bc61536]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20251209 08:59:02.126330 29141 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0ea73d83465640babe82c28dfc0c369e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251209 08:59:02.126382 28999 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0ea73d83465640babe82c28dfc0c369e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251209 08:59:02.128527 29379 tablet_bootstrap.cc:492] T 0ea73d83465640babe82c28dfc0c369e P 84d473e778bd4553a757338c4cb05fb4: Bootstrap starting.
I20251209 08:59:02.129022 29283 tablet_service.cc:1505] Processing CreateTablet for tablet 0ea73d83465640babe82c28dfc0c369e (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=3ee08b784d1b4c29aaacf2138bc61536]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20251209 08:59:02.129191 29378 tablet_bootstrap.cc:492] T 0ea73d83465640babe82c28dfc0c369e P 3cbbcfcfb7634dc588875d9c22e2328e: Bootstrap starting.
I20251209 08:59:02.129416 29283 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0ea73d83465640babe82c28dfc0c369e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251209 08:59:02.129834 29379 tablet_bootstrap.cc:654] T 0ea73d83465640babe82c28dfc0c369e P 84d473e778bd4553a757338c4cb05fb4: Neither blocks nor log segments found. Creating new log.
I20251209 08:59:02.130237 29378 tablet_bootstrap.cc:654] T 0ea73d83465640babe82c28dfc0c369e P 3cbbcfcfb7634dc588875d9c22e2328e: Neither blocks nor log segments found. Creating new log.
I20251209 08:59:02.131552 29379 tablet_bootstrap.cc:492] T 0ea73d83465640babe82c28dfc0c369e P 84d473e778bd4553a757338c4cb05fb4: No bootstrap required, opened a new log
I20251209 08:59:02.131646 29379 ts_tablet_manager.cc:1403] T 0ea73d83465640babe82c28dfc0c369e P 84d473e778bd4553a757338c4cb05fb4: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20251209 08:59:02.132118 29379 raft_consensus.cc:359] T 0ea73d83465640babe82c28dfc0c369e P 84d473e778bd4553a757338c4cb05fb4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cbbcfcfb7634dc588875d9c22e2328e" member_type: VOTER last_known_addr { host: "127.20.49.65" port: 39793 } } peers { permanent_uuid: "84d473e778bd4553a757338c4cb05fb4" member_type: VOTER last_known_addr { host: "127.20.49.66" port: 44029 } } peers { permanent_uuid: "7a440d3ea874483980ad5a12a47ce1d9" member_type: VOTER last_known_addr { host: "127.20.49.67" port: 38547 } }
I20251209 08:59:02.132249 29379 raft_consensus.cc:385] T 0ea73d83465640babe82c28dfc0c369e P 84d473e778bd4553a757338c4cb05fb4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251209 08:59:02.132289 29379 raft_consensus.cc:740] T 0ea73d83465640babe82c28dfc0c369e P 84d473e778bd4553a757338c4cb05fb4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84d473e778bd4553a757338c4cb05fb4, State: Initialized, Role: FOLLOWER
I20251209 08:59:02.132412 29379 consensus_queue.cc:260] T 0ea73d83465640babe82c28dfc0c369e P 84d473e778bd4553a757338c4cb05fb4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cbbcfcfb7634dc588875d9c22e2328e" member_type: VOTER last_known_addr { host: "127.20.49.65" port: 39793 } } peers { permanent_uuid: "84d473e778bd4553a757338c4cb05fb4" member_type: VOTER last_known_addr { host: "127.20.49.66" port: 44029 } } peers { permanent_uuid: "7a440d3ea874483980ad5a12a47ce1d9" member_type: VOTER last_known_addr { host: "127.20.49.67" port: 38547 } }
I20251209 08:59:02.132678 29379 ts_tablet_manager.cc:1434] T 0ea73d83465640babe82c28dfc0c369e P 84d473e778bd4553a757338c4cb05fb4: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20251209 08:59:02.132915 29377 tablet_bootstrap.cc:492] T 0ea73d83465640babe82c28dfc0c369e P 7a440d3ea874483980ad5a12a47ce1d9: Bootstrap starting.
I20251209 08:59:02.133199 29378 tablet_bootstrap.cc:492] T 0ea73d83465640babe82c28dfc0c369e P 3cbbcfcfb7634dc588875d9c22e2328e: No bootstrap required, opened a new log
I20251209 08:59:02.133276 29378 ts_tablet_manager.cc:1403] T 0ea73d83465640babe82c28dfc0c369e P 3cbbcfcfb7634dc588875d9c22e2328e: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20251209 08:59:02.133770 29378 raft_consensus.cc:359] T 0ea73d83465640babe82c28dfc0c369e P 3cbbcfcfb7634dc588875d9c22e2328e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cbbcfcfb7634dc588875d9c22e2328e" member_type: VOTER last_known_addr { host: "127.20.49.65" port: 39793 } } peers { permanent_uuid: "84d473e778bd4553a757338c4cb05fb4" member_type: VOTER last_known_addr { host: "127.20.49.66" port: 44029 } } peers { permanent_uuid: "7a440d3ea874483980ad5a12a47ce1d9" member_type: VOTER last_known_addr { host: "127.20.49.67" port: 38547 } }
I20251209 08:59:02.133874 29378 raft_consensus.cc:385] T 0ea73d83465640babe82c28dfc0c369e P 3cbbcfcfb7634dc588875d9c22e2328e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251209 08:59:02.133905 29378 raft_consensus.cc:740] T 0ea73d83465640babe82c28dfc0c369e P 3cbbcfcfb7634dc588875d9c22e2328e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3cbbcfcfb7634dc588875d9c22e2328e, State: Initialized, Role: FOLLOWER
I20251209 08:59:02.134033 29377 tablet_bootstrap.cc:654] T 0ea73d83465640babe82c28dfc0c369e P 7a440d3ea874483980ad5a12a47ce1d9: Neither blocks nor log segments found. Creating new log.
I20251209 08:59:02.134017 29378 consensus_queue.cc:260] T 0ea73d83465640babe82c28dfc0c369e P 3cbbcfcfb7634dc588875d9c22e2328e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cbbcfcfb7634dc588875d9c22e2328e" member_type: VOTER last_known_addr { host: "127.20.49.65" port: 39793 } } peers { permanent_uuid: "84d473e778bd4553a757338c4cb05fb4" member_type: VOTER last_known_addr { host: "127.20.49.66" port: 44029 } } peers { permanent_uuid: "7a440d3ea874483980ad5a12a47ce1d9" member_type: VOTER last_known_addr { host: "127.20.49.67" port: 38547 } }
I20251209 08:59:02.134253 29378 ts_tablet_manager.cc:1434] T 0ea73d83465640babe82c28dfc0c369e P 3cbbcfcfb7634dc588875d9c22e2328e: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20251209 08:59:02.134474 29384 tablet_replica.cc:442] T 0ea73d83465640babe82c28dfc0c369e P 3cbbcfcfb7634dc588875d9c22e2328e: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cbbcfcfb7634dc588875d9c22e2328e" member_type: VOTER last_known_addr { host: "127.20.49.65" port: 39793 } } peers { permanent_uuid: "84d473e778bd4553a757338c4cb05fb4" member_type: VOTER last_known_addr { host: "127.20.49.66" port: 44029 } } peers { permanent_uuid: "7a440d3ea874483980ad5a12a47ce1d9" member_type: VOTER last_known_addr { host: "127.20.49.67" port: 38547 } } }
I20251209 08:59:02.134584 29384 tablet_replica.cc:445] T 0ea73d83465640babe82c28dfc0c369e P 3cbbcfcfb7634dc588875d9c22e2328e: This TxnStatusTablet replica's current role is: FOLLOWER
I20251209 08:59:02.134860 29383 tablet_replica.cc:442] T 0ea73d83465640babe82c28dfc0c369e P 84d473e778bd4553a757338c4cb05fb4: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cbbcfcfb7634dc588875d9c22e2328e" member_type: VOTER last_known_addr { host: "127.20.49.65" port: 39793 } } peers { permanent_uuid: "84d473e778bd4553a757338c4cb05fb4" member_type: VOTER last_known_addr { host: "127.20.49.66" port: 44029 } } peers { permanent_uuid: "7a440d3ea874483980ad5a12a47ce1d9" member_type: VOTER last_known_addr { host: "127.20.49.67" port: 38547 } } }
I20251209 08:59:02.134976 29383 tablet_replica.cc:445] T 0ea73d83465640babe82c28dfc0c369e P 84d473e778bd4553a757338c4cb05fb4: This TxnStatusTablet replica's current role is: FOLLOWER
I20251209 08:59:02.136094 29377 tablet_bootstrap.cc:492] T 0ea73d83465640babe82c28dfc0c369e P 7a440d3ea874483980ad5a12a47ce1d9: No bootstrap required, opened a new log
I20251209 08:59:02.136198 29377 ts_tablet_manager.cc:1403] T 0ea73d83465640babe82c28dfc0c369e P 7a440d3ea874483980ad5a12a47ce1d9: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20251209 08:59:02.136700 29377 raft_consensus.cc:359] T 0ea73d83465640babe82c28dfc0c369e P 7a440d3ea874483980ad5a12a47ce1d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cbbcfcfb7634dc588875d9c22e2328e" member_type: VOTER last_known_addr { host: "127.20.49.65" port: 39793 } } peers { permanent_uuid: "84d473e778bd4553a757338c4cb05fb4" member_type: VOTER last_known_addr { host: "127.20.49.66" port: 44029 } } peers { permanent_uuid: "7a440d3ea874483980ad5a12a47ce1d9" member_type: VOTER last_known_addr { host: "127.20.49.67" port: 38547 } }
I20251209 08:59:02.136816 29377 raft_consensus.cc:385] T 0ea73d83465640babe82c28dfc0c369e P 7a440d3ea874483980ad5a12a47ce1d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251209 08:59:02.136859 29377 raft_consensus.cc:740] T 0ea73d83465640babe82c28dfc0c369e P 7a440d3ea874483980ad5a12a47ce1d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a440d3ea874483980ad5a12a47ce1d9, State: Initialized, Role: FOLLOWER
I20251209 08:59:02.136982 29377 consensus_queue.cc:260] T 0ea73d83465640babe82c28dfc0c369e P 7a440d3ea874483980ad5a12a47ce1d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cbbcfcfb7634dc588875d9c22e2328e" member_type: VOTER last_known_addr { host: "127.20.49.65" port: 39793 } } peers { permanent_uuid: "84d473e778bd4553a757338c4cb05fb4" member_type: VOTER last_known_addr { host: "127.20.49.66" port: 44029 } } peers { permanent_uuid: "7a440d3ea874483980ad5a12a47ce1d9" member_type: VOTER last_known_addr { host: "127.20.49.67" port: 38547 } }
I20251209 08:59:02.137231 29377 ts_tablet_manager.cc:1434] T 0ea73d83465640babe82c28dfc0c369e P 7a440d3ea874483980ad5a12a47ce1d9: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20251209 08:59:02.137413 29396 tablet_replica.cc:442] T 0ea73d83465640babe82c28dfc0c369e P 7a440d3ea874483980ad5a12a47ce1d9: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cbbcfcfb7634dc588875d9c22e2328e" member_type: VOTER last_known_addr { host: "127.20.49.65" port: 39793 } } peers { permanent_uuid: "84d473e778bd4553a757338c4cb05fb4" member_type: VOTER last_known_addr { host: "127.20.49.66" port: 44029 } } peers { permanent_uuid: "7a440d3ea874483980ad5a12a47ce1d9" member_type: VOTER last_known_addr { host: "127.20.49.67" port: 38547 } } }
I20251209 08:59:02.137497 29396 tablet_replica.cc:445] T 0ea73d83465640babe82c28dfc0c369e P 7a440d3ea874483980ad5a12a47ce1d9: This TxnStatusTablet replica's current role is: FOLLOWER
I20251209 08:59:02.151769 29384 raft_consensus.cc:493] T 0ea73d83465640babe82c28dfc0c369e P 3cbbcfcfb7634dc588875d9c22e2328e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251209 08:59:02.151908 29384 raft_consensus.cc:515] T 0ea73d83465640babe82c28dfc0c369e P 3cbbcfcfb7634dc588875d9c22e2328e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cbbcfcfb7634dc588875d9c22e2328e" member_type: VOTER last_known_addr { host: "127.20.49.65" port: 39793 } } peers { permanent_uuid: "84d473e778bd4553a757338c4cb05fb4" member_type: VOTER last_known_addr { host: "127.20.49.66" port: 44029 } } peers { permanent_uuid: "7a440d3ea874483980ad5a12a47ce1d9" member_type: VOTER last_known_addr { host: "127.20.49.67" port: 38547 } }
I20251209 08:59:02.152904 29384 leader_election.cc:290] T 0ea73d83465640babe82c28dfc0c369e P 3cbbcfcfb7634dc588875d9c22e2328e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 84d473e778bd4553a757338c4cb05fb4 (127.20.49.66:44029), 7a440d3ea874483980ad5a12a47ce1d9 (127.20.49.67:38547)
I20251209 08:59:02.156598 29416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:59:02.153124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.49.66:44029 (local address 127.20.49.65:47403)
1209 08:59:02.153464 (+ 340us) negotiation.cc:107] Waiting for socket to connect
1209 08:59:02.153480 (+ 16us) client_negotiation.cc:174] Beginning negotiation
1209 08:59:02.153636 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1209 08:59:02.154070 (+ 434us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1209 08:59:02.154074 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
1209 08:59:02.154094 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1209 08:59:02.154335 (+ 241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:59:02.154341 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:59:02.155398 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1209 08:59:02.155402 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1209 08:59:02.156005 (+ 603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:59:02.156012 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:59:02.156141 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:59:02.156165 (+ 24us) client_negotiation.cc:769] Sending connection context
1209 08:59:02.156219 (+ 54us) client_negotiation.cc:240] Negotiation successful
1209 08:59:02.156438 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":103,"threads_started":1}
I20251209 08:59:02.156891 29375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:59:02.153234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.49.65:47403 (local address 127.20.49.66:44029)
1209 08:59:02.153388 (+ 154us) server_negotiation.cc:207] Beginning negotiation
1209 08:59:02.153393 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1209 08:59:02.153556 (+ 163us) server_negotiation.cc:408] Connection header received
1209 08:59:02.153885 (+ 329us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1209 08:59:02.153892 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
1209 08:59:02.153956 (+ 64us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1209 08:59:02.154027 (+ 71us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1209 08:59:02.154471 (+ 444us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:59:02.155256 (+ 785us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1209 08:59:02.156128 (+ 872us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:59:02.156588 (+ 460us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:59:02.156637 (+ 49us) server_negotiation.cc:1036] Waiting for connection context
1209 08:59:02.156698 (+ 61us) server_negotiation.cc:300] Negotiation successful
1209 08:59:02.156771 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20251209 08:59:02.157413 29161 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0ea73d83465640babe82c28dfc0c369e" candidate_uuid: "3cbbcfcfb7634dc588875d9c22e2328e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84d473e778bd4553a757338c4cb05fb4" is_pre_election: true
I20251209 08:59:02.157485 29372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:59:02.153543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.49.65:54623 (local address 127.20.49.67:38547)
1209 08:59:02.153717 (+ 174us) server_negotiation.cc:207] Beginning negotiation
1209 08:59:02.153723 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1209 08:59:02.153740 (+ 17us) server_negotiation.cc:408] Connection header received
1209 08:59:02.153796 (+ 56us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1209 08:59:02.153799 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1209 08:59:02.153853 (+ 54us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1209 08:59:02.153933 (+ 80us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1209 08:59:02.154507 (+ 574us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:59:02.155256 (+ 749us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1209 08:59:02.156460 (+ 1204us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:59:02.156940 (+ 480us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:59:02.156978 (+ 38us) server_negotiation.cc:1036] Waiting for connection context
1209 08:59:02.157294 (+ 316us) server_negotiation.cc:300] Negotiation successful
1209 08:59:02.157373 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20251209 08:59:02.157586 29161 raft_consensus.cc:2468] T 0ea73d83465640babe82c28dfc0c369e P 84d473e778bd4553a757338c4cb05fb4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3cbbcfcfb7634dc588875d9c22e2328e in term 0.
I20251209 08:59:02.157886 29417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:59:02.153223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.49.67:38547 (local address 127.20.49.65:54623)
1209 08:59:02.153550 (+ 327us) negotiation.cc:107] Waiting for socket to connect
1209 08:59:02.153564 (+ 14us) client_negotiation.cc:174] Beginning negotiation
1209 08:59:02.153656 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1209 08:59:02.153957 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1209 08:59:02.153961 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
1209 08:59:02.153982 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1209 08:59:02.154317 (+ 335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:59:02.154326 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:59:02.155379 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1209 08:59:02.155384 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1209 08:59:02.156305 (+ 921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:59:02.156316 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:59:02.157158 (+ 842us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:59:02.157175 (+ 17us) client_negotiation.cc:769] Sending connection context
1209 08:59:02.157657 (+ 482us) client_negotiation.cc:240] Negotiation successful
1209 08:59:02.157747 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":100,"threads_started":1}
I20251209 08:59:02.157969 28950 leader_election.cc:304] T 0ea73d83465640babe82c28dfc0c369e P 3cbbcfcfb7634dc588875d9c22e2328e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3cbbcfcfb7634dc588875d9c22e2328e, 84d473e778bd4553a757338c4cb05fb4; no voters:
I20251209 08:59:02.158231 29384 raft_consensus.cc:2804] T 0ea73d83465640babe82c28dfc0c369e P 3cbbcfcfb7634dc588875d9c22e2328e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251209 08:59:02.158293 29384 raft_consensus.cc:493] T 0ea73d83465640babe82c28dfc0c369e P 3cbbcfcfb7634dc588875d9c22e2328e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251209 08:59:02.158337 29384 raft_consensus.cc:3060] T 0ea73d83465640babe82c28dfc0c369e P 3cbbcfcfb7634dc588875d9c22e2328e [term 0 FOLLOWER]: Advancing to term 1
I20251209 08:59:02.158399 29302 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0ea73d83465640babe82c28dfc0c369e" candidate_uuid: "3cbbcfcfb7634dc588875d9c22e2328e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a440d3ea874483980ad5a12a47ce1d9" is_pre_election: true
I20251209 08:59:02.158604 29302 raft_consensus.cc:2468] T 0ea73d83465640babe82c28dfc0c369e P 7a440d3ea874483980ad5a12a47ce1d9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3cbbcfcfb7634dc588875d9c22e2328e in term 0.
I20251209 08:59:02.159466 29384 raft_consensus.cc:515] T 0ea73d83465640babe82c28dfc0c369e P 3cbbcfcfb7634dc588875d9c22e2328e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cbbcfcfb7634dc588875d9c22e2328e" member_type: VOTER last_known_addr { host: "127.20.49.65" port: 39793 } } peers { permanent_uuid: "84d473e778bd4553a757338c4cb05fb4" member_type: VOTER last_known_addr { host: "127.20.49.66" port: 44029 } } peers { permanent_uuid: "7a440d3ea874483980ad5a12a47ce1d9" member_type: VOTER last_known_addr { host: "127.20.49.67" port: 38547 } }
I20251209 08:59:02.159828 29384 leader_election.cc:290] T 0ea73d83465640babe82c28dfc0c369e P 3cbbcfcfb7634dc588875d9c22e2328e [CANDIDATE]: Term 1 election: Requested vote from peers 84d473e778bd4553a757338c4cb05fb4 (127.20.49.66:44029), 7a440d3ea874483980ad5a12a47ce1d9 (127.20.49.67:38547)
I20251209 08:59:02.160161 29302 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0ea73d83465640babe82c28dfc0c369e" candidate_uuid: "3cbbcfcfb7634dc588875d9c22e2328e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a440d3ea874483980ad5a12a47ce1d9"
I20251209 08:59:02.160270 29302 raft_consensus.cc:3060] T 0ea73d83465640babe82c28dfc0c369e P 7a440d3ea874483980ad5a12a47ce1d9 [term 0 FOLLOWER]: Advancing to term 1
I20251209 08:59:02.160228 29161 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0ea73d83465640babe82c28dfc0c369e" candidate_uuid: "3cbbcfcfb7634dc588875d9c22e2328e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84d473e778bd4553a757338c4cb05fb4"
I20251209 08:59:02.160358 29161 raft_consensus.cc:3060] T 0ea73d83465640babe82c28dfc0c369e P 84d473e778bd4553a757338c4cb05fb4 [term 0 FOLLOWER]: Advancing to term 1
I20251209 08:59:02.161175 29302 raft_consensus.cc:2468] T 0ea73d83465640babe82c28dfc0c369e P 7a440d3ea874483980ad5a12a47ce1d9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3cbbcfcfb7634dc588875d9c22e2328e in term 1.
I20251209 08:59:02.161456 28952 leader_election.cc:304] T 0ea73d83465640babe82c28dfc0c369e P 3cbbcfcfb7634dc588875d9c22e2328e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3cbbcfcfb7634dc588875d9c22e2328e, 7a440d3ea874483980ad5a12a47ce1d9; no voters:
I20251209 08:59:02.161499 29161 raft_consensus.cc:2468] T 0ea73d83465640babe82c28dfc0c369e P 84d473e778bd4553a757338c4cb05fb4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3cbbcfcfb7634dc588875d9c22e2328e in term 1.
I20251209 08:59:02.161648 29384 raft_consensus.cc:2804] T 0ea73d83465640babe82c28dfc0c369e P 3cbbcfcfb7634dc588875d9c22e2328e [term 1 FOLLOWER]: Leader election won for term 1
I20251209 08:59:02.161907 29384 raft_consensus.cc:697] T 0ea73d83465640babe82c28dfc0c369e P 3cbbcfcfb7634dc588875d9c22e2328e [term 1 LEADER]: Becoming Leader. State: Replica: 3cbbcfcfb7634dc588875d9c22e2328e, State: Running, Role: LEADER
I20251209 08:59:02.162153 29384 consensus_queue.cc:237] T 0ea73d83465640babe82c28dfc0c369e P 3cbbcfcfb7634dc588875d9c22e2328e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cbbcfcfb7634dc588875d9c22e2328e" member_type: VOTER last_known_addr { host: "127.20.49.65" port: 39793 } } peers { permanent_uuid: "84d473e778bd4553a757338c4cb05fb4" member_type: VOTER last_known_addr { host: "127.20.49.66" port: 44029 } } peers { permanent_uuid: "7a440d3ea874483980ad5a12a47ce1d9" member_type: VOTER last_known_addr { host: "127.20.49.67" port: 38547 } }
I20251209 08:59:02.164088 29418 tablet_replica.cc:442] T 0ea73d83465640babe82c28dfc0c369e P 3cbbcfcfb7634dc588875d9c22e2328e: TxnStatusTablet state changed. Reason: New leader 3cbbcfcfb7634dc588875d9c22e2328e. Latest consensus state: current_term: 1 leader_uuid: "3cbbcfcfb7634dc588875d9c22e2328e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cbbcfcfb7634dc588875d9c22e2328e" member_type: VOTER last_known_addr { host: "127.20.49.65" port: 39793 } } peers { permanent_uuid: "84d473e778bd4553a757338c4cb05fb4" member_type: VOTER last_known_addr { host: "127.20.49.66" port: 44029 } } peers { permanent_uuid: "7a440d3ea874483980ad5a12a47ce1d9" member_type: VOTER last_known_addr { host: "127.20.49.67" port: 38547 } } }
I20251209 08:59:02.164268 29418 tablet_replica.cc:445] T 0ea73d83465640babe82c28dfc0c369e P 3cbbcfcfb7634dc588875d9c22e2328e: This TxnStatusTablet replica's current role is: LEADER
I20251209 08:59:02.164615 29420 txn_status_manager.cc:874] Waiting until node catch up with all replicated operations in previous term...
I20251209 08:59:02.164609 28857 catalog_manager.cc:5654] T 0ea73d83465640babe82c28dfc0c369e P 3cbbcfcfb7634dc588875d9c22e2328e reported cstate change: term changed from 0 to 1, leader changed from <none> to 3cbbcfcfb7634dc588875d9c22e2328e (127.20.49.65). New cstate: current_term: 1 leader_uuid: "3cbbcfcfb7634dc588875d9c22e2328e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cbbcfcfb7634dc588875d9c22e2328e" member_type: VOTER last_known_addr { host: "127.20.49.65" port: 39793 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "84d473e778bd4553a757338c4cb05fb4" member_type: VOTER last_known_addr { host: "127.20.49.66" port: 44029 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a440d3ea874483980ad5a12a47ce1d9" member_type: VOTER last_known_addr { host: "127.20.49.67" port: 38547 } health_report { overall_health: UNKNOWN } } }
I20251209 08:59:02.164749 29420 txn_status_manager.cc:930] Loading transaction status metadata into memory...
I20251209 08:59:02.180819 29412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:59:02.177510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.49.65:39793 (local address 127.0.0.1:60564)
1209 08:59:02.177821 (+ 311us) negotiation.cc:107] Waiting for socket to connect
1209 08:59:02.177839 (+ 18us) client_negotiation.cc:174] Beginning negotiation
1209 08:59:02.177963 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1209 08:59:02.178242 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1209 08:59:02.178245 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1209 08:59:02.178254 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
1209 08:59:02.178448 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:59:02.178454 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:59:02.179147 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1209 08:59:02.179151 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1209 08:59:02.179594 (+ 443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:59:02.179602 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:59:02.179700 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:59:02.179766 (+ 66us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
1209 08:59:02.180507 (+ 741us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
1209 08:59:02.180525 (+ 18us) client_negotiation.cc:769] Sending connection context
1209 08:59:02.180604 (+ 79us) client_negotiation.cc:240] Negotiation successful
1209 08:59:02.180682 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":211}
I20251209 08:59:02.180822 29376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:59:02.177598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60564 (local address 127.20.49.65:39793)
1209 08:59:02.177795 (+ 197us) server_negotiation.cc:207] Beginning negotiation
1209 08:59:02.177820 (+ 25us) server_negotiation.cc:400] Waiting for connection header
1209 08:59:02.177891 (+ 71us) server_negotiation.cc:408] Connection header received
1209 08:59:02.178082 (+ 191us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1209 08:59:02.178085 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1209 08:59:02.178139 (+ 54us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1209 08:59:02.178212 (+ 73us) server_negotiation.cc:227] Negotiated authn=TOKEN
1209 08:59:02.178563 (+ 351us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:59:02.179042 (+ 479us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1209 08:59:02.179707 (+ 665us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:59:02.179928 (+ 221us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:59:02.179979 (+ 51us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1209 08:59:02.180409 (+ 430us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1209 08:59:02.180515 (+ 106us) server_negotiation.cc:1036] Waiting for connection context
1209 08:59:02.180625 (+ 110us) server_negotiation.cc:300] Negotiation successful
1209 08:59:02.180675 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20251209 08:59:02.183823 29161 raft_consensus.cc:1275] T 0ea73d83465640babe82c28dfc0c369e P 84d473e778bd4553a757338c4cb05fb4 [term 1 FOLLOWER]: Refusing update from remote peer 3cbbcfcfb7634dc588875d9c22e2328e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251209 08:59:02.184082 29302 raft_consensus.cc:1275] T 0ea73d83465640babe82c28dfc0c369e P 7a440d3ea874483980ad5a12a47ce1d9 [term 1 FOLLOWER]: Refusing update from remote peer 3cbbcfcfb7634dc588875d9c22e2328e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251209 08:59:02.184343 29384 consensus_queue.cc:1048] T 0ea73d83465640babe82c28dfc0c369e P 3cbbcfcfb7634dc588875d9c22e2328e [LEADER]: Connected to new peer: Peer: permanent_uuid: "84d473e778bd4553a757338c4cb05fb4" member_type: VOTER last_known_addr { host: "127.20.49.66" port: 44029 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251209 08:59:02.184676 29418 consensus_queue.cc:1048] T 0ea73d83465640babe82c28dfc0c369e P 3cbbcfcfb7634dc588875d9c22e2328e [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a440d3ea874483980ad5a12a47ce1d9" member_type: VOTER last_known_addr { host: "127.20.49.67" port: 38547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251209 08:59:02.186450 29396 tablet_replica.cc:442] T 0ea73d83465640babe82c28dfc0c369e P 7a440d3ea874483980ad5a12a47ce1d9: TxnStatusTablet state changed. Reason: New leader 3cbbcfcfb7634dc588875d9c22e2328e. Latest consensus state: current_term: 1 leader_uuid: "3cbbcfcfb7634dc588875d9c22e2328e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cbbcfcfb7634dc588875d9c22e2328e" member_type: VOTER last_known_addr { host: "127.20.49.65" port: 39793 } } peers { permanent_uuid: "84d473e778bd4553a757338c4cb05fb4" member_type: VOTER last_known_addr { host: "127.20.49.66" port: 44029 } } peers { permanent_uuid: "7a440d3ea874483980ad5a12a47ce1d9" member_type: VOTER last_known_addr { host: "127.20.49.67" port: 38547 } } }
I20251209 08:59:02.186579 29396 tablet_replica.cc:445] T 0ea73d83465640babe82c28dfc0c369e P 7a440d3ea874483980ad5a12a47ce1d9: This TxnStatusTablet replica's current role is: FOLLOWER
I20251209 08:59:02.187117 29383 tablet_replica.cc:442] T 0ea73d83465640babe82c28dfc0c369e P 84d473e778bd4553a757338c4cb05fb4: TxnStatusTablet state changed. Reason: New leader 3cbbcfcfb7634dc588875d9c22e2328e. Latest consensus state: current_term: 1 leader_uuid: "3cbbcfcfb7634dc588875d9c22e2328e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cbbcfcfb7634dc588875d9c22e2328e" member_type: VOTER last_known_addr { host: "127.20.49.65" port: 39793 } } peers { permanent_uuid: "84d473e778bd4553a757338c4cb05fb4" member_type: VOTER last_known_addr { host: "127.20.49.66" port: 44029 } } peers { permanent_uuid: "7a440d3ea874483980ad5a12a47ce1d9" member_type: VOTER last_known_addr { host: "127.20.49.67" port: 38547 } } }
I20251209 08:59:02.187233 29383 tablet_replica.cc:445] T 0ea73d83465640babe82c28dfc0c369e P 84d473e778bd4553a757338c4cb05fb4: This TxnStatusTablet replica's current role is: FOLLOWER
I20251209 08:59:02.188807 29418 tablet_replica.cc:442] T 0ea73d83465640babe82c28dfc0c369e P 3cbbcfcfb7634dc588875d9c22e2328e: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3cbbcfcfb7634dc588875d9c22e2328e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cbbcfcfb7634dc588875d9c22e2328e" member_type: VOTER last_known_addr { host: "127.20.49.65" port: 39793 } } peers { permanent_uuid: "84d473e778bd4553a757338c4cb05fb4" member_type: VOTER last_known_addr { host: "127.20.49.66" port: 44029 } } peers { permanent_uuid: "7a440d3ea874483980ad5a12a47ce1d9" member_type: VOTER last_known_addr { host: "127.20.49.67" port: 38547 } } }
I20251209 08:59:02.188902 29418 tablet_replica.cc:445] T 0ea73d83465640babe82c28dfc0c369e P 3cbbcfcfb7634dc588875d9c22e2328e: This TxnStatusTablet replica's current role is: LEADER
I20251209 08:59:02.191366 29391 mvcc.cc:204] Tried to move back new op lower bound from 7230548959979331584 to 7230548959898398720. Current Snapshot: MvccSnapshot[applied={T|T < 7230548959979331584}]
I20251209 08:59:02.191566 29383 tablet_replica.cc:442] T 0ea73d83465640babe82c28dfc0c369e P 84d473e778bd4553a757338c4cb05fb4: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3cbbcfcfb7634dc588875d9c22e2328e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cbbcfcfb7634dc588875d9c22e2328e" member_type: VOTER last_known_addr { host: "127.20.49.65" port: 39793 } } peers { permanent_uuid: "84d473e778bd4553a757338c4cb05fb4" member_type: VOTER last_known_addr { host: "127.20.49.66" port: 44029 } } peers { permanent_uuid: "7a440d3ea874483980ad5a12a47ce1d9" member_type: VOTER last_known_addr { host: "127.20.49.67" port: 38547 } } }
I20251209 08:59:02.191654 29383 tablet_replica.cc:445] T 0ea73d83465640babe82c28dfc0c369e P 84d473e778bd4553a757338c4cb05fb4: This TxnStatusTablet replica's current role is: FOLLOWER
I20251209 08:59:02.189594 29392 mvcc.cc:204] Tried to move back new op lower bound from 7230548959979331584 to 7230548959898398720. Current Snapshot: MvccSnapshot[applied={T|T < 7230548959979331584 or (T in {7230548959979331584})}]
I20251209 08:59:02.197309 29384 tablet_replica.cc:442] T 0ea73d83465640babe82c28dfc0c369e P 3cbbcfcfb7634dc588875d9c22e2328e: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3cbbcfcfb7634dc588875d9c22e2328e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cbbcfcfb7634dc588875d9c22e2328e" member_type: VOTER last_known_addr { host: "127.20.49.65" port: 39793 } } peers { permanent_uuid: "84d473e778bd4553a757338c4cb05fb4" member_type: VOTER last_known_addr { host: "127.20.49.66" port: 44029 } } peers { permanent_uuid: "7a440d3ea874483980ad5a12a47ce1d9" member_type: VOTER last_known_addr { host: "127.20.49.67" port: 38547 } } }
I20251209 08:59:02.197427 29384 tablet_replica.cc:445] T 0ea73d83465640babe82c28dfc0c369e P 3cbbcfcfb7634dc588875d9c22e2328e: This TxnStatusTablet replica's current role is: LEADER
I20251209 08:59:02.198470 29396 tablet_replica.cc:442] T 0ea73d83465640babe82c28dfc0c369e P 7a440d3ea874483980ad5a12a47ce1d9: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3cbbcfcfb7634dc588875d9c22e2328e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cbbcfcfb7634dc588875d9c22e2328e" member_type: VOTER last_known_addr { host: "127.20.49.65" port: 39793 } } peers { permanent_uuid: "84d473e778bd4553a757338c4cb05fb4" member_type: VOTER last_known_addr { host: "127.20.49.66" port: 44029 } } peers { permanent_uuid: "7a440d3ea874483980ad5a12a47ce1d9" member_type: VOTER last_known_addr { host: "127.20.49.67" port: 38547 } } }
I20251209 08:59:02.198593 29396 tablet_replica.cc:445] T 0ea73d83465640babe82c28dfc0c369e P 7a440d3ea874483980ad5a12a47ce1d9: This TxnStatusTablet replica's current role is: FOLLOWER
I20251209 08:59:02.207965 29355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:59:02.205285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.49.65:39793 (local address 127.20.49.67:39717)
1209 08:59:02.205432 (+ 147us) negotiation.cc:107] Waiting for socket to connect
1209 08:59:02.205448 (+ 16us) client_negotiation.cc:174] Beginning negotiation
1209 08:59:02.205566 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1209 08:59:02.205825 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1209 08:59:02.205829 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
1209 08:59:02.205838 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
1209 08:59:02.206064 (+ 226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:59:02.206071 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:59:02.206751 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1209 08:59:02.206754 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1209 08:59:02.207166 (+ 412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:59:02.207172 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:59:02.207277 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:59:02.207326 (+ 49us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
1209 08:59:02.207687 (+ 361us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
1209 08:59:02.207694 (+ 7us) client_negotiation.cc:769] Sending connection context
1209 08:59:02.207757 (+ 63us) client_negotiation.cc:240] Negotiation successful
1209 08:59:02.207830 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20251209 08:59:02.207965 29376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:59:02.205331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.49.67:39717 (local address 127.20.49.65:39793)
1209 08:59:02.205437 (+ 106us) server_negotiation.cc:207] Beginning negotiation
1209 08:59:02.205441 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1209 08:59:02.205511 (+ 70us) server_negotiation.cc:408] Connection header received
1209 08:59:02.205653 (+ 142us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1209 08:59:02.205656 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1209 08:59:02.205734 (+ 78us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1209 08:59:02.205827 (+ 93us) server_negotiation.cc:227] Negotiated authn=TOKEN
1209 08:59:02.206172 (+ 345us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:59:02.206631 (+ 459us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1209 08:59:02.207278 (+ 647us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:59:02.207490 (+ 212us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:59:02.207534 (+ 44us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1209 08:59:02.207603 (+ 69us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1209 08:59:02.207669 (+ 66us) server_negotiation.cc:1036] Waiting for connection context
1209 08:59:02.207771 (+ 102us) server_negotiation.cc:300] Negotiation successful
1209 08:59:02.207827 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20251209 08:59:02.215900 29019 raft_consensus.cc:1275] T c7c0d2789b5c4aceb7a1f436559d634b P 3cbbcfcfb7634dc588875d9c22e2328e [term 1 FOLLOWER]: Refusing update from remote peer 7a440d3ea874483980ad5a12a47ce1d9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251209 08:59:02.215900 29161 raft_consensus.cc:1275] T c7c0d2789b5c4aceb7a1f436559d634b P 84d473e778bd4553a757338c4cb05fb4 [term 1 FOLLOWER]: Refusing update from remote peer 7a440d3ea874483980ad5a12a47ce1d9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251209 08:59:02.216434 29385 consensus_queue.cc:1048] T c7c0d2789b5c4aceb7a1f436559d634b P 7a440d3ea874483980ad5a12a47ce1d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3cbbcfcfb7634dc588875d9c22e2328e" member_type: VOTER last_known_addr { host: "127.20.49.65" port: 39793 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251209 08:59:02.216766 29385 consensus_queue.cc:1048] T c7c0d2789b5c4aceb7a1f436559d634b P 7a440d3ea874483980ad5a12a47ce1d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84d473e778bd4553a757338c4cb05fb4" member_type: VOTER last_known_addr { host: "127.20.49.66" port: 44029 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251209 08:59:02.223587 29390 mvcc.cc:204] Tried to move back new op lower bound from 7230548960112680960 to 7230548959520784384. Current Snapshot: MvccSnapshot[applied={T|T < 7230548960112680960}]
I20251209 08:59:02.246572 29372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:59:02.242921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.49.65:36663 (local address 127.20.49.67:38547)
1209 08:59:02.243040 (+ 119us) server_negotiation.cc:207] Beginning negotiation
1209 08:59:02.243047 (+ 7us) server_negotiation.cc:400] Waiting for connection header
1209 08:59:02.243468 (+ 421us) server_negotiation.cc:408] Connection header received
1209 08:59:02.243681 (+ 213us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1209 08:59:02.243685 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1209 08:59:02.243743 (+ 58us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1209 08:59:02.243795 (+ 52us) server_negotiation.cc:227] Negotiated authn=TOKEN
1209 08:59:02.244514 (+ 719us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:59:02.244979 (+ 465us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1209 08:59:02.245828 (+ 849us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:59:02.246048 (+ 220us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:59:02.246094 (+ 46us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1209 08:59:02.246161 (+ 67us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1209 08:59:02.246224 (+ 63us) server_negotiation.cc:1036] Waiting for connection context
1209 08:59:02.246354 (+ 130us) server_negotiation.cc:300] Negotiation successful
1209 08:59:02.246417 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20251209 08:59:02.247157 29432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:59:02.242688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.49.67:38547 (local address 127.20.49.65:36663)
1209 08:59:02.243385 (+ 697us) negotiation.cc:107] Waiting for socket to connect
1209 08:59:02.243400 (+ 15us) client_negotiation.cc:174] Beginning negotiation
1209 08:59:02.243555 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1209 08:59:02.243949 (+ 394us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1209 08:59:02.243953 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
1209 08:59:02.243966 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
1209 08:59:02.244404 (+ 438us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:59:02.244413 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:59:02.245126 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1209 08:59:02.245130 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1209 08:59:02.245644 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:59:02.245653 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:59:02.245855 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:59:02.245904 (+ 49us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
1209 08:59:02.246242 (+ 338us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
1209 08:59:02.246248 (+ 6us) client_negotiation.cc:769] Sending connection context
1209 08:59:02.246921 (+ 673us) client_negotiation.cc:240] Negotiation successful
1209 08:59:02.247006 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":562,"thread_start_us":94,"threads_started":1}
I20251209 08:59:02.275313 28860 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59162:
table { table_name: "test-table" } modify_external_catalogs: true
I20251209 08:59:02.275481 28860 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59162:
table { table_name: "test-table" } modify_external_catalogs: true
I20251209 08:59:02.277572 28860 catalog_manager.cc:5941] T 00000000000000000000000000000000 P 7f2aac7cb38e4c609b71560ad54a20b1: Sending DeleteTablet for 3 replicas of tablet c7c0d2789b5c4aceb7a1f436559d634b
I20251209 08:59:02.278378 29283 tablet_service.cc:1552] Processing DeleteTablet for tablet c7c0d2789b5c4aceb7a1f436559d634b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-09 08:59:02 UTC) from {username='oryx', principal='oryx/127.20.49.126@KRBTEST.COM'} at 127.0.0.1:35670
I20251209 08:59:02.278549 29141 tablet_service.cc:1552] Processing DeleteTablet for tablet c7c0d2789b5c4aceb7a1f436559d634b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-09 08:59:02 UTC) from {username='oryx', principal='oryx/127.20.49.126@KRBTEST.COM'} at 127.0.0.1:52844
I20251209 08:59:02.278740 29400 tablet_replica.cc:333] T c7c0d2789b5c4aceb7a1f436559d634b P 7a440d3ea874483980ad5a12a47ce1d9: stopping tablet replica
I20251209 08:59:02.278867 29400 raft_consensus.cc:2243] T c7c0d2789b5c4aceb7a1f436559d634b P 7a440d3ea874483980ad5a12a47ce1d9 [term 1 LEADER]: Raft consensus shutting down.
I20251209 08:59:02.279091 29400 raft_consensus.cc:2272] T c7c0d2789b5c4aceb7a1f436559d634b P 7a440d3ea874483980ad5a12a47ce1d9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251209 08:59:02.279295 29401 tablet_replica.cc:333] T c7c0d2789b5c4aceb7a1f436559d634b P 84d473e778bd4553a757338c4cb05fb4: stopping tablet replica
I20251209 08:59:02.279412 29401 raft_consensus.cc:2243] T c7c0d2789b5c4aceb7a1f436559d634b P 84d473e778bd4553a757338c4cb05fb4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251209 08:59:02.279546 29401 raft_consensus.cc:2272] T c7c0d2789b5c4aceb7a1f436559d634b P 84d473e778bd4553a757338c4cb05fb4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251209 08:59:02.280344 29401 ts_tablet_manager.cc:1916] T c7c0d2789b5c4aceb7a1f436559d634b P 84d473e778bd4553a757338c4cb05fb4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251209 08:59:02.280957 29400 ts_tablet_manager.cc:1916] T c7c0d2789b5c4aceb7a1f436559d634b P 7a440d3ea874483980ad5a12a47ce1d9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251209 08:59:02.281876 28999 tablet_service.cc:1552] Processing DeleteTablet for tablet c7c0d2789b5c4aceb7a1f436559d634b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-09 08:59:02 UTC) from {username='oryx', principal='oryx/127.20.49.126@KRBTEST.COM'} at 127.0.0.1:60562
I20251209 08:59:02.282071 29402 tablet_replica.cc:333] T c7c0d2789b5c4aceb7a1f436559d634b P 3cbbcfcfb7634dc588875d9c22e2328e: stopping tablet replica
I20251209 08:59:02.282189 29402 raft_consensus.cc:2243] T c7c0d2789b5c4aceb7a1f436559d634b P 3cbbcfcfb7634dc588875d9c22e2328e [term 1 FOLLOWER]: Raft consensus shutting down.
I20251209 08:59:02.282287 29402 raft_consensus.cc:2272] T c7c0d2789b5c4aceb7a1f436559d634b P 3cbbcfcfb7634dc588875d9c22e2328e [term 1 FOLLOWER]: Raft consensus is shut down!
I20251209 08:59:02.283011 29402 ts_tablet_manager.cc:1916] T c7c0d2789b5c4aceb7a1f436559d634b P 3cbbcfcfb7634dc588875d9c22e2328e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251209 08:59:02.283792 29400 ts_tablet_manager.cc:1929] T c7c0d2789b5c4aceb7a1f436559d634b P 7a440d3ea874483980ad5a12a47ce1d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20251209 08:59:02.283857 29400 log.cc:1199] T c7c0d2789b5c4aceb7a1f436559d634b P 7a440d3ea874483980ad5a12a47ce1d9: Deleting WAL directory at /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765270668420570-20677-0/minicluster-data/ts-2/wal/wals/c7c0d2789b5c4aceb7a1f436559d634b
I20251209 08:59:02.284106 29400 ts_tablet_manager.cc:1950] T c7c0d2789b5c4aceb7a1f436559d634b P 7a440d3ea874483980ad5a12a47ce1d9: Deleting consensus metadata
I20251209 08:59:02.284650 29401 ts_tablet_manager.cc:1929] T c7c0d2789b5c4aceb7a1f436559d634b P 84d473e778bd4553a757338c4cb05fb4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20251209 08:59:02.284721 29401 log.cc:1199] T c7c0d2789b5c4aceb7a1f436559d634b P 84d473e778bd4553a757338c4cb05fb4: Deleting WAL directory at /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765270668420570-20677-0/minicluster-data/ts-1/wal/wals/c7c0d2789b5c4aceb7a1f436559d634b
I20251209 08:59:02.284989 29401 ts_tablet_manager.cc:1950] T c7c0d2789b5c4aceb7a1f436559d634b P 84d473e778bd4553a757338c4cb05fb4: Deleting consensus metadata
I20251209 08:59:02.285183 28846 catalog_manager.cc:4985] TS 7a440d3ea874483980ad5a12a47ce1d9 (127.20.49.67:38547): tablet c7c0d2789b5c4aceb7a1f436559d634b (table test-table [id=a8a0991b38364a7bbd9fdbfb0b9fe13a]) successfully deleted
I20251209 08:59:02.285526 29402 ts_tablet_manager.cc:1929] T c7c0d2789b5c4aceb7a1f436559d634b P 3cbbcfcfb7634dc588875d9c22e2328e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20251209 08:59:02.285601 29402 log.cc:1199] T c7c0d2789b5c4aceb7a1f436559d634b P 3cbbcfcfb7634dc588875d9c22e2328e: Deleting WAL directory at /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765270668420570-20677-0/minicluster-data/ts-0/wal/wals/c7c0d2789b5c4aceb7a1f436559d634b
I20251209 08:59:02.285879 29402 ts_tablet_manager.cc:1950] T c7c0d2789b5c4aceb7a1f436559d634b P 3cbbcfcfb7634dc588875d9c22e2328e: Deleting consensus metadata
I20251209 08:59:02.286703 28847 catalog_manager.cc:4985] TS 3cbbcfcfb7634dc588875d9c22e2328e (127.20.49.65:39793): tablet c7c0d2789b5c4aceb7a1f436559d634b (table test-table [id=a8a0991b38364a7bbd9fdbfb0b9fe13a]) successfully deleted
I20251209 08:59:02.289727 28844 catalog_manager.cc:4985] TS 84d473e778bd4553a757338c4cb05fb4 (127.20.49.66:44029): tablet c7c0d2789b5c4aceb7a1f436559d634b (table test-table [id=a8a0991b38364a7bbd9fdbfb0b9fe13a]) successfully deleted
Dec 09 08:59:02 dist-test-slave-8w6c krb5kdc[28812](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765270742, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Dec 09 08:59:02 dist-test-slave-8w6c krb5kdc[28812](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.20.49.126@KRBTEST.COM, Server not found in Kerberos database
Dec 09 08:59:02 dist-test-slave-8w6c krb5kdc[28812](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.20.49.126@KRBTEST.COM, Server not found in Kerberos database
W20251209 08:59:02.309796 28909 negotiation.cc:337] Failed RPC negotiation. Trace:
1209 08:59:02.303975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59178 (local address 127.20.49.126:41745)
1209 08:59:02.304128 (+ 153us) server_negotiation.cc:207] Beginning negotiation
1209 08:59:02.304133 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1209 08:59:02.304152 (+ 19us) server_negotiation.cc:408] Connection header received
1209 08:59:02.304276 (+ 124us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1209 08:59:02.304280 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1209 08:59:02.304341 (+ 61us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1209 08:59:02.304442 (+ 101us) server_negotiation.cc:227] Negotiated authn=SASL
1209 08:59:02.305226 (+ 784us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:59:02.306073 (+ 847us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1209 08:59:02.306711 (+ 638us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:59:02.306918 (+ 207us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:59:02.309603 (+ 2685us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59178: BlockingRecv error: recv got EOF from 127.0.0.1:59178 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Dec 09 08:59:02 dist-test-slave-8w6c krb5kdc[28812](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765270742, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.20.49.126@KRBTEST.COM
I20251209 08:59:02.319173 28909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:59:02.311200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59180 (local address 127.20.49.126:41745)
1209 08:59:02.311336 (+ 136us) server_negotiation.cc:207] Beginning negotiation
1209 08:59:02.311342 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1209 08:59:02.311570 (+ 228us) server_negotiation.cc:408] Connection header received
1209 08:59:02.311670 (+ 100us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1209 08:59:02.311674 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1209 08:59:02.311731 (+ 57us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1209 08:59:02.311824 (+ 93us) server_negotiation.cc:227] Negotiated authn=SASL
1209 08:59:02.312439 (+ 615us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:59:02.313209 (+ 770us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1209 08:59:02.313828 (+ 619us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:59:02.314022 (+ 194us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:59:02.316019 (+ 1997us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1209 08:59:02.316039 (+ 20us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1209 08:59:02.316042 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1209 08:59:02.316072 (+ 30us) server_negotiation.cc:923] calling sasl_server_start()
1209 08:59:02.317656 (+ 1584us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1209 08:59:02.318168 (+ 512us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1209 08:59:02.318171 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1209 08:59:02.318174 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
1209 08:59:02.318228 (+ 54us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1209 08:59:02.318409 (+ 181us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1209 08:59:02.318413 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1209 08:59:02.318415 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1209 08:59:02.318616 (+ 201us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1209 08:59:02.318749 (+ 133us) server_negotiation.cc:1036] Waiting for connection context
1209 08:59:02.318890 (+ 141us) server_negotiation.cc:300] Negotiation successful
1209 08:59:02.319005 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20251209 08:59:02.321624 28860 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59180:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251209 08:59:02.322170 28860 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251209 08:59:02.327165 28999 tablet_service.cc:1505] Processing CreateTablet for tablet 686a629c7f3a458c8ee0a4276d7a5f27 (DEFAULT_TABLE table=test-table [id=b0e95c299a954d578b23c830ca337f71]), partition=RANGE (key) PARTITION UNBOUNDED
I20251209 08:59:02.327165 29141 tablet_service.cc:1505] Processing CreateTablet for tablet 686a629c7f3a458c8ee0a4276d7a5f27 (DEFAULT_TABLE table=test-table [id=b0e95c299a954d578b23c830ca337f71]), partition=RANGE (key) PARTITION UNBOUNDED
I20251209 08:59:02.327471 29141 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 686a629c7f3a458c8ee0a4276d7a5f27. 1 dirs total, 0 dirs full, 0 dirs failed
I20251209 08:59:02.327471 28999 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 686a629c7f3a458c8ee0a4276d7a5f27. 1 dirs total, 0 dirs full, 0 dirs failed
I20251209 08:59:02.329779 29378 tablet_bootstrap.cc:492] T 686a629c7f3a458c8ee0a4276d7a5f27 P 3cbbcfcfb7634dc588875d9c22e2328e: Bootstrap starting.
I20251209 08:59:02.329825 29379 tablet_bootstrap.cc:492] T 686a629c7f3a458c8ee0a4276d7a5f27 P 84d473e778bd4553a757338c4cb05fb4: Bootstrap starting.
I20251209 08:59:02.330806 29378 tablet_bootstrap.cc:654] T 686a629c7f3a458c8ee0a4276d7a5f27 P 3cbbcfcfb7634dc588875d9c22e2328e: Neither blocks nor log segments found. Creating new log.
I20251209 08:59:02.330868 29379 tablet_bootstrap.cc:654] T 686a629c7f3a458c8ee0a4276d7a5f27 P 84d473e778bd4553a757338c4cb05fb4: Neither blocks nor log segments found. Creating new log.
I20251209 08:59:02.331768 29283 tablet_service.cc:1505] Processing CreateTablet for tablet 686a629c7f3a458c8ee0a4276d7a5f27 (DEFAULT_TABLE table=test-table [id=b0e95c299a954d578b23c830ca337f71]), partition=RANGE (key) PARTITION UNBOUNDED
I20251209 08:59:02.332064 29283 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 686a629c7f3a458c8ee0a4276d7a5f27. 1 dirs total, 0 dirs full, 0 dirs failed
I20251209 08:59:02.332616 29378 tablet_bootstrap.cc:492] T 686a629c7f3a458c8ee0a4276d7a5f27 P 3cbbcfcfb7634dc588875d9c22e2328e: No bootstrap required, opened a new log
I20251209 08:59:02.332712 29378 ts_tablet_manager.cc:1403] T 686a629c7f3a458c8ee0a4276d7a5f27 P 3cbbcfcfb7634dc588875d9c22e2328e: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20251209 08:59:02.333235 29378 raft_consensus.cc:359] T 686a629c7f3a458c8ee0a4276d7a5f27 P 3cbbcfcfb7634dc588875d9c22e2328e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cbbcfcfb7634dc588875d9c22e2328e" member_type: VOTER last_known_addr { host: "127.20.49.65" port: 39793 } } peers { permanent_uuid: "84d473e778bd4553a757338c4cb05fb4" member_type: VOTER last_known_addr { host: "127.20.49.66" port: 44029 } } peers { permanent_uuid: "7a440d3ea874483980ad5a12a47ce1d9" member_type: VOTER last_known_addr { host: "127.20.49.67" port: 38547 } }
I20251209 08:59:02.333359 29378 raft_consensus.cc:385] T 686a629c7f3a458c8ee0a4276d7a5f27 P 3cbbcfcfb7634dc588875d9c22e2328e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251209 08:59:02.333425 29378 raft_consensus.cc:740] T 686a629c7f3a458c8ee0a4276d7a5f27 P 3cbbcfcfb7634dc588875d9c22e2328e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3cbbcfcfb7634dc588875d9c22e2328e, State: Initialized, Role: FOLLOWER
I20251209 08:59:02.333572 29378 consensus_queue.cc:260] T 686a629c7f3a458c8ee0a4276d7a5f27 P 3cbbcfcfb7634dc588875d9c22e2328e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cbbcfcfb7634dc588875d9c22e2328e" member_type: VOTER last_known_addr { host: "127.20.49.65" port: 39793 } } peers { permanent_uuid: "84d473e778bd4553a757338c4cb05fb4" member_type: VOTER last_known_addr { host: "127.20.49.66" port: 44029 } } peers { permanent_uuid: "7a440d3ea874483980ad5a12a47ce1d9" member_type: VOTER last_known_addr { host: "127.20.49.67" port: 38547 } }
I20251209 08:59:02.333913 29378 ts_tablet_manager.cc:1434] T 686a629c7f3a458c8ee0a4276d7a5f27 P 3cbbcfcfb7634dc588875d9c22e2328e: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20251209 08:59:02.334129 29379 tablet_bootstrap.cc:492] T 686a629c7f3a458c8ee0a4276d7a5f27 P 84d473e778bd4553a757338c4cb05fb4: No bootstrap required, opened a new log
I20251209 08:59:02.334226 29379 ts_tablet_manager.cc:1403] T 686a629c7f3a458c8ee0a4276d7a5f27 P 84d473e778bd4553a757338c4cb05fb4: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.001s
I20251209 08:59:02.334689 29379 raft_consensus.cc:359] T 686a629c7f3a458c8ee0a4276d7a5f27 P 84d473e778bd4553a757338c4cb05fb4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cbbcfcfb7634dc588875d9c22e2328e" member_type: VOTER last_known_addr { host: "127.20.49.65" port: 39793 } } peers { permanent_uuid: "84d473e778bd4553a757338c4cb05fb4" member_type: VOTER last_known_addr { host: "127.20.49.66" port: 44029 } } peers { permanent_uuid: "7a440d3ea874483980ad5a12a47ce1d9" member_type: VOTER last_known_addr { host: "127.20.49.67" port: 38547 } }
I20251209 08:59:02.334812 29379 raft_consensus.cc:385] T 686a629c7f3a458c8ee0a4276d7a5f27 P 84d473e778bd4553a757338c4cb05fb4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251209 08:59:02.334852 29379 raft_consensus.cc:740] T 686a629c7f3a458c8ee0a4276d7a5f27 P 84d473e778bd4553a757338c4cb05fb4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84d473e778bd4553a757338c4cb05fb4, State: Initialized, Role: FOLLOWER
I20251209 08:59:02.334970 29379 consensus_queue.cc:260] T 686a629c7f3a458c8ee0a4276d7a5f27 P 84d473e778bd4553a757338c4cb05fb4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cbbcfcfb7634dc588875d9c22e2328e" member_type: VOTER last_known_addr { host: "127.20.49.65" port: 39793 } } peers { permanent_uuid: "84d473e778bd4553a757338c4cb05fb4" member_type: VOTER last_known_addr { host: "127.20.49.66" port: 44029 } } peers { permanent_uuid: "7a440d3ea874483980ad5a12a47ce1d9" member_type: VOTER last_known_addr { host: "127.20.49.67" port: 38547 } }
I20251209 08:59:02.335218 29379 ts_tablet_manager.cc:1434] T 686a629c7f3a458c8ee0a4276d7a5f27 P 84d473e778bd4553a757338c4cb05fb4: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20251209 08:59:02.336738 29377 tablet_bootstrap.cc:492] T 686a629c7f3a458c8ee0a4276d7a5f27 P 7a440d3ea874483980ad5a12a47ce1d9: Bootstrap starting.
I20251209 08:59:02.337744 29377 tablet_bootstrap.cc:654] T 686a629c7f3a458c8ee0a4276d7a5f27 P 7a440d3ea874483980ad5a12a47ce1d9: Neither blocks nor log segments found. Creating new log.
I20251209 08:59:02.338876 29377 tablet_bootstrap.cc:492] T 686a629c7f3a458c8ee0a4276d7a5f27 P 7a440d3ea874483980ad5a12a47ce1d9: No bootstrap required, opened a new log
I20251209 08:59:02.338950 29377 ts_tablet_manager.cc:1403] T 686a629c7f3a458c8ee0a4276d7a5f27 P 7a440d3ea874483980ad5a12a47ce1d9: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20251209 08:59:02.339262 29377 raft_consensus.cc:359] T 686a629c7f3a458c8ee0a4276d7a5f27 P 7a440d3ea874483980ad5a12a47ce1d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cbbcfcfb7634dc588875d9c22e2328e" member_type: VOTER last_known_addr { host: "127.20.49.65" port: 39793 } } peers { permanent_uuid: "84d473e778bd4553a757338c4cb05fb4" member_type: VOTER last_known_addr { host: "127.20.49.66" port: 44029 } } peers { permanent_uuid: "7a440d3ea874483980ad5a12a47ce1d9" member_type: VOTER last_known_addr { host: "127.20.49.67" port: 38547 } }
I20251209 08:59:02.339350 29377 raft_consensus.cc:385] T 686a629c7f3a458c8ee0a4276d7a5f27 P 7a440d3ea874483980ad5a12a47ce1d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251209 08:59:02.339371 29377 raft_consensus.cc:740] T 686a629c7f3a458c8ee0a4276d7a5f27 P 7a440d3ea874483980ad5a12a47ce1d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a440d3ea874483980ad5a12a47ce1d9, State: Initialized, Role: FOLLOWER
I20251209 08:59:02.339474 29377 consensus_queue.cc:260] T 686a629c7f3a458c8ee0a4276d7a5f27 P 7a440d3ea874483980ad5a12a47ce1d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cbbcfcfb7634dc588875d9c22e2328e" member_type: VOTER last_known_addr { host: "127.20.49.65" port: 39793 } } peers { permanent_uuid: "84d473e778bd4553a757338c4cb05fb4" member_type: VOTER last_known_addr { host: "127.20.49.66" port: 44029 } } peers { permanent_uuid: "7a440d3ea874483980ad5a12a47ce1d9" member_type: VOTER last_known_addr { host: "127.20.49.67" port: 38547 } }
I20251209 08:59:02.339670 29377 ts_tablet_manager.cc:1434] T 686a629c7f3a458c8ee0a4276d7a5f27 P 7a440d3ea874483980ad5a12a47ce1d9: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20251209 08:59:02.389802 29387 raft_consensus.cc:493] T 686a629c7f3a458c8ee0a4276d7a5f27 P 7a440d3ea874483980ad5a12a47ce1d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251209 08:59:02.389950 29387 raft_consensus.cc:515] T 686a629c7f3a458c8ee0a4276d7a5f27 P 7a440d3ea874483980ad5a12a47ce1d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cbbcfcfb7634dc588875d9c22e2328e" member_type: VOTER last_known_addr { host: "127.20.49.65" port: 39793 } } peers { permanent_uuid: "84d473e778bd4553a757338c4cb05fb4" member_type: VOTER last_known_addr { host: "127.20.49.66" port: 44029 } } peers { permanent_uuid: "7a440d3ea874483980ad5a12a47ce1d9" member_type: VOTER last_known_addr { host: "127.20.49.67" port: 38547 } }
I20251209 08:59:02.390400 29387 leader_election.cc:290] T 686a629c7f3a458c8ee0a4276d7a5f27 P 7a440d3ea874483980ad5a12a47ce1d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3cbbcfcfb7634dc588875d9c22e2328e (127.20.49.65:39793), 84d473e778bd4553a757338c4cb05fb4 (127.20.49.66:44029)
I20251209 08:59:02.390902 29161 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "686a629c7f3a458c8ee0a4276d7a5f27" candidate_uuid: "7a440d3ea874483980ad5a12a47ce1d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84d473e778bd4553a757338c4cb05fb4" is_pre_election: true
I20251209 08:59:02.390903 29019 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "686a629c7f3a458c8ee0a4276d7a5f27" candidate_uuid: "7a440d3ea874483980ad5a12a47ce1d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3cbbcfcfb7634dc588875d9c22e2328e" is_pre_election: true
I20251209 08:59:02.391049 29161 raft_consensus.cc:2468] T 686a629c7f3a458c8ee0a4276d7a5f27 P 84d473e778bd4553a757338c4cb05fb4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a440d3ea874483980ad5a12a47ce1d9 in term 0.
I20251209 08:59:02.391058 29019 raft_consensus.cc:2468] T 686a629c7f3a458c8ee0a4276d7a5f27 P 3cbbcfcfb7634dc588875d9c22e2328e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a440d3ea874483980ad5a12a47ce1d9 in term 0.
I20251209 08:59:02.391361 29237 leader_election.cc:304] T 686a629c7f3a458c8ee0a4276d7a5f27 P 7a440d3ea874483980ad5a12a47ce1d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3cbbcfcfb7634dc588875d9c22e2328e, 7a440d3ea874483980ad5a12a47ce1d9; no voters:
I20251209 08:59:02.391527 29387 raft_consensus.cc:2804] T 686a629c7f3a458c8ee0a4276d7a5f27 P 7a440d3ea874483980ad5a12a47ce1d9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251209 08:59:02.391599 29387 raft_consensus.cc:493] T 686a629c7f3a458c8ee0a4276d7a5f27 P 7a440d3ea874483980ad5a12a47ce1d9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251209 08:59:02.391641 29387 raft_consensus.cc:3060] T 686a629c7f3a458c8ee0a4276d7a5f27 P 7a440d3ea874483980ad5a12a47ce1d9 [term 0 FOLLOWER]: Advancing to term 1
I20251209 08:59:02.392861 29387 raft_consensus.cc:515] T 686a629c7f3a458c8ee0a4276d7a5f27 P 7a440d3ea874483980ad5a12a47ce1d9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cbbcfcfb7634dc588875d9c22e2328e" member_type: VOTER last_known_addr { host: "127.20.49.65" port: 39793 } } peers { permanent_uuid: "84d473e778bd4553a757338c4cb05fb4" member_type: VOTER last_known_addr { host: "127.20.49.66" port: 44029 } } peers { permanent_uuid: "7a440d3ea874483980ad5a12a47ce1d9" member_type: VOTER last_known_addr { host: "127.20.49.67" port: 38547 } }
I20251209 08:59:02.393262 29387 leader_election.cc:290] T 686a629c7f3a458c8ee0a4276d7a5f27 P 7a440d3ea874483980ad5a12a47ce1d9 [CANDIDATE]: Term 1 election: Requested vote from peers 3cbbcfcfb7634dc588875d9c22e2328e (127.20.49.65:39793), 84d473e778bd4553a757338c4cb05fb4 (127.20.49.66:44029)
I20251209 08:59:02.393607 29019 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "686a629c7f3a458c8ee0a4276d7a5f27" candidate_uuid: "7a440d3ea874483980ad5a12a47ce1d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3cbbcfcfb7634dc588875d9c22e2328e"
I20251209 08:59:02.393618 29161 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "686a629c7f3a458c8ee0a4276d7a5f27" candidate_uuid: "7a440d3ea874483980ad5a12a47ce1d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84d473e778bd4553a757338c4cb05fb4"
I20251209 08:59:02.393766 29161 raft_consensus.cc:3060] T 686a629c7f3a458c8ee0a4276d7a5f27 P 84d473e778bd4553a757338c4cb05fb4 [term 0 FOLLOWER]: Advancing to term 1
I20251209 08:59:02.393747 29019 raft_consensus.cc:3060] T 686a629c7f3a458c8ee0a4276d7a5f27 P 3cbbcfcfb7634dc588875d9c22e2328e [term 0 FOLLOWER]: Advancing to term 1
I20251209 08:59:02.394836 29161 raft_consensus.cc:2468] T 686a629c7f3a458c8ee0a4276d7a5f27 P 84d473e778bd4553a757338c4cb05fb4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a440d3ea874483980ad5a12a47ce1d9 in term 1.
I20251209 08:59:02.394968 29019 raft_consensus.cc:2468] T 686a629c7f3a458c8ee0a4276d7a5f27 P 3cbbcfcfb7634dc588875d9c22e2328e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a440d3ea874483980ad5a12a47ce1d9 in term 1.
I20251209 08:59:02.395155 29234 leader_election.cc:304] T 686a629c7f3a458c8ee0a4276d7a5f27 P 7a440d3ea874483980ad5a12a47ce1d9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a440d3ea874483980ad5a12a47ce1d9, 84d473e778bd4553a757338c4cb05fb4; no voters:
I20251209 08:59:02.395345 29387 raft_consensus.cc:2804] T 686a629c7f3a458c8ee0a4276d7a5f27 P 7a440d3ea874483980ad5a12a47ce1d9 [term 1 FOLLOWER]: Leader election won for term 1
I20251209 08:59:02.395428 29387 raft_consensus.cc:697] T 686a629c7f3a458c8ee0a4276d7a5f27 P 7a440d3ea874483980ad5a12a47ce1d9 [term 1 LEADER]: Becoming Leader. State: Replica: 7a440d3ea874483980ad5a12a47ce1d9, State: Running, Role: LEADER
I20251209 08:59:02.395576 29387 consensus_queue.cc:237] T 686a629c7f3a458c8ee0a4276d7a5f27 P 7a440d3ea874483980ad5a12a47ce1d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cbbcfcfb7634dc588875d9c22e2328e" member_type: VOTER last_known_addr { host: "127.20.49.65" port: 39793 } } peers { permanent_uuid: "84d473e778bd4553a757338c4cb05fb4" member_type: VOTER last_known_addr { host: "127.20.49.66" port: 44029 } } peers { permanent_uuid: "7a440d3ea874483980ad5a12a47ce1d9" member_type: VOTER last_known_addr { host: "127.20.49.67" port: 38547 } }
I20251209 08:59:02.397256 28857 catalog_manager.cc:5654] T 686a629c7f3a458c8ee0a4276d7a5f27 P 7a440d3ea874483980ad5a12a47ce1d9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7a440d3ea874483980ad5a12a47ce1d9 (127.20.49.67). New cstate: current_term: 1 leader_uuid: "7a440d3ea874483980ad5a12a47ce1d9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cbbcfcfb7634dc588875d9c22e2328e" member_type: VOTER last_known_addr { host: "127.20.49.65" port: 39793 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "84d473e778bd4553a757338c4cb05fb4" member_type: VOTER last_known_addr { host: "127.20.49.66" port: 44029 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a440d3ea874483980ad5a12a47ce1d9" member_type: VOTER last_known_addr { host: "127.20.49.67" port: 38547 } health_report { overall_health: HEALTHY } } }
I20251209 08:59:02.416009 29372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:59:02.413642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35696 (local address 127.20.49.67:38547)
1209 08:59:02.413779 (+ 137us) server_negotiation.cc:207] Beginning negotiation
1209 08:59:02.413782 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1209 08:59:02.413799 (+ 17us) server_negotiation.cc:408] Connection header received
1209 08:59:02.413953 (+ 154us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1209 08:59:02.413956 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1209 08:59:02.414005 (+ 49us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1209 08:59:02.414065 (+ 60us) server_negotiation.cc:227] Negotiated authn=TOKEN
1209 08:59:02.414411 (+ 346us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:59:02.414890 (+ 479us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1209 08:59:02.415469 (+ 579us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:59:02.415607 (+ 138us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:59:02.415646 (+ 39us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1209 08:59:02.415712 (+ 66us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1209 08:59:02.415769 (+ 57us) server_negotiation.cc:1036] Waiting for connection context
1209 08:59:02.415857 (+ 88us) server_negotiation.cc:300] Negotiation successful
1209 08:59:02.415901 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20251209 08:59:02.417990 29019 raft_consensus.cc:1275] T 686a629c7f3a458c8ee0a4276d7a5f27 P 3cbbcfcfb7634dc588875d9c22e2328e [term 1 FOLLOWER]: Refusing update from remote peer 7a440d3ea874483980ad5a12a47ce1d9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251209 08:59:02.418022 29161 raft_consensus.cc:1275] T 686a629c7f3a458c8ee0a4276d7a5f27 P 84d473e778bd4553a757338c4cb05fb4 [term 1 FOLLOWER]: Refusing update from remote peer 7a440d3ea874483980ad5a12a47ce1d9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251209 08:59:02.418485 29387 consensus_queue.cc:1048] T 686a629c7f3a458c8ee0a4276d7a5f27 P 7a440d3ea874483980ad5a12a47ce1d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3cbbcfcfb7634dc588875d9c22e2328e" member_type: VOTER last_known_addr { host: "127.20.49.65" port: 39793 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251209 08:59:02.418658 29396 consensus_queue.cc:1048] T 686a629c7f3a458c8ee0a4276d7a5f27 P 7a440d3ea874483980ad5a12a47ce1d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84d473e778bd4553a757338c4cb05fb4" member_type: VOTER last_known_addr { host: "127.20.49.66" port: 44029 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251209 08:59:02.426025 28857 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59180:
table { table_name: "test-table" } modify_external_catalogs: true
I20251209 08:59:02.426177 28857 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59180:
table { table_name: "test-table" } modify_external_catalogs: true
I20251209 08:59:02.427995 28857 catalog_manager.cc:5941] T 00000000000000000000000000000000 P 7f2aac7cb38e4c609b71560ad54a20b1: Sending DeleteTablet for 3 replicas of tablet 686a629c7f3a458c8ee0a4276d7a5f27
I20251209 08:59:02.428467 28999 tablet_service.cc:1552] Processing DeleteTablet for tablet 686a629c7f3a458c8ee0a4276d7a5f27 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-09 08:59:02 UTC) from {username='oryx', principal='oryx/127.20.49.126@KRBTEST.COM'} at 127.0.0.1:60562
I20251209 08:59:02.428525 29141 tablet_service.cc:1552] Processing DeleteTablet for tablet 686a629c7f3a458c8ee0a4276d7a5f27 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-09 08:59:02 UTC) from {username='oryx', principal='oryx/127.20.49.126@KRBTEST.COM'} at 127.0.0.1:52844
I20251209 08:59:02.428680 29402 tablet_replica.cc:333] T 686a629c7f3a458c8ee0a4276d7a5f27 P 3cbbcfcfb7634dc588875d9c22e2328e: stopping tablet replica
I20251209 08:59:02.428718 29401 tablet_replica.cc:333] T 686a629c7f3a458c8ee0a4276d7a5f27 P 84d473e778bd4553a757338c4cb05fb4: stopping tablet replica
I20251209 08:59:02.428800 29402 raft_consensus.cc:2243] T 686a629c7f3a458c8ee0a4276d7a5f27 P 3cbbcfcfb7634dc588875d9c22e2328e [term 1 FOLLOWER]: Raft consensus shutting down.
I20251209 08:59:02.428830 29401 raft_consensus.cc:2243] T 686a629c7f3a458c8ee0a4276d7a5f27 P 84d473e778bd4553a757338c4cb05fb4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251209 08:59:02.428921 29402 raft_consensus.cc:2272] T 686a629c7f3a458c8ee0a4276d7a5f27 P 3cbbcfcfb7634dc588875d9c22e2328e [term 1 FOLLOWER]: Raft consensus is shut down!
I20251209 08:59:02.428907 29283 tablet_service.cc:1552] Processing DeleteTablet for tablet 686a629c7f3a458c8ee0a4276d7a5f27 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-09 08:59:02 UTC) from {username='oryx', principal='oryx/127.20.49.126@KRBTEST.COM'} at 127.0.0.1:35670
I20251209 08:59:02.428957 29401 raft_consensus.cc:2272] T 686a629c7f3a458c8ee0a4276d7a5f27 P 84d473e778bd4553a757338c4cb05fb4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251209 08:59:02.429069 29400 tablet_replica.cc:333] T 686a629c7f3a458c8ee0a4276d7a5f27 P 7a440d3ea874483980ad5a12a47ce1d9: stopping tablet replica
I20251209 08:59:02.429181 29400 raft_consensus.cc:2243] T 686a629c7f3a458c8ee0a4276d7a5f27 P 7a440d3ea874483980ad5a12a47ce1d9 [term 1 LEADER]: Raft consensus shutting down.
I20251209 08:59:02.429374 29400 raft_consensus.cc:2272] T 686a629c7f3a458c8ee0a4276d7a5f27 P 7a440d3ea874483980ad5a12a47ce1d9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251209 08:59:02.429455 28857 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59180:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251209 08:59:02.429850 29401 ts_tablet_manager.cc:1916] T 686a629c7f3a458c8ee0a4276d7a5f27 P 84d473e778bd4553a757338c4cb05fb4: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251209 08:59:02.429960 28857 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251209 08:59:02.430117 29400 ts_tablet_manager.cc:1916] T 686a629c7f3a458c8ee0a4276d7a5f27 P 7a440d3ea874483980ad5a12a47ce1d9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251209 08:59:02.431913 29400 ts_tablet_manager.cc:1929] T 686a629c7f3a458c8ee0a4276d7a5f27 P 7a440d3ea874483980ad5a12a47ce1d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251209 08:59:02.431998 29400 log.cc:1199] T 686a629c7f3a458c8ee0a4276d7a5f27 P 7a440d3ea874483980ad5a12a47ce1d9: Deleting WAL directory at /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765270668420570-20677-0/minicluster-data/ts-2/wal/wals/686a629c7f3a458c8ee0a4276d7a5f27
I20251209 08:59:02.432294 29402 ts_tablet_manager.cc:1916] T 686a629c7f3a458c8ee0a4276d7a5f27 P 3cbbcfcfb7634dc588875d9c22e2328e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251209 08:59:02.432330 29400 ts_tablet_manager.cc:1950] T 686a629c7f3a458c8ee0a4276d7a5f27 P 7a440d3ea874483980ad5a12a47ce1d9: Deleting consensus metadata
I20251209 08:59:02.432508 29401 ts_tablet_manager.cc:1929] T 686a629c7f3a458c8ee0a4276d7a5f27 P 84d473e778bd4553a757338c4cb05fb4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251209 08:59:02.432571 29401 log.cc:1199] T 686a629c7f3a458c8ee0a4276d7a5f27 P 84d473e778bd4553a757338c4cb05fb4: Deleting WAL directory at /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765270668420570-20677-0/minicluster-data/ts-1/wal/wals/686a629c7f3a458c8ee0a4276d7a5f27
I20251209 08:59:02.432827 29401 ts_tablet_manager.cc:1950] T 686a629c7f3a458c8ee0a4276d7a5f27 P 84d473e778bd4553a757338c4cb05fb4: Deleting consensus metadata
I20251209 08:59:02.433408 28846 catalog_manager.cc:4985] TS 7a440d3ea874483980ad5a12a47ce1d9 (127.20.49.67:38547): tablet 686a629c7f3a458c8ee0a4276d7a5f27 (table test-table [id=b0e95c299a954d578b23c830ca337f71]) successfully deleted
I20251209 08:59:02.433511 28844 catalog_manager.cc:4985] TS 84d473e778bd4553a757338c4cb05fb4 (127.20.49.66:44029): tablet 686a629c7f3a458c8ee0a4276d7a5f27 (table test-table [id=b0e95c299a954d578b23c830ca337f71]) successfully deleted
I20251209 08:59:02.434556 29402 ts_tablet_manager.cc:1929] T 686a629c7f3a458c8ee0a4276d7a5f27 P 3cbbcfcfb7634dc588875d9c22e2328e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251209 08:59:02.434650 29402 log.cc:1199] T 686a629c7f3a458c8ee0a4276d7a5f27 P 3cbbcfcfb7634dc588875d9c22e2328e: Deleting WAL directory at /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765270668420570-20677-0/minicluster-data/ts-0/wal/wals/686a629c7f3a458c8ee0a4276d7a5f27
I20251209 08:59:02.434882 29402 ts_tablet_manager.cc:1950] T 686a629c7f3a458c8ee0a4276d7a5f27 P 3cbbcfcfb7634dc588875d9c22e2328e: Deleting consensus metadata
I20251209 08:59:02.435556 28847 catalog_manager.cc:4985] TS 3cbbcfcfb7634dc588875d9c22e2328e (127.20.49.65:39793): tablet 686a629c7f3a458c8ee0a4276d7a5f27 (table test-table [id=b0e95c299a954d578b23c830ca337f71]) successfully deleted
I20251209 08:59:02.436271 28999 tablet_service.cc:1505] Processing CreateTablet for tablet 9658c511bbae4950913c55dec3b2a0ae (DEFAULT_TABLE table=test-table [id=61e3bace7c654bdd84012c40e7564bfe]), partition=RANGE (key) PARTITION UNBOUNDED
I20251209 08:59:02.436409 29141 tablet_service.cc:1505] Processing CreateTablet for tablet 9658c511bbae4950913c55dec3b2a0ae (DEFAULT_TABLE table=test-table [id=61e3bace7c654bdd84012c40e7564bfe]), partition=RANGE (key) PARTITION UNBOUNDED
I20251209 08:59:02.436549 28999 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9658c511bbae4950913c55dec3b2a0ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20251209 08:59:02.436676 29141 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9658c511bbae4950913c55dec3b2a0ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20251209 08:59:02.437961 29283 tablet_service.cc:1505] Processing CreateTablet for tablet 9658c511bbae4950913c55dec3b2a0ae (DEFAULT_TABLE table=test-table [id=61e3bace7c654bdd84012c40e7564bfe]), partition=RANGE (key) PARTITION UNBOUNDED
I20251209 08:59:02.438237 29283 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9658c511bbae4950913c55dec3b2a0ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20251209 08:59:02.438287 29378 tablet_bootstrap.cc:492] T 9658c511bbae4950913c55dec3b2a0ae P 3cbbcfcfb7634dc588875d9c22e2328e: Bootstrap starting.
I20251209 08:59:02.438647 29379 tablet_bootstrap.cc:492] T 9658c511bbae4950913c55dec3b2a0ae P 84d473e778bd4553a757338c4cb05fb4: Bootstrap starting.
I20251209 08:59:02.439136 29378 tablet_bootstrap.cc:654] T 9658c511bbae4950913c55dec3b2a0ae P 3cbbcfcfb7634dc588875d9c22e2328e: Neither blocks nor log segments found. Creating new log.
I20251209 08:59:02.439706 29379 tablet_bootstrap.cc:654] T 9658c511bbae4950913c55dec3b2a0ae P 84d473e778bd4553a757338c4cb05fb4: Neither blocks nor log segments found. Creating new log.
I20251209 08:59:02.440078 29378 tablet_bootstrap.cc:492] T 9658c511bbae4950913c55dec3b2a0ae P 3cbbcfcfb7634dc588875d9c22e2328e: No bootstrap required, opened a new log
I20251209 08:59:02.440163 29378 ts_tablet_manager.cc:1403] T 9658c511bbae4950913c55dec3b2a0ae P 3cbbcfcfb7634dc588875d9c22e2328e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20251209 08:59:02.440636 29378 raft_consensus.cc:359] T 9658c511bbae4950913c55dec3b2a0ae P 3cbbcfcfb7634dc588875d9c22e2328e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cbbcfcfb7634dc588875d9c22e2328e" member_type: VOTER last_known_addr { host: "127.20.49.65" port: 39793 } } peers { permanent_uuid: "84d473e778bd4553a757338c4cb05fb4" member_type: VOTER last_known_addr { host: "127.20.49.66" port: 44029 } } peers { permanent_uuid: "7a440d3ea874483980ad5a12a47ce1d9" member_type: VOTER last_known_addr { host: "127.20.49.67" port: 38547 } }
I20251209 08:59:02.440732 29378 raft_consensus.cc:385] T 9658c511bbae4950913c55dec3b2a0ae P 3cbbcfcfb7634dc588875d9c22e2328e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251209 08:59:02.440768 29378 raft_consensus.cc:740] T 9658c511bbae4950913c55dec3b2a0ae P 3cbbcfcfb7634dc588875d9c22e2328e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3cbbcfcfb7634dc588875d9c22e2328e, State: Initialized, Role: FOLLOWER
I20251209 08:59:02.440908 29378 consensus_queue.cc:260] T 9658c511bbae4950913c55dec3b2a0ae P 3cbbcfcfb7634dc588875d9c22e2328e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cbbcfcfb7634dc588875d9c22e2328e" member_type: VOTER last_known_addr { host: "127.20.49.65" port: 39793 } } peers { permanent_uuid: "84d473e778bd4553a757338c4cb05fb4" member_type: VOTER last_known_addr { host: "127.20.49.66" port: 44029 } } peers { permanent_uuid: "7a440d3ea874483980ad5a12a47ce1d9" member_type: VOTER last_known_addr { host: "127.20.49.67" port: 38547 } }
I20251209 08:59:02.441143 29378 ts_tablet_manager.cc:1434] T 9658c511bbae4950913c55dec3b2a0ae P 3cbbcfcfb7634dc588875d9c22e2328e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20251209 08:59:02.441350 29377 tablet_bootstrap.cc:492] T 9658c511bbae4950913c55dec3b2a0ae P 7a440d3ea874483980ad5a12a47ce1d9: Bootstrap starting.
I20251209 08:59:02.441362 29379 tablet_bootstrap.cc:492] T 9658c511bbae4950913c55dec3b2a0ae P 84d473e778bd4553a757338c4cb05fb4: No bootstrap required, opened a new log
I20251209 08:59:02.441438 29379 ts_tablet_manager.cc:1403] T 9658c511bbae4950913c55dec3b2a0ae P 84d473e778bd4553a757338c4cb05fb4: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20251209 08:59:02.441905 29379 raft_consensus.cc:359] T 9658c511bbae4950913c55dec3b2a0ae P 84d473e778bd4553a757338c4cb05fb4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cbbcfcfb7634dc588875d9c22e2328e" member_type: VOTER last_known_addr { host: "127.20.49.65" port: 39793 } } peers { permanent_uuid: "84d473e778bd4553a757338c4cb05fb4" member_type: VOTER last_known_addr { host: "127.20.49.66" port: 44029 } } peers { permanent_uuid: "7a440d3ea874483980ad5a12a47ce1d9" member_type: VOTER last_known_addr { host: "127.20.49.67" port: 38547 } }
I20251209 08:59:02.442018 29379 raft_consensus.cc:385] T 9658c511bbae4950913c55dec3b2a0ae P 84d473e778bd4553a757338c4cb05fb4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251209 08:59:02.442058 29379 raft_consensus.cc:740] T 9658c511bbae4950913c55dec3b2a0ae P 84d473e778bd4553a757338c4cb05fb4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84d473e778bd4553a757338c4cb05fb4, State: Initialized, Role: FOLLOWER
I20251209 08:59:02.442225 29377 tablet_bootstrap.cc:654] T 9658c511bbae4950913c55dec3b2a0ae P 7a440d3ea874483980ad5a12a47ce1d9: Neither blocks nor log segments found. Creating new log.
I20251209 08:59:02.442184 29379 consensus_queue.cc:260] T 9658c511bbae4950913c55dec3b2a0ae P 84d473e778bd4553a757338c4cb05fb4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cbbcfcfb7634dc588875d9c22e2328e" member_type: VOTER last_known_addr { host: "127.20.49.65" port: 39793 } } peers { permanent_uuid: "84d473e778bd4553a757338c4cb05fb4" member_type: VOTER last_known_addr { host: "127.20.49.66" port: 44029 } } peers { permanent_uuid: "7a440d3ea874483980ad5a12a47ce1d9" member_type: VOTER last_known_addr { host: "127.20.49.67" port: 38547 } }
I20251209 08:59:02.442476 29379 ts_tablet_manager.cc:1434] T 9658c511bbae4950913c55dec3b2a0ae P 84d473e778bd4553a757338c4cb05fb4: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20251209 08:59:02.443369 29377 tablet_bootstrap.cc:492] T 9658c511bbae4950913c55dec3b2a0ae P 7a440d3ea874483980ad5a12a47ce1d9: No bootstrap required, opened a new log
I20251209 08:59:02.443470 29377 ts_tablet_manager.cc:1403] T 9658c511bbae4950913c55dec3b2a0ae P 7a440d3ea874483980ad5a12a47ce1d9: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20251209 08:59:02.443966 29377 raft_consensus.cc:359] T 9658c511bbae4950913c55dec3b2a0ae P 7a440d3ea874483980ad5a12a47ce1d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cbbcfcfb7634dc588875d9c22e2328e" member_type: VOTER last_known_addr { host: "127.20.49.65" port: 39793 } } peers { permanent_uuid: "84d473e778bd4553a757338c4cb05fb4" member_type: VOTER last_known_addr { host: "127.20.49.66" port: 44029 } } peers { permanent_uuid: "7a440d3ea874483980ad5a12a47ce1d9" member_type: VOTER last_known_addr { host: "127.20.49.67" port: 38547 } }
I20251209 08:59:02.444082 29377 raft_consensus.cc:385] T 9658c511bbae4950913c55dec3b2a0ae P 7a440d3ea874483980ad5a12a47ce1d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251209 08:59:02.444134 29377 raft_consensus.cc:740] T 9658c511bbae4950913c55dec3b2a0ae P 7a440d3ea874483980ad5a12a47ce1d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a440d3ea874483980ad5a12a47ce1d9, State: Initialized, Role: FOLLOWER
I20251209 08:59:02.444272 29377 consensus_queue.cc:260] T 9658c511bbae4950913c55dec3b2a0ae P 7a440d3ea874483980ad5a12a47ce1d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cbbcfcfb7634dc588875d9c22e2328e" member_type: VOTER last_known_addr { host: "127.20.49.65" port: 39793 } } peers { permanent_uuid: "84d473e778bd4553a757338c4cb05fb4" member_type: VOTER last_known_addr { host: "127.20.49.66" port: 44029 } } peers { permanent_uuid: "7a440d3ea874483980ad5a12a47ce1d9" member_type: VOTER last_known_addr { host: "127.20.49.67" port: 38547 } }
I20251209 08:59:02.444579 29377 ts_tablet_manager.cc:1434] T 9658c511bbae4950913c55dec3b2a0ae P 7a440d3ea874483980ad5a12a47ce1d9: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
W20251209 08:59:02.527249 29066 tablet.cc:2378] T 9658c511bbae4950913c55dec3b2a0ae P 3cbbcfcfb7634dc588875d9c22e2328e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20251209 08:59:02.625756 29396 raft_consensus.cc:493] T 9658c511bbae4950913c55dec3b2a0ae P 7a440d3ea874483980ad5a12a47ce1d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251209 08:59:02.625921 29396 raft_consensus.cc:515] T 9658c511bbae4950913c55dec3b2a0ae P 7a440d3ea874483980ad5a12a47ce1d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cbbcfcfb7634dc588875d9c22e2328e" member_type: VOTER last_known_addr { host: "127.20.49.65" port: 39793 } } peers { permanent_uuid: "84d473e778bd4553a757338c4cb05fb4" member_type: VOTER last_known_addr { host: "127.20.49.66" port: 44029 } } peers { permanent_uuid: "7a440d3ea874483980ad5a12a47ce1d9" member_type: VOTER last_known_addr { host: "127.20.49.67" port: 38547 } }
I20251209 08:59:02.626379 29396 leader_election.cc:290] T 9658c511bbae4950913c55dec3b2a0ae P 7a440d3ea874483980ad5a12a47ce1d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3cbbcfcfb7634dc588875d9c22e2328e (127.20.49.65:39793), 84d473e778bd4553a757338c4cb05fb4 (127.20.49.66:44029)
I20251209 08:59:02.626833 29019 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9658c511bbae4950913c55dec3b2a0ae" candidate_uuid: "7a440d3ea874483980ad5a12a47ce1d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3cbbcfcfb7634dc588875d9c22e2328e" is_pre_election: true
I20251209 08:59:02.626863 29161 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9658c511bbae4950913c55dec3b2a0ae" candidate_uuid: "7a440d3ea874483980ad5a12a47ce1d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84d473e778bd4553a757338c4cb05fb4" is_pre_election: true
I20251209 08:59:02.626981 29019 raft_consensus.cc:2468] T 9658c511bbae4950913c55dec3b2a0ae P 3cbbcfcfb7634dc588875d9c22e2328e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a440d3ea874483980ad5a12a47ce1d9 in term 0.
I20251209 08:59:02.626991 29161 raft_consensus.cc:2468] T 9658c511bbae4950913c55dec3b2a0ae P 84d473e778bd4553a757338c4cb05fb4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a440d3ea874483980ad5a12a47ce1d9 in term 0.
I20251209 08:59:02.627291 29234 leader_election.cc:304] T 9658c511bbae4950913c55dec3b2a0ae P 7a440d3ea874483980ad5a12a47ce1d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a440d3ea874483980ad5a12a47ce1d9, 84d473e778bd4553a757338c4cb05fb4; no voters:
I20251209 08:59:02.627465 29396 raft_consensus.cc:2804] T 9658c511bbae4950913c55dec3b2a0ae P 7a440d3ea874483980ad5a12a47ce1d9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251209 08:59:02.627524 29396 raft_consensus.cc:493] T 9658c511bbae4950913c55dec3b2a0ae P 7a440d3ea874483980ad5a12a47ce1d9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251209 08:59:02.627575 29396 raft_consensus.cc:3060] T 9658c511bbae4950913c55dec3b2a0ae P 7a440d3ea874483980ad5a12a47ce1d9 [term 0 FOLLOWER]: Advancing to term 1
I20251209 08:59:02.628490 29396 raft_consensus.cc:515] T 9658c511bbae4950913c55dec3b2a0ae P 7a440d3ea874483980ad5a12a47ce1d9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cbbcfcfb7634dc588875d9c22e2328e" member_type: VOTER last_known_addr { host: "127.20.49.65" port: 39793 } } peers { permanent_uuid: "84d473e778bd4553a757338c4cb05fb4" member_type: VOTER last_known_addr { host: "127.20.49.66" port: 44029 } } peers { permanent_uuid: "7a440d3ea874483980ad5a12a47ce1d9" member_type: VOTER last_known_addr { host: "127.20.49.67" port: 38547 } }
I20251209 08:59:02.628841 29396 leader_election.cc:290] T 9658c511bbae4950913c55dec3b2a0ae P 7a440d3ea874483980ad5a12a47ce1d9 [CANDIDATE]: Term 1 election: Requested vote from peers 3cbbcfcfb7634dc588875d9c22e2328e (127.20.49.65:39793), 84d473e778bd4553a757338c4cb05fb4 (127.20.49.66:44029)
I20251209 08:59:02.629201 29019 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9658c511bbae4950913c55dec3b2a0ae" candidate_uuid: "7a440d3ea874483980ad5a12a47ce1d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3cbbcfcfb7634dc588875d9c22e2328e"
I20251209 08:59:02.629216 29161 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9658c511bbae4950913c55dec3b2a0ae" candidate_uuid: "7a440d3ea874483980ad5a12a47ce1d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84d473e778bd4553a757338c4cb05fb4"
I20251209 08:59:02.629312 29161 raft_consensus.cc:3060] T 9658c511bbae4950913c55dec3b2a0ae P 84d473e778bd4553a757338c4cb05fb4 [term 0 FOLLOWER]: Advancing to term 1
I20251209 08:59:02.629323 29019 raft_consensus.cc:3060] T 9658c511bbae4950913c55dec3b2a0ae P 3cbbcfcfb7634dc588875d9c22e2328e [term 0 FOLLOWER]: Advancing to term 1
I20251209 08:59:02.630317 29161 raft_consensus.cc:2468] T 9658c511bbae4950913c55dec3b2a0ae P 84d473e778bd4553a757338c4cb05fb4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a440d3ea874483980ad5a12a47ce1d9 in term 1.
I20251209 08:59:02.630337 29019 raft_consensus.cc:2468] T 9658c511bbae4950913c55dec3b2a0ae P 3cbbcfcfb7634dc588875d9c22e2328e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a440d3ea874483980ad5a12a47ce1d9 in term 1.
I20251209 08:59:02.630614 29237 leader_election.cc:304] T 9658c511bbae4950913c55dec3b2a0ae P 7a440d3ea874483980ad5a12a47ce1d9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3cbbcfcfb7634dc588875d9c22e2328e, 7a440d3ea874483980ad5a12a47ce1d9; no voters:
I20251209 08:59:02.630779 29396 raft_consensus.cc:2804] T 9658c511bbae4950913c55dec3b2a0ae P 7a440d3ea874483980ad5a12a47ce1d9 [term 1 FOLLOWER]: Leader election won for term 1
I20251209 08:59:02.630882 29396 raft_consensus.cc:697] T 9658c511bbae4950913c55dec3b2a0ae P 7a440d3ea874483980ad5a12a47ce1d9 [term 1 LEADER]: Becoming Leader. State: Replica: 7a440d3ea874483980ad5a12a47ce1d9, State: Running, Role: LEADER
I20251209 08:59:02.631023 29396 consensus_queue.cc:237] T 9658c511bbae4950913c55dec3b2a0ae P 7a440d3ea874483980ad5a12a47ce1d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cbbcfcfb7634dc588875d9c22e2328e" member_type: VOTER last_known_addr { host: "127.20.49.65" port: 39793 } } peers { permanent_uuid: "84d473e778bd4553a757338c4cb05fb4" member_type: VOTER last_known_addr { host: "127.20.49.66" port: 44029 } } peers { permanent_uuid: "7a440d3ea874483980ad5a12a47ce1d9" member_type: VOTER last_known_addr { host: "127.20.49.67" port: 38547 } }
I20251209 08:59:02.632691 28859 catalog_manager.cc:5654] T 9658c511bbae4950913c55dec3b2a0ae P 7a440d3ea874483980ad5a12a47ce1d9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7a440d3ea874483980ad5a12a47ce1d9 (127.20.49.67). New cstate: current_term: 1 leader_uuid: "7a440d3ea874483980ad5a12a47ce1d9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cbbcfcfb7634dc588875d9c22e2328e" member_type: VOTER last_known_addr { host: "127.20.49.65" port: 39793 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "84d473e778bd4553a757338c4cb05fb4" member_type: VOTER last_known_addr { host: "127.20.49.66" port: 44029 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a440d3ea874483980ad5a12a47ce1d9" member_type: VOTER last_known_addr { host: "127.20.49.67" port: 38547 } health_report { overall_health: HEALTHY } } }
I20251209 08:59:02.691172 29161 raft_consensus.cc:1275] T 9658c511bbae4950913c55dec3b2a0ae P 84d473e778bd4553a757338c4cb05fb4 [term 1 FOLLOWER]: Refusing update from remote peer 7a440d3ea874483980ad5a12a47ce1d9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251209 08:59:02.691172 29019 raft_consensus.cc:1275] T 9658c511bbae4950913c55dec3b2a0ae P 3cbbcfcfb7634dc588875d9c22e2328e [term 1 FOLLOWER]: Refusing update from remote peer 7a440d3ea874483980ad5a12a47ce1d9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251209 08:59:02.691576 29396 consensus_queue.cc:1048] T 9658c511bbae4950913c55dec3b2a0ae P 7a440d3ea874483980ad5a12a47ce1d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84d473e778bd4553a757338c4cb05fb4" member_type: VOTER last_known_addr { host: "127.20.49.66" port: 44029 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251209 08:59:02.691726 29385 consensus_queue.cc:1048] T 9658c511bbae4950913c55dec3b2a0ae P 7a440d3ea874483980ad5a12a47ce1d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3cbbcfcfb7634dc588875d9c22e2328e" member_type: VOTER last_known_addr { host: "127.20.49.65" port: 39793 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251209 08:59:02.724274 28859 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59180:
table { table_name: "test-table" } modify_external_catalogs: true
I20251209 08:59:02.724426 28859 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59180:
table { table_name: "test-table" } modify_external_catalogs: true
I20251209 08:59:02.726480 28859 catalog_manager.cc:5941] T 00000000000000000000000000000000 P 7f2aac7cb38e4c609b71560ad54a20b1: Sending DeleteTablet for 3 replicas of tablet 9658c511bbae4950913c55dec3b2a0ae
I20251209 08:59:02.727089 29141 tablet_service.cc:1552] Processing DeleteTablet for tablet 9658c511bbae4950913c55dec3b2a0ae with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-09 08:59:02 UTC) from {username='oryx', principal='oryx/127.20.49.126@KRBTEST.COM'} at 127.0.0.1:52844
I20251209 08:59:02.727089 29283 tablet_service.cc:1552] Processing DeleteTablet for tablet 9658c511bbae4950913c55dec3b2a0ae with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-09 08:59:02 UTC) from {username='oryx', principal='oryx/127.20.49.126@KRBTEST.COM'} at 127.0.0.1:35670
I20251209 08:59:02.727110 28999 tablet_service.cc:1552] Processing DeleteTablet for tablet 9658c511bbae4950913c55dec3b2a0ae with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-09 08:59:02 UTC) from {username='oryx', principal='oryx/127.20.49.126@KRBTEST.COM'} at 127.0.0.1:60562
I20251209 08:59:02.727286 29401 tablet_replica.cc:333] T 9658c511bbae4950913c55dec3b2a0ae P 84d473e778bd4553a757338c4cb05fb4: stopping tablet replica
I20251209 08:59:02.727286 29400 tablet_replica.cc:333] T 9658c511bbae4950913c55dec3b2a0ae P 7a440d3ea874483980ad5a12a47ce1d9: stopping tablet replica
I20251209 08:59:02.727312 29402 tablet_replica.cc:333] T 9658c511bbae4950913c55dec3b2a0ae P 3cbbcfcfb7634dc588875d9c22e2328e: stopping tablet replica
I20251209 08:59:02.727406 29401 raft_consensus.cc:2243] T 9658c511bbae4950913c55dec3b2a0ae P 84d473e778bd4553a757338c4cb05fb4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251209 08:59:02.727406 29400 raft_consensus.cc:2243] T 9658c511bbae4950913c55dec3b2a0ae P 7a440d3ea874483980ad5a12a47ce1d9 [term 1 LEADER]: Raft consensus shutting down.
I20251209 08:59:02.727434 29402 raft_consensus.cc:2243] T 9658c511bbae4950913c55dec3b2a0ae P 3cbbcfcfb7634dc588875d9c22e2328e [term 1 FOLLOWER]: Raft consensus shutting down.
I20251209 08:59:02.727530 29401 raft_consensus.cc:2272] T 9658c511bbae4950913c55dec3b2a0ae P 84d473e778bd4553a757338c4cb05fb4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251209 08:59:02.727600 29402 raft_consensus.cc:2272] T 9658c511bbae4950913c55dec3b2a0ae P 3cbbcfcfb7634dc588875d9c22e2328e [term 1 FOLLOWER]: Raft consensus is shut down!
I20251209 08:59:02.727628 29400 raft_consensus.cc:2272] T 9658c511bbae4950913c55dec3b2a0ae P 7a440d3ea874483980ad5a12a47ce1d9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251209 08:59:02.728437 29401 ts_tablet_manager.cc:1916] T 9658c511bbae4950913c55dec3b2a0ae P 84d473e778bd4553a757338c4cb05fb4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251209 08:59:02.728457 20677 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task8XuDZV/build/debug/bin/kudu with pid 28934
I20251209 08:59:02.728576 29400 ts_tablet_manager.cc:1916] T 9658c511bbae4950913c55dec3b2a0ae P 7a440d3ea874483980ad5a12a47ce1d9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251209 08:59:02.731343 29400 ts_tablet_manager.cc:1929] T 9658c511bbae4950913c55dec3b2a0ae P 7a440d3ea874483980ad5a12a47ce1d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20251209 08:59:02.731421 29400 log.cc:1199] T 9658c511bbae4950913c55dec3b2a0ae P 7a440d3ea874483980ad5a12a47ce1d9: Deleting WAL directory at /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765270668420570-20677-0/minicluster-data/ts-2/wal/wals/9658c511bbae4950913c55dec3b2a0ae
I20251209 08:59:02.731534 29401 ts_tablet_manager.cc:1929] T 9658c511bbae4950913c55dec3b2a0ae P 84d473e778bd4553a757338c4cb05fb4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20251209 08:59:02.731627 29401 log.cc:1199] T 9658c511bbae4950913c55dec3b2a0ae P 84d473e778bd4553a757338c4cb05fb4: Deleting WAL directory at /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1765270668420570-20677-0/minicluster-data/ts-1/wal/wals/9658c511bbae4950913c55dec3b2a0ae
I20251209 08:59:02.731691 29400 ts_tablet_manager.cc:1950] T 9658c511bbae4950913c55dec3b2a0ae P 7a440d3ea874483980ad5a12a47ce1d9: Deleting consensus metadata
I20251209 08:59:02.731937 29401 ts_tablet_manager.cc:1950] T 9658c511bbae4950913c55dec3b2a0ae P 84d473e778bd4553a757338c4cb05fb4: Deleting consensus metadata
I20251209 08:59:02.732358 28846 catalog_manager.cc:4985] TS 7a440d3ea874483980ad5a12a47ce1d9 (127.20.49.67:38547): tablet 9658c511bbae4950913c55dec3b2a0ae (table test-table [id=61e3bace7c654bdd84012c40e7564bfe]) successfully deleted
I20251209 08:59:02.732625 28844 catalog_manager.cc:4985] TS 84d473e778bd4553a757338c4cb05fb4 (127.20.49.66:44029): tablet 9658c511bbae4950913c55dec3b2a0ae (table test-table [id=61e3bace7c654bdd84012c40e7564bfe]) successfully deleted
W20251209 08:59:02.738353 28847 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.20.49.65:39793 (error 108)
I20251209 08:59:02.738488 20677 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task8XuDZV/build/debug/bin/kudu with pid 29076
I20251209 08:59:02.740245 29460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:59:02.739424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.49.65:39793 (local address 127.0.0.1:60568)
1209 08:59:02.740027 (+ 603us) negotiation.cc:107] Waiting for socket to connect
1209 08:59:02.740102 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.49.65:39793: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":481,"thread_start_us":70,"threads_started":1}
W20251209 08:59:02.740417 28847 catalog_manager.cc:4712] TS 3cbbcfcfb7634dc588875d9c22e2328e (127.20.49.65:39793): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 9658c511bbae4950913c55dec3b2a0ae: Network error: Client connection negotiation failed: client connection to 127.20.49.65:39793: connect: Connection refused (error 111)
I20251209 08:59:02.745477 20677 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task8XuDZV/build/debug/bin/kudu with pid 29218
I20251209 08:59:02.752868 20677 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task8XuDZV/build/debug/bin/kudu with pid 28828
2025-12-09T08:59:02Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (4104 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1765270668420570-20677-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Dec 09 08:59:02 dist-test-slave-8w6c krb5kdc[29463](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Dec 09 08:59:02 dist-test-slave-8w6c krb5kdc[29463](info): set up 2 sockets
Dec 09 08:59:02 dist-test-slave-8w6c krb5kdc[29463](info): commencing operation
krb5kdc: starting...
W20251209 08:59:04.754748 20677 mini_kdc.cc:121] Time spent starting KDC: real 1.977s user 0.002s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1765270668420570-20677-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1765270668420570-20677-0/krb5kdc/test-user.keytab.
Dec 09 08:59:04 dist-test-slave-8w6c krb5kdc[29463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765270744, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-12-09T08:59:04Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-12-09T08:59:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.49.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.49.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.49.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1765270668420570-20677-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.20.49.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1765270668420570-20677-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.20.49.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.20.49.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.20.49.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1765270668420570-20677-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.20.49.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1765270668420570-20677-0/krb5kdc/kudu.keytab.
I20251209 08:59:04.900786 20677 external_mini_cluster.cc:1366] Running /tmp/dist-test-task8XuDZV/build/debug/bin/kudu
/tmp/dist-test-task8XuDZV/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1765270668420570-20677-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1765270668420570-20677-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1765270668420570-20677-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1765270668420570-20677-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.49.126:39329
--webserver_interface=127.20.49.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1765270668420570-20677-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.49.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.20.49.84:36091
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.49.126:39329
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1765270668420570-20677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1765270668420570-20677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1765270668420570-20677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251209 08:59:05.015686 29479 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251209 08:59:05.015935 29479 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251209 08:59:05.016009 29479 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251209 08:59:05.019411 29479 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251209 08:59:05.019490 29479 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251209 08:59:05.019513 29479 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251209 08:59:05.019534 29479 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251209 08:59:05.019629 29479 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251209 08:59:05.024093 29479 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.49.84:36091
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1765270668420570-20677-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1765270668420570-20677-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.49.126:39329
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1765270668420570-20677-0/krb5kdc/kudu.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.49.126:39329
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1765270668420570-20677-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.49.126
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29479
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1765270668420570-20677-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 468897755969d686fb08386f42f7835685e7953a
build type DEBUG
built by None at 09 Dec 2025 08:43:15 UTC on 24a791456cd2
build id 9426
I20251209 08:59:05.025239 29479 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251209 08:59:05.026124 29479 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251209 08:59:05.031649 29485 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251209 08:59:05.031626 29484 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251209 08:59:05.031708 29479 server_base.cc:1047] running on GCE node
W20251209 08:59:05.031867 29487 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251209 08:59:05.032366 29479 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251209 08:59:05.033286 29479 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251209 08:59:05.034461 29479 hybrid_clock.cc:648] HybridClock initialized: now 1765270745034448 us; error 27 us; skew 500 ppm
W20251209 08:59:05.037878 29479 builtin_ntp.cc:688] coult not shutdown socket: Network error: shutdown error: Transport endpoint is not connected (error 107)
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2025-12-09T08:59:05Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (2283 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765270668420570-20677-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Dec 09 08:59:05 dist-test-slave-8w6c krb5kdc[29492](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Dec 09 08:59:05 dist-test-slave-8w6c krb5kdc[29492](info): set up 2 sockets
Dec 09 08:59:05 dist-test-slave-8w6c krb5kdc[29492](info): commencing operation
krb5kdc: starting...
W20251209 08:59:07.046888 20677 mini_kdc.cc:121] Time spent starting KDC: real 1.986s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765270668420570-20677-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765270668420570-20677-0/krb5kdc/test-user.keytab.
Dec 09 08:59:07 dist-test-slave-8w6c krb5kdc[29492](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765270747, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-12-09T08:59:07Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-12-09T08:59:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.49.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.49.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.49.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765270668420570-20677-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.20.49.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765270668420570-20677-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.20.49.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.20.49.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.20.49.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765270668420570-20677-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.20.49.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765270668420570-20677-0/krb5kdc/kudu.keytab.
I20251209 08:59:07.189114 20677 external_mini_cluster.cc:1366] Running /tmp/dist-test-task8XuDZV/build/debug/bin/kudu
/tmp/dist-test-task8XuDZV/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765270668420570-20677-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765270668420570-20677-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765270668420570-20677-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765270668420570-20677-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.49.126:44103
--webserver_interface=127.20.49.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765270668420570-20677-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.49.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.20.49.84:41011
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.49.126:44103
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765270668420570-20677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765270668420570-20677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765270668420570-20677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251209 08:59:07.301436 29508 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251209 08:59:07.301726 29508 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251209 08:59:07.301800 29508 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251209 08:59:07.305112 29508 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251209 08:59:07.305179 29508 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251209 08:59:07.305203 29508 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251209 08:59:07.305222 29508 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251209 08:59:07.305240 29508 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251209 08:59:07.309559 29508 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.49.84:41011
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765270668420570-20677-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765270668420570-20677-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.49.126:44103
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765270668420570-20677-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.49.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.49.126:44103
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765270668420570-20677-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.49.126
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29508
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765270668420570-20677-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 468897755969d686fb08386f42f7835685e7953a
build type DEBUG
built by None at 09 Dec 2025 08:43:15 UTC on 24a791456cd2
build id 9426
I20251209 08:59:07.310693 29508 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251209 08:59:07.311553 29508 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251209 08:59:07.317137 29516 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251209 08:59:07.317270 29508 server_base.cc:1047] running on GCE node
W20251209 08:59:07.317138 29513 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251209 08:59:07.317423 29514 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251209 08:59:07.317983 29508 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251209 08:59:07.318924 29508 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251209 08:59:07.320088 29508 hybrid_clock.cc:648] HybridClock initialized: now 1765270747320072 us; error 35 us; skew 500 ppm
Dec 09 08:59:07 dist-test-slave-8w6c krb5kdc[29492](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765270747, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.49.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251209 08:59:07.322894 29508 init.cc:377] Logged in from keytab as kudu/127.20.49.126@KRBTEST.COM (short username kudu)
I20251209 08:59:07.323998 29508 webserver.cc:492] Webserver started at http://127.20.49.126:41583/ using document root <none> and password file <none>
I20251209 08:59:07.324548 29508 fs_manager.cc:362] Metadata directory not provided
I20251209 08:59:07.324601 29508 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251209 08:59:07.324841 29508 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251209 08:59:07.326682 29508 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765270668420570-20677-0/minicluster-data/master-0/data/instance:
uuid: "28875c92005b46f788bbde9310eb27b5"
format_stamp: "Formatted at 2025-12-09 08:59:07 on dist-test-slave-8w6c"
server_key: "8d592e39650c3efb8056dc9eac6aa69a"
server_key_iv: "2e3135fd11787e11b5fe0a7f6a35cd02"
server_key_version: "encryptionkey@0"
I20251209 08:59:07.327165 29508 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765270668420570-20677-0/minicluster-data/master-0/wal/instance:
uuid: "28875c92005b46f788bbde9310eb27b5"
format_stamp: "Formatted at 2025-12-09 08:59:07 on dist-test-slave-8w6c"
server_key: "8d592e39650c3efb8056dc9eac6aa69a"
server_key_iv: "2e3135fd11787e11b5fe0a7f6a35cd02"
server_key_version: "encryptionkey@0"
I20251209 08:59:07.330736 29508 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20251209 08:59:07.333024 29523 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251209 08:59:07.334125 29508 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20251209 08:59:07.334232 29508 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765270668420570-20677-0/minicluster-data/master-0/data,/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765270668420570-20677-0/minicluster-data/master-0/wal
uuid: "28875c92005b46f788bbde9310eb27b5"
format_stamp: "Formatted at 2025-12-09 08:59:07 on dist-test-slave-8w6c"
server_key: "8d592e39650c3efb8056dc9eac6aa69a"
server_key_iv: "2e3135fd11787e11b5fe0a7f6a35cd02"
server_key_version: "encryptionkey@0"
I20251209 08:59:07.334337 29508 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765270668420570-20677-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765270668420570-20677-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765270668420570-20677-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251209 08:59:07.344133 29508 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251209 08:59:07.347589 29508 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251209 08:59:07.347929 29508 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251209 08:59:07.355464 29508 rpc_server.cc:307] RPC server started. Bound to: 127.20.49.126:44103
I20251209 08:59:07.355475 29575 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.49.126:44103 every 8 connection(s)
I20251209 08:59:07.356514 29508 server_base.cc:1179] Dumped server information to /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765270668420570-20677-0/minicluster-data/master-0/data/info.pb
I20251209 08:59:07.359390 29576 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251209 08:59:07.365211 29576 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 28875c92005b46f788bbde9310eb27b5: Bootstrap starting.
I20251209 08:59:07.365643 20677 external_mini_cluster.cc:1428] Started /tmp/dist-test-task8XuDZV/build/debug/bin/kudu as pid 29508
I20251209 08:59:07.365805 20677 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765270668420570-20677-0/minicluster-data/master-0/wal/instance
I20251209 08:59:07.366042 20677 external_mini_cluster.cc:1468] Setting key a77304134f2614d1aa7cf6b486408cb0
I20251209 08:59:07.367913 29576 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 28875c92005b46f788bbde9310eb27b5: Neither blocks nor log segments found. Creating new log.
I20251209 08:59:07.368649 29576 log.cc:826] T 00000000000000000000000000000000 P 28875c92005b46f788bbde9310eb27b5: Log is configured to *not* fsync() on all Append() calls
I20251209 08:59:07.370493 29576 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 28875c92005b46f788bbde9310eb27b5: No bootstrap required, opened a new log
I20251209 08:59:07.373270 29576 raft_consensus.cc:359] T 00000000000000000000000000000000 P 28875c92005b46f788bbde9310eb27b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28875c92005b46f788bbde9310eb27b5" member_type: VOTER last_known_addr { host: "127.20.49.126" port: 44103 } }
I20251209 08:59:07.373481 29576 raft_consensus.cc:385] T 00000000000000000000000000000000 P 28875c92005b46f788bbde9310eb27b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251209 08:59:07.373553 29576 raft_consensus.cc:740] T 00000000000000000000000000000000 P 28875c92005b46f788bbde9310eb27b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28875c92005b46f788bbde9310eb27b5, State: Initialized, Role: FOLLOWER
Dec 09 08:59:07 dist-test-slave-8w6c krb5kdc[29492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765270747, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.49.126@KRBTEST.COM
I20251209 08:59:07.374161 29576 consensus_queue.cc:260] T 00000000000000000000000000000000 P 28875c92005b46f788bbde9310eb27b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28875c92005b46f788bbde9310eb27b5" member_type: VOTER last_known_addr { host: "127.20.49.126" port: 44103 } }
I20251209 08:59:07.374353 29576 raft_consensus.cc:399] T 00000000000000000000000000000000 P 28875c92005b46f788bbde9310eb27b5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251209 08:59:07.374435 29576 raft_consensus.cc:493] T 00000000000000000000000000000000 P 28875c92005b46f788bbde9310eb27b5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251209 08:59:07.374528 29576 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 28875c92005b46f788bbde9310eb27b5 [term 0 FOLLOWER]: Advancing to term 1
I20251209 08:59:07.375665 29576 raft_consensus.cc:515] T 00000000000000000000000000000000 P 28875c92005b46f788bbde9310eb27b5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28875c92005b46f788bbde9310eb27b5" member_type: VOTER last_known_addr { host: "127.20.49.126" port: 44103 } }
I20251209 08:59:07.376035 29576 leader_election.cc:304] T 00000000000000000000000000000000 P 28875c92005b46f788bbde9310eb27b5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 28875c92005b46f788bbde9310eb27b5; no voters:
I20251209 08:59:07.376359 29576 leader_election.cc:290] T 00000000000000000000000000000000 P 28875c92005b46f788bbde9310eb27b5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20251209 08:59:07.376458 29581 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 28875c92005b46f788bbde9310eb27b5 [term 1 FOLLOWER]: Leader election won for term 1
I20251209 08:59:07.376631 29581 raft_consensus.cc:697] T 00000000000000000000000000000000 P 28875c92005b46f788bbde9310eb27b5 [term 1 LEADER]: Becoming Leader. State: Replica: 28875c92005b46f788bbde9310eb27b5, State: Running, Role: LEADER
I20251209 08:59:07.376940 29581 consensus_queue.cc:237] T 00000000000000000000000000000000 P 28875c92005b46f788bbde9310eb27b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28875c92005b46f788bbde9310eb27b5" member_type: VOTER last_known_addr { host: "127.20.49.126" port: 44103 } }
I20251209 08:59:07.377595 29576 sys_catalog.cc:565] T 00000000000000000000000000000000 P 28875c92005b46f788bbde9310eb27b5 [sys.catalog]: configured and running, proceeding with master startup.
I20251209 08:59:07.378860 29582 sys_catalog.cc:455] T 00000000000000000000000000000000 P 28875c92005b46f788bbde9310eb27b5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "28875c92005b46f788bbde9310eb27b5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28875c92005b46f788bbde9310eb27b5" member_type: VOTER last_known_addr { host: "127.20.49.126" port: 44103 } } }
I20251209 08:59:07.378983 29582 sys_catalog.cc:458] T 00000000000000000000000000000000 P 28875c92005b46f788bbde9310eb27b5 [sys.catalog]: This master's current role is: LEADER
I20251209 08:59:07.379298 29581 sys_catalog.cc:455] T 00000000000000000000000000000000 P 28875c92005b46f788bbde9310eb27b5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 28875c92005b46f788bbde9310eb27b5. Latest consensus state: current_term: 1 leader_uuid: "28875c92005b46f788bbde9310eb27b5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28875c92005b46f788bbde9310eb27b5" member_type: VOTER last_known_addr { host: "127.20.49.126" port: 44103 } } }
I20251209 08:59:07.379395 29581 sys_catalog.cc:458] T 00000000000000000000000000000000 P 28875c92005b46f788bbde9310eb27b5 [sys.catalog]: This master's current role is: LEADER
I20251209 08:59:07.381167 29579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:59:07.367329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52468 (local address 127.20.49.126:44103)
1209 08:59:07.367832 (+ 503us) server_negotiation.cc:207] Beginning negotiation
1209 08:59:07.367843 (+ 11us) server_negotiation.cc:400] Waiting for connection header
1209 08:59:07.367879 (+ 36us) server_negotiation.cc:408] Connection header received
1209 08:59:07.368486 (+ 607us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1209 08:59:07.368505 (+ 19us) server_negotiation.cc:462] Received NEGOTIATE request from client
1209 08:59:07.368832 (+ 327us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1209 08:59:07.369170 (+ 338us) server_negotiation.cc:227] Negotiated authn=SASL
1209 08:59:07.370145 (+ 975us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:59:07.370942 (+ 797us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1209 08:59:07.371619 (+ 677us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:59:07.371928 (+ 309us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:59:07.374402 (+ 2474us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1209 08:59:07.374429 (+ 27us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1209 08:59:07.374445 (+ 16us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1209 08:59:07.374499 (+ 54us) server_negotiation.cc:923] calling sasl_server_start()
1209 08:59:07.377385 (+ 2886us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1209 08:59:07.378015 (+ 630us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1209 08:59:07.378020 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1209 08:59:07.378025 (+ 5us) server_negotiation.cc:968] calling sasl_server_step()
1209 08:59:07.378116 (+ 91us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1209 08:59:07.378433 (+ 317us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1209 08:59:07.378438 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1209 08:59:07.378443 (+ 5us) server_negotiation.cc:968] calling sasl_server_step()
1209 08:59:07.378992 (+ 549us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1209 08:59:07.379215 (+ 223us) server_negotiation.cc:1036] Waiting for connection context
1209 08:59:07.380057 (+ 842us) server_negotiation.cc:300] Negotiation successful
1209 08:59:07.380365 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":361,"thread_start_us":157,"threads_started":1}
W20251209 08:59:07.383071 29596 catalog_manager.cc:1568] T 00000000000000000000000000000000 P 28875c92005b46f788bbde9310eb27b5: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20251209 08:59:07.383155 29596 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20251209 08:59:07.383256 29590 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251209 08:59:07.384182 29590 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251209 08:59:07.389971 29590 catalog_manager.cc:1357] Generated new cluster ID: d78cb938e88640a8b342384a6220f4c1
I20251209 08:59:07.390038 29590 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251209 08:59:07.411325 29590 catalog_manager.cc:1380] Generated new certificate authority record
I20251209 08:59:07.412298 29590 catalog_manager.cc:1514] Loading token signing keys...
I20251209 08:59:07.419023 29590 catalog_manager.cc:6027] T 00000000000000000000000000000000 P 28875c92005b46f788bbde9310eb27b5: Generated new TSK 0
I20251209 08:59:07.419878 29590 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.20.49.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.49.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.49.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765270668420570-20677-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.20.49.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765270668420570-20677-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.20.49.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.20.49.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.20.49.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765270668420570-20677-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.20.49.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765270668420570-20677-0/krb5kdc/kudu.keytab.
I20251209 08:59:07.479866 20677 external_mini_cluster.cc:1366] Running /tmp/dist-test-task8XuDZV/build/debug/bin/kudu
/tmp/dist-test-task8XuDZV/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765270668420570-20677-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765270668420570-20677-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765270668420570-20677-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765270668420570-20677-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.49.65:0
--local_ip_for_outbound_sockets=127.20.49.65
--webserver_interface=127.20.49.65
--webserver_port=0
--tserver_master_addrs=127.20.49.126:44103
--keytab_file=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765270668420570-20677-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.49.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.20.49.84:41011
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765270668420570-20677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765270668420570-20677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765270668420570-20677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251209 08:59:07.621138 29604 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251209 08:59:07.621459 29604 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251209 08:59:07.621570 29604 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251209 08:59:07.625818 29604 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251209 08:59:07.625891 29604 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251209 08:59:07.626003 29604 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.49.65
I20251209 08:59:07.630405 29604 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.49.84:41011
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765270668420570-20677-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765270668420570-20677-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765270668420570-20677-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.49.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.49.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765270668420570-20677-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.49.65
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.20.49.126:44103
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29604
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.49.65
--log_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765270668420570-20677-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 468897755969d686fb08386f42f7835685e7953a
build type DEBUG
built by None at 09 Dec 2025 08:43:15 UTC on 24a791456cd2
build id 9426
I20251209 08:59:07.631563 29604 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251209 08:59:07.632427 29604 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251209 08:59:07.639117 29612 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251209 08:59:07.639117 29610 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251209 08:59:07.639117 29609 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251209 08:59:07.639344 29604 server_base.cc:1047] running on GCE node
I20251209 08:59:07.639756 29604 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251209 08:59:07.640333 29604 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251209 08:59:07.641495 29604 hybrid_clock.cc:648] HybridClock initialized: now 1765270747641488 us; error 30 us; skew 500 ppm
Dec 09 08:59:07 dist-test-slave-8w6c krb5kdc[29492](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765270747, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.49.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251209 08:59:07.644234 29604 init.cc:377] Logged in from keytab as kudu/127.20.49.65@KRBTEST.COM (short username kudu)
I20251209 08:59:07.645409 29604 webserver.cc:492] Webserver started at http://127.20.49.65:37609/ using document root <none> and password file <none>
I20251209 08:59:07.646039 29604 fs_manager.cc:362] Metadata directory not provided
I20251209 08:59:07.646098 29604 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251209 08:59:07.646281 29604 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251209 08:59:07.648029 29604 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765270668420570-20677-0/minicluster-data/ts-0/data/instance:
uuid: "e9a726712f134b10b7d49758a1b94e5d"
format_stamp: "Formatted at 2025-12-09 08:59:07 on dist-test-slave-8w6c"
server_key: "3e45b0bed1c2fd5cd0bf3b86e688260b"
server_key_iv: "7e2222127b71da878c594f90784b0676"
server_key_version: "encryptionkey@0"
I20251209 08:59:07.648492 29604 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765270668420570-20677-0/minicluster-data/ts-0/wal/instance:
uuid: "e9a726712f134b10b7d49758a1b94e5d"
format_stamp: "Formatted at 2025-12-09 08:59:07 on dist-test-slave-8w6c"
server_key: "3e45b0bed1c2fd5cd0bf3b86e688260b"
server_key_iv: "7e2222127b71da878c594f90784b0676"
server_key_version: "encryptionkey@0"
I20251209 08:59:07.651885 29604 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20251209 08:59:07.654059 29619 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251209 08:59:07.655036 29604 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20251209 08:59:07.655140 29604 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765270668420570-20677-0/minicluster-data/ts-0/data,/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765270668420570-20677-0/minicluster-data/ts-0/wal
uuid: "e9a726712f134b10b7d49758a1b94e5d"
format_stamp: "Formatted at 2025-12-09 08:59:07 on dist-test-slave-8w6c"
server_key: "3e45b0bed1c2fd5cd0bf3b86e688260b"
server_key_iv: "7e2222127b71da878c594f90784b0676"
server_key_version: "encryptionkey@0"
I20251209 08:59:07.655225 29604 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765270668420570-20677-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765270668420570-20677-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765270668420570-20677-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251209 08:59:07.680881 29604 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251209 08:59:07.684106 29604 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251209 08:59:07.684311 29604 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251209 08:59:07.684908 29604 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251209 08:59:07.685874 29604 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251209 08:59:07.685935 29604 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251209 08:59:07.685999 29604 ts_tablet_manager.cc:616] Registered 0 tablets
I20251209 08:59:07.686033 29604 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251209 08:59:07.695449 29604 rpc_server.cc:307] RPC server started. Bound to: 127.20.49.65:40937
I20251209 08:59:07.695467 29732 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.49.65:40937 every 8 connection(s)
I20251209 08:59:07.696496 29604 server_base.cc:1179] Dumped server information to /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765270668420570-20677-0/minicluster-data/ts-0/data/info.pb
I20251209 08:59:07.697453 20677 external_mini_cluster.cc:1428] Started /tmp/dist-test-task8XuDZV/build/debug/bin/kudu as pid 29604
I20251209 08:59:07.697589 20677 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765270668420570-20677-0/minicluster-data/ts-0/wal/instance
I20251209 08:59:07.697943 20677 external_mini_cluster.cc:1468] Setting key 146f9a94fbe8d776fa9511accca20c21
Dec 09 08:59:07 dist-test-slave-8w6c krb5kdc[29492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765270747, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.49.65@KRBTEST.COM for kudu/127.20.49.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.20.49.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.49.66@KRBTEST.COM" created.
I20251209 08:59:07.711053 29579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:59:07.698630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.49.65:38007 (local address 127.20.49.126:44103)
1209 08:59:07.698789 (+ 159us) server_negotiation.cc:207] Beginning negotiation
1209 08:59:07.698794 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1209 08:59:07.699532 (+ 738us) server_negotiation.cc:408] Connection header received
1209 08:59:07.700438 (+ 906us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1209 08:59:07.700444 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
1209 08:59:07.700510 (+ 66us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1209 08:59:07.700609 (+ 99us) server_negotiation.cc:227] Negotiated authn=SASL
1209 08:59:07.702841 (+ 2232us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:59:07.703359 (+ 518us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1209 08:59:07.704338 (+ 979us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:59:07.704464 (+ 126us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:59:07.707311 (+ 2847us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1209 08:59:07.707335 (+ 24us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1209 08:59:07.707337 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1209 08:59:07.707364 (+ 27us) server_negotiation.cc:923] calling sasl_server_start()
1209 08:59:07.708856 (+ 1492us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1209 08:59:07.709396 (+ 540us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1209 08:59:07.709399 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1209 08:59:07.709400 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1209 08:59:07.709445 (+ 45us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1209 08:59:07.709781 (+ 336us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1209 08:59:07.709784 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1209 08:59:07.709786 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1209 08:59:07.710004 (+ 218us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1209 08:59:07.710103 (+ 99us) server_negotiation.cc:1036] Waiting for connection context
1209 08:59:07.710728 (+ 625us) server_negotiation.cc:300] Negotiation successful
1209 08:59:07.710920 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20251209 08:59:07.711704 29735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:59:07.698934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.49.126:44103 (local address 127.20.49.65:38007)
1209 08:59:07.699396 (+ 462us) negotiation.cc:107] Waiting for socket to connect
1209 08:59:07.699441 (+ 45us) client_negotiation.cc:174] Beginning negotiation
1209 08:59:07.700226 (+ 785us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1209 08:59:07.700787 (+ 561us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1209 08:59:07.700795 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
1209 08:59:07.701209 (+ 414us) client_negotiation.cc:189] Negotiated authn=SASL
1209 08:59:07.702626 (+ 1417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:59:07.702665 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:59:07.703504 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1209 08:59:07.703511 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1209 08:59:07.704216 (+ 705us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:59:07.704230 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:59:07.704491 (+ 261us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:59:07.705191 (+ 700us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1209 08:59:07.705242 (+ 51us) client_negotiation.cc:656] calling sasl_client_start()
1209 08:59:07.707125 (+ 1883us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1209 08:59:07.708987 (+ 1862us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1209 08:59:07.708992 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1209 08:59:07.709001 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
1209 08:59:07.709296 (+ 295us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1209 08:59:07.709532 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1209 08:59:07.709534 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1209 08:59:07.709535 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1209 08:59:07.709633 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1209 08:59:07.710139 (+ 506us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1209 08:59:07.710145 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1209 08:59:07.710392 (+ 247us) client_negotiation.cc:769] Sending connection context
1209 08:59:07.710619 (+ 227us) client_negotiation.cc:240] Negotiation successful
1209 08:59:07.710950 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":125,"threads_started":1}
I20251209 08:59:07.713022 29733 heartbeater.cc:344] Connected to a master server at 127.20.49.126:44103
I20251209 08:59:07.713271 29733 heartbeater.cc:461] Registering TS with master...
I20251209 08:59:07.713850 29733 heartbeater.cc:507] Master 127.20.49.126:44103 requested a full tablet report, sending...
I20251209 08:59:07.715339 29540 ts_manager.cc:194] Registered new tserver with Master: e9a726712f134b10b7d49758a1b94e5d (127.20.49.65:40937)
I20251209 08:59:07.716698 29540 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.49.65@KRBTEST.COM'} at 127.20.49.65:38007
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.49.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765270668420570-20677-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.20.49.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765270668420570-20677-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.20.49.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.20.49.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.20.49.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765270668420570-20677-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.20.49.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765270668420570-20677-0/krb5kdc/kudu.keytab.
I20251209 08:59:07.750183 20677 external_mini_cluster.cc:1366] Running /tmp/dist-test-task8XuDZV/build/debug/bin/kudu
/tmp/dist-test-task8XuDZV/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765270668420570-20677-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765270668420570-20677-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765270668420570-20677-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765270668420570-20677-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.49.66:0
--local_ip_for_outbound_sockets=127.20.49.66
--webserver_interface=127.20.49.66
--webserver_port=0
--tserver_master_addrs=127.20.49.126:44103
--keytab_file=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765270668420570-20677-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.49.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.20.49.84:41011
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765270668420570-20677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765270668420570-20677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765270668420570-20677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251209 08:59:07.863125 29740 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251209 08:59:07.863365 29740 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251209 08:59:07.863454 29740 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251209 08:59:07.866784 29740 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251209 08:59:07.866863 29740 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251209 08:59:07.866974 29740 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.49.66
I20251209 08:59:07.871361 29740 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.49.84:41011
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765270668420570-20677-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765270668420570-20677-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765270668420570-20677-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.49.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.49.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765270668420570-20677-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.49.66
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.20.49.126:44103
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29740
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.49.66
--log_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765270668420570-20677-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 468897755969d686fb08386f42f7835685e7953a
build type DEBUG
built by None at 09 Dec 2025 08:43:15 UTC on 24a791456cd2
build id 9426
I20251209 08:59:07.872514 29740 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251209 08:59:07.873394 29740 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251209 08:59:07.880012 29748 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251209 08:59:07.880012 29745 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251209 08:59:07.880259 29746 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251209 08:59:07.880486 29740 server_base.cc:1047] running on GCE node
I20251209 08:59:07.880849 29740 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251209 08:59:07.881397 29740 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251209 08:59:07.882547 29740 hybrid_clock.cc:648] HybridClock initialized: now 1765270747882519 us; error 48 us; skew 500 ppm
Dec 09 08:59:07 dist-test-slave-8w6c krb5kdc[29492](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765270747, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.49.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251209 08:59:07.885198 29740 init.cc:377] Logged in from keytab as kudu/127.20.49.66@KRBTEST.COM (short username kudu)
I20251209 08:59:07.886339 29740 webserver.cc:492] Webserver started at http://127.20.49.66:42547/ using document root <none> and password file <none>
I20251209 08:59:07.886886 29740 fs_manager.cc:362] Metadata directory not provided
I20251209 08:59:07.886950 29740 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251209 08:59:07.887130 29740 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251209 08:59:07.888823 29740 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765270668420570-20677-0/minicluster-data/ts-1/data/instance:
uuid: "e1df360180a247ce867f5c22eaf9cfb4"
format_stamp: "Formatted at 2025-12-09 08:59:07 on dist-test-slave-8w6c"
server_key: "e0fa2f988bc92fd80ee260638beb70e1"
server_key_iv: "f3cc25b37281be03451eb190c9dc3ad9"
server_key_version: "encryptionkey@0"
I20251209 08:59:07.889307 29740 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765270668420570-20677-0/minicluster-data/ts-1/wal/instance:
uuid: "e1df360180a247ce867f5c22eaf9cfb4"
format_stamp: "Formatted at 2025-12-09 08:59:07 on dist-test-slave-8w6c"
server_key: "e0fa2f988bc92fd80ee260638beb70e1"
server_key_iv: "f3cc25b37281be03451eb190c9dc3ad9"
server_key_version: "encryptionkey@0"
I20251209 08:59:07.892683 29740 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.003s
I20251209 08:59:07.895015 29755 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251209 08:59:07.896073 29740 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20251209 08:59:07.896176 29740 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765270668420570-20677-0/minicluster-data/ts-1/data,/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765270668420570-20677-0/minicluster-data/ts-1/wal
uuid: "e1df360180a247ce867f5c22eaf9cfb4"
format_stamp: "Formatted at 2025-12-09 08:59:07 on dist-test-slave-8w6c"
server_key: "e0fa2f988bc92fd80ee260638beb70e1"
server_key_iv: "f3cc25b37281be03451eb190c9dc3ad9"
server_key_version: "encryptionkey@0"
I20251209 08:59:07.896268 29740 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765270668420570-20677-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765270668420570-20677-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765270668420570-20677-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251209 08:59:07.906805 29740 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251209 08:59:07.913115 29740 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251209 08:59:07.913318 29740 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251209 08:59:07.913962 29740 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251209 08:59:07.914877 29740 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251209 08:59:07.914939 29740 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251209 08:59:07.915000 29740 ts_tablet_manager.cc:616] Registered 0 tablets
I20251209 08:59:07.915028 29740 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251209 08:59:07.924938 29740 rpc_server.cc:307] RPC server started. Bound to: 127.20.49.66:44559
I20251209 08:59:07.924954 29868 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.49.66:44559 every 8 connection(s)
I20251209 08:59:07.926023 29740 server_base.cc:1179] Dumped server information to /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765270668420570-20677-0/minicluster-data/ts-1/data/info.pb
I20251209 08:59:07.926528 20677 external_mini_cluster.cc:1428] Started /tmp/dist-test-task8XuDZV/build/debug/bin/kudu as pid 29740
I20251209 08:59:07.926640 20677 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765270668420570-20677-0/minicluster-data/ts-1/wal/instance
I20251209 08:59:07.926923 20677 external_mini_cluster.cc:1468] Setting key cad005b2a1e305f224c84a49a1c15acb
Dec 09 08:59:07 dist-test-slave-8w6c krb5kdc[29492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765270747, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.49.66@KRBTEST.COM for kudu/127.20.49.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.20.49.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.49.67@KRBTEST.COM" created.
I20251209 08:59:07.940968 29579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:59:07.927848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.49.66:41841 (local address 127.20.49.126:44103)
1209 08:59:07.927969 (+ 121us) server_negotiation.cc:207] Beginning negotiation
1209 08:59:07.927973 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1209 08:59:07.928863 (+ 890us) server_negotiation.cc:408] Connection header received
1209 08:59:07.930168 (+ 1305us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1209 08:59:07.930170 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1209 08:59:07.930219 (+ 49us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1209 08:59:07.930297 (+ 78us) server_negotiation.cc:227] Negotiated authn=SASL
1209 08:59:07.932410 (+ 2113us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:59:07.932988 (+ 578us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1209 08:59:07.934010 (+ 1022us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:59:07.934159 (+ 149us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:59:07.936901 (+ 2742us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1209 08:59:07.936925 (+ 24us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1209 08:59:07.936927 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1209 08:59:07.936956 (+ 29us) server_negotiation.cc:923] calling sasl_server_start()
1209 08:59:07.938553 (+ 1597us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1209 08:59:07.939222 (+ 669us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1209 08:59:07.939226 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1209 08:59:07.939228 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1209 08:59:07.939294 (+ 66us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1209 08:59:07.939626 (+ 332us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1209 08:59:07.939632 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1209 08:59:07.939636 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
1209 08:59:07.939897 (+ 261us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1209 08:59:07.939996 (+ 99us) server_negotiation.cc:1036] Waiting for connection context
1209 08:59:07.940694 (+ 698us) server_negotiation.cc:300] Negotiation successful
1209 08:59:07.940822 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20251209 08:59:07.941992 29871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:59:07.928243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.49.126:44103 (local address 127.20.49.66:41841)
1209 08:59:07.928716 (+ 473us) negotiation.cc:107] Waiting for socket to connect
1209 08:59:07.928768 (+ 52us) client_negotiation.cc:174] Beginning negotiation
1209 08:59:07.929929 (+ 1161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1209 08:59:07.930505 (+ 576us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1209 08:59:07.930519 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
1209 08:59:07.930971 (+ 452us) client_negotiation.cc:189] Negotiated authn=SASL
1209 08:59:07.932235 (+ 1264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:59:07.932257 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:59:07.933131 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1209 08:59:07.933138 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1209 08:59:07.933871 (+ 733us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:59:07.933885 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:59:07.934163 (+ 278us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:59:07.934873 (+ 710us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1209 08:59:07.934924 (+ 51us) client_negotiation.cc:656] calling sasl_client_start()
1209 08:59:07.936718 (+ 1794us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1209 08:59:07.938718 (+ 2000us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1209 08:59:07.938724 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1209 08:59:07.938738 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
1209 08:59:07.939081 (+ 343us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1209 08:59:07.939393 (+ 312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1209 08:59:07.939396 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1209 08:59:07.939397 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1209 08:59:07.939504 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1209 08:59:07.940024 (+ 520us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1209 08:59:07.940032 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1209 08:59:07.940373 (+ 341us) client_negotiation.cc:769] Sending connection context
1209 08:59:07.940683 (+ 310us) client_negotiation.cc:240] Negotiation successful
1209 08:59:07.941019 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":116,"threads_started":1}
I20251209 08:59:07.943367 29869 heartbeater.cc:344] Connected to a master server at 127.20.49.126:44103
I20251209 08:59:07.943631 29869 heartbeater.cc:461] Registering TS with master...
I20251209 08:59:07.944206 29869 heartbeater.cc:507] Master 127.20.49.126:44103 requested a full tablet report, sending...
I20251209 08:59:07.945310 29540 ts_manager.cc:194] Registered new tserver with Master: e1df360180a247ce867f5c22eaf9cfb4 (127.20.49.66:44559)
I20251209 08:59:07.946004 29540 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.49.66@KRBTEST.COM'} at 127.20.49.66:41841
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.49.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765270668420570-20677-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.20.49.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765270668420570-20677-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.20.49.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.20.49.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.20.49.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765270668420570-20677-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.20.49.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765270668420570-20677-0/krb5kdc/kudu.keytab.
I20251209 08:59:07.980317 20677 external_mini_cluster.cc:1366] Running /tmp/dist-test-task8XuDZV/build/debug/bin/kudu
/tmp/dist-test-task8XuDZV/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765270668420570-20677-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765270668420570-20677-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765270668420570-20677-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765270668420570-20677-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.49.67:0
--local_ip_for_outbound_sockets=127.20.49.67
--webserver_interface=127.20.49.67
--webserver_port=0
--tserver_master_addrs=127.20.49.126:44103
--keytab_file=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765270668420570-20677-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.49.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.20.49.84:41011
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765270668420570-20677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765270668420570-20677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765270668420570-20677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251209 08:59:08.097715 29876 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251209 08:59:08.098053 29876 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251209 08:59:08.098186 29876 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251209 08:59:08.104740 29876 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251209 08:59:08.104863 29876 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251209 08:59:08.105058 29876 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.49.67
I20251209 08:59:08.113482 29876 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.49.84:41011
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765270668420570-20677-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765270668420570-20677-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765270668420570-20677-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.49.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.49.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765270668420570-20677-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.49.67
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.20.49.126:44103
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29876
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.49.67
--log_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765270668420570-20677-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 468897755969d686fb08386f42f7835685e7953a
build type DEBUG
built by None at 09 Dec 2025 08:43:15 UTC on 24a791456cd2
build id 9426
I20251209 08:59:08.115206 29876 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251209 08:59:08.116500 29876 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251209 08:59:08.125918 29882 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251209 08:59:08.125975 29881 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251209 08:59:08.126220 29876 server_base.cc:1047] running on GCE node
W20251209 08:59:08.125885 29884 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251209 08:59:08.126685 29876 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251209 08:59:08.127296 29876 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251209 08:59:08.128471 29876 hybrid_clock.cc:648] HybridClock initialized: now 1765270748128452 us; error 44 us; skew 500 ppm
Dec 09 08:59:08 dist-test-slave-8w6c krb5kdc[29492](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765270748, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.49.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251209 08:59:08.131371 29876 init.cc:377] Logged in from keytab as kudu/127.20.49.67@KRBTEST.COM (short username kudu)
I20251209 08:59:08.132493 29876 webserver.cc:492] Webserver started at http://127.20.49.67:38479/ using document root <none> and password file <none>
I20251209 08:59:08.133080 29876 fs_manager.cc:362] Metadata directory not provided
I20251209 08:59:08.133133 29876 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251209 08:59:08.133328 29876 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251209 08:59:08.135190 29876 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765270668420570-20677-0/minicluster-data/ts-2/data/instance:
uuid: "6d524d7b0cbb4204ab6d9e9204d5da91"
format_stamp: "Formatted at 2025-12-09 08:59:08 on dist-test-slave-8w6c"
server_key: "303364481f2d019926121332e5e10439"
server_key_iv: "c4e68bc00f1fcb9d16cf0ba1ebcfd935"
server_key_version: "encryptionkey@0"
I20251209 08:59:08.135692 29876 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765270668420570-20677-0/minicluster-data/ts-2/wal/instance:
uuid: "6d524d7b0cbb4204ab6d9e9204d5da91"
format_stamp: "Formatted at 2025-12-09 08:59:08 on dist-test-slave-8w6c"
server_key: "303364481f2d019926121332e5e10439"
server_key_iv: "c4e68bc00f1fcb9d16cf0ba1ebcfd935"
server_key_version: "encryptionkey@0"
I20251209 08:59:08.139557 29876 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20251209 08:59:08.141980 29891 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251209 08:59:08.143157 29876 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20251209 08:59:08.143272 29876 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765270668420570-20677-0/minicluster-data/ts-2/data,/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765270668420570-20677-0/minicluster-data/ts-2/wal
uuid: "6d524d7b0cbb4204ab6d9e9204d5da91"
format_stamp: "Formatted at 2025-12-09 08:59:08 on dist-test-slave-8w6c"
server_key: "303364481f2d019926121332e5e10439"
server_key_iv: "c4e68bc00f1fcb9d16cf0ba1ebcfd935"
server_key_version: "encryptionkey@0"
I20251209 08:59:08.143369 29876 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765270668420570-20677-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765270668420570-20677-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765270668420570-20677-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251209 08:59:08.154280 29876 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251209 08:59:08.157481 29876 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251209 08:59:08.157675 29876 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251209 08:59:08.158339 29876 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251209 08:59:08.159332 29876 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251209 08:59:08.159387 29876 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251209 08:59:08.159448 29876 ts_tablet_manager.cc:616] Registered 0 tablets
I20251209 08:59:08.159488 29876 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251209 08:59:08.169956 29876 rpc_server.cc:307] RPC server started. Bound to: 127.20.49.67:40601
I20251209 08:59:08.170017 30004 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.49.67:40601 every 8 connection(s)
I20251209 08:59:08.170950 29876 server_base.cc:1179] Dumped server information to /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765270668420570-20677-0/minicluster-data/ts-2/data/info.pb
I20251209 08:59:08.177148 20677 external_mini_cluster.cc:1428] Started /tmp/dist-test-task8XuDZV/build/debug/bin/kudu as pid 29876
I20251209 08:59:08.177280 20677 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765270668420570-20677-0/minicluster-data/ts-2/wal/instance
I20251209 08:59:08.177531 20677 external_mini_cluster.cc:1468] Setting key 1a194e6235072bb30c383918cfcb2e13
Dec 09 08:59:08 dist-test-slave-8w6c krb5kdc[29492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765270748, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.49.67@KRBTEST.COM for kudu/127.20.49.126@KRBTEST.COM
I20251209 08:59:08.186422 29579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:59:08.172992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.49.67:51449 (local address 127.20.49.126:44103)
1209 08:59:08.173205 (+ 213us) server_negotiation.cc:207] Beginning negotiation
1209 08:59:08.173212 (+ 7us) server_negotiation.cc:400] Waiting for connection header
1209 08:59:08.173967 (+ 755us) server_negotiation.cc:408] Connection header received
1209 08:59:08.175265 (+ 1298us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1209 08:59:08.175268 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1209 08:59:08.175327 (+ 59us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1209 08:59:08.175420 (+ 93us) server_negotiation.cc:227] Negotiated authn=SASL
1209 08:59:08.177986 (+ 2566us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:59:08.178699 (+ 713us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1209 08:59:08.179429 (+ 730us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:59:08.179617 (+ 188us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:59:08.182229 (+ 2612us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1209 08:59:08.182246 (+ 17us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1209 08:59:08.182249 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1209 08:59:08.182277 (+ 28us) server_negotiation.cc:923] calling sasl_server_start()
1209 08:59:08.184040 (+ 1763us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1209 08:59:08.184725 (+ 685us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1209 08:59:08.184728 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1209 08:59:08.184730 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1209 08:59:08.184792 (+ 62us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1209 08:59:08.185144 (+ 352us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1209 08:59:08.185146 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1209 08:59:08.185148 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1209 08:59:08.185321 (+ 173us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1209 08:59:08.185405 (+ 84us) server_negotiation.cc:1036] Waiting for connection context
1209 08:59:08.186171 (+ 766us) server_negotiation.cc:300] Negotiation successful
1209 08:59:08.186295 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20251209 08:59:08.187415 30007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:59:08.173229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.49.126:44103 (local address 127.20.49.67:51449)
1209 08:59:08.173791 (+ 562us) negotiation.cc:107] Waiting for socket to connect
1209 08:59:08.173852 (+ 61us) client_negotiation.cc:174] Beginning negotiation
1209 08:59:08.175007 (+ 1155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1209 08:59:08.175625 (+ 618us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1209 08:59:08.175639 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
1209 08:59:08.176283 (+ 644us) client_negotiation.cc:189] Negotiated authn=SASL
1209 08:59:08.177768 (+ 1485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:59:08.177793 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:59:08.178830 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1209 08:59:08.178842 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1209 08:59:08.179320 (+ 478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:59:08.179330 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:59:08.179557 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:59:08.180224 (+ 667us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1209 08:59:08.180249 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
1209 08:59:08.182006 (+ 1757us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1209 08:59:08.184173 (+ 2167us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1209 08:59:08.184182 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1209 08:59:08.184199 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
1209 08:59:08.184581 (+ 382us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1209 08:59:08.184887 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1209 08:59:08.184893 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1209 08:59:08.184897 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
1209 08:59:08.185042 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1209 08:59:08.185424 (+ 382us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1209 08:59:08.185434 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1209 08:59:08.185820 (+ 386us) client_negotiation.cc:769] Sending connection context
1209 08:59:08.186098 (+ 278us) client_negotiation.cc:240] Negotiation successful
1209 08:59:08.186433 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":112,"threads_started":1}
I20251209 08:59:08.188694 30005 heartbeater.cc:344] Connected to a master server at 127.20.49.126:44103
I20251209 08:59:08.188957 30005 heartbeater.cc:461] Registering TS with master...
I20251209 08:59:08.189476 30005 heartbeater.cc:507] Master 127.20.49.126:44103 requested a full tablet report, sending...
I20251209 08:59:08.190658 29540 ts_manager.cc:194] Registered new tserver with Master: 6d524d7b0cbb4204ab6d9e9204d5da91 (127.20.49.67:40601)
I20251209 08:59:08.191401 29540 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.49.67@KRBTEST.COM'} at 127.20.49.67:51449
I20251209 08:59:08.191906 20677 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.20.49.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.20.49.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.20.49.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765270668420570-20677-0/krb5kdc/oryx_127.20.49.65.keytab.
Entry for principal oryx/127.20.49.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765270668420570-20677-0/krb5kdc/oryx_127.20.49.65.keytab.
WARNING: no policy specified for oryx/127.20.49.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.20.49.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.20.49.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765270668420570-20677-0/krb5kdc/oryx_127.20.49.66.keytab.
Entry for principal oryx/127.20.49.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765270668420570-20677-0/krb5kdc/oryx_127.20.49.66.keytab.
WARNING: no policy specified for oryx/127.20.49.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.20.49.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.20.49.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765270668420570-20677-0/krb5kdc/oryx_127.20.49.67.keytab.
Entry for principal oryx/127.20.49.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765270668420570-20677-0/krb5kdc/oryx_127.20.49.67.keytab.
I20251209 08:59:08.263033 20677 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task8XuDZV/build/debug/bin/kudu with pid 29604
I20251209 08:59:08.269461 20677 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task8XuDZV/build/debug/bin/kudu with pid 29740
I20251209 08:59:08.275111 20677 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task8XuDZV/build/debug/bin/kudu with pid 29876
I20251209 08:59:08.280788 20677 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task8XuDZV/build/debug/bin/kudu with pid 29508
I20251209 08:59:08.288151 20677 external_mini_cluster.cc:1366] Running /tmp/dist-test-task8XuDZV/build/debug/bin/kudu
/tmp/dist-test-task8XuDZV/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765270668420570-20677-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765270668420570-20677-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765270668420570-20677-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765270668420570-20677-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.49.126:44103
--webserver_interface=127.20.49.126
--webserver_port=41583
--keytab_file=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765270668420570-20677-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.49.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.20.49.84:41011
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.49.126:44103
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765270668420570-20677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765270668420570-20677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765270668420570-20677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251209 08:59:08.399427 30014 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251209 08:59:08.399667 30014 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251209 08:59:08.399748 30014 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251209 08:59:08.403151 30014 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251209 08:59:08.403234 30014 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251209 08:59:08.403259 30014 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251209 08:59:08.403278 30014 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251209 08:59:08.403297 30014 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251209 08:59:08.407604 30014 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.49.84:41011
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765270668420570-20677-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765270668420570-20677-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.49.126:44103
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765270668420570-20677-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.49.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.49.126:44103
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765270668420570-20677-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.49.126
--webserver_port=41583
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30014
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765270668420570-20677-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 468897755969d686fb08386f42f7835685e7953a
build type DEBUG
built by None at 09 Dec 2025 08:43:15 UTC on 24a791456cd2
build id 9426
I20251209 08:59:08.408686 30014 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251209 08:59:08.409499 30014 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251209 08:59:08.415150 30020 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251209 08:59:08.415150 30022 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251209 08:59:08.415149 30019 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251209 08:59:08.415367 30014 server_base.cc:1047] running on GCE node
I20251209 08:59:08.416291 30014 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251209 08:59:08.417259 30014 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251209 08:59:08.418411 30014 hybrid_clock.cc:648] HybridClock initialized: now 1765270748418392 us; error 30 us; skew 500 ppm
Dec 09 08:59:08 dist-test-slave-8w6c krb5kdc[29492](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765270748, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.49.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251209 08:59:08.422322 30014 init.cc:377] Logged in from keytab as kudu/127.20.49.126@KRBTEST.COM (short username kudu)
I20251209 08:59:08.423823 30014 webserver.cc:492] Webserver started at http://127.20.49.126:41583/ using document root <none> and password file <none>
I20251209 08:59:08.424574 30014 fs_manager.cc:362] Metadata directory not provided
I20251209 08:59:08.424669 30014 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251209 08:59:08.429056 30014 fs_manager.cc:714] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20251209 08:59:08.431140 30029 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251209 08:59:08.432168 30014 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20251209 08:59:08.432278 30014 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765270668420570-20677-0/minicluster-data/master-0/data,/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765270668420570-20677-0/minicluster-data/master-0/wal
uuid: "28875c92005b46f788bbde9310eb27b5"
format_stamp: "Formatted at 2025-12-09 08:59:07 on dist-test-slave-8w6c"
server_key: "8d592e39650c3efb8056dc9eac6aa69a"
server_key_iv: "2e3135fd11787e11b5fe0a7f6a35cd02"
server_key_version: "encryptionkey@0"
I20251209 08:59:08.432709 30014 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765270668420570-20677-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765270668420570-20677-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765270668420570-20677-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251209 08:59:08.451435 30014 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251209 08:59:08.454818 30014 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251209 08:59:08.455025 30014 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251209 08:59:08.462713 30081 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.49.126:44103 every 8 connection(s)
I20251209 08:59:08.462710 30014 rpc_server.cc:307] RPC server started. Bound to: 127.20.49.126:44103
I20251209 08:59:08.463835 30014 server_base.cc:1179] Dumped server information to /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765270668420570-20677-0/minicluster-data/master-0/data/info.pb
I20251209 08:59:08.464533 20677 external_mini_cluster.cc:1428] Started /tmp/dist-test-task8XuDZV/build/debug/bin/kudu as pid 30014
I20251209 08:59:08.465051 20677 external_mini_cluster.cc:1366] Running /tmp/dist-test-task8XuDZV/build/debug/bin/kudu
/tmp/dist-test-task8XuDZV/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765270668420570-20677-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765270668420570-20677-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765270668420570-20677-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765270668420570-20677-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.49.65:40937
--local_ip_for_outbound_sockets=127.20.49.65
--tserver_master_addrs=127.20.49.126:44103
--webserver_port=37609
--webserver_interface=127.20.49.65
--keytab_file=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765270668420570-20677-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.49.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.20.49.84:41011
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.20.49.65
--keytab_file=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765270668420570-20677-0/krb5kdc/oryx_127.20.49.65.keytab with env {KRB5CCNAME=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765270668420570-20677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765270668420570-20677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765270668420570-20677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20251209 08:59:08.469396 30082 sys_catalog.cc:263] Verifying existing consensus state
I20251209 08:59:08.471071 30082 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 28875c92005b46f788bbde9310eb27b5: Bootstrap starting.
I20251209 08:59:08.487462 30082 log.cc:826] T 00000000000000000000000000000000 P 28875c92005b46f788bbde9310eb27b5: Log is configured to *not* fsync() on all Append() calls
I20251209 08:59:08.493645 30082 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 28875c92005b46f788bbde9310eb27b5: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251209 08:59:08.494081 30082 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 28875c92005b46f788bbde9310eb27b5: Bootstrap complete.
I20251209 08:59:08.497579 30082 raft_consensus.cc:359] T 00000000000000000000000000000000 P 28875c92005b46f788bbde9310eb27b5 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28875c92005b46f788bbde9310eb27b5" member_type: VOTER last_known_addr { host: "127.20.49.126" port: 44103 } }
I20251209 08:59:08.497961 30082 raft_consensus.cc:740] T 00000000000000000000000000000000 P 28875c92005b46f788bbde9310eb27b5 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28875c92005b46f788bbde9310eb27b5, State: Initialized, Role: FOLLOWER
I20251209 08:59:08.498442 30082 consensus_queue.cc:260] T 00000000000000000000000000000000 P 28875c92005b46f788bbde9310eb27b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28875c92005b46f788bbde9310eb27b5" member_type: VOTER last_known_addr { host: "127.20.49.126" port: 44103 } }
I20251209 08:59:08.498570 30082 raft_consensus.cc:399] T 00000000000000000000000000000000 P 28875c92005b46f788bbde9310eb27b5 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251209 08:59:08.498658 30082 raft_consensus.cc:493] T 00000000000000000000000000000000 P 28875c92005b46f788bbde9310eb27b5 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251209 08:59:08.498752 30082 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 28875c92005b46f788bbde9310eb27b5 [term 1 FOLLOWER]: Advancing to term 2
I20251209 08:59:08.500274 30082 raft_consensus.cc:515] T 00000000000000000000000000000000 P 28875c92005b46f788bbde9310eb27b5 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28875c92005b46f788bbde9310eb27b5" member_type: VOTER last_known_addr { host: "127.20.49.126" port: 44103 } }
I20251209 08:59:08.500602 30082 leader_election.cc:304] T 00000000000000000000000000000000 P 28875c92005b46f788bbde9310eb27b5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 28875c92005b46f788bbde9310eb27b5; no voters:
I20251209 08:59:08.500891 30082 leader_election.cc:290] T 00000000000000000000000000000000 P 28875c92005b46f788bbde9310eb27b5 [CANDIDATE]: Term 2 election: Requested vote from peers
I20251209 08:59:08.501036 30086 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 28875c92005b46f788bbde9310eb27b5 [term 2 FOLLOWER]: Leader election won for term 2
I20251209 08:59:08.501269 30086 raft_consensus.cc:697] T 00000000000000000000000000000000 P 28875c92005b46f788bbde9310eb27b5 [term 2 LEADER]: Becoming Leader. State: Replica: 28875c92005b46f788bbde9310eb27b5, State: Running, Role: LEADER
I20251209 08:59:08.501556 30086 consensus_queue.cc:237] T 00000000000000000000000000000000 P 28875c92005b46f788bbde9310eb27b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28875c92005b46f788bbde9310eb27b5" member_type: VOTER last_known_addr { host: "127.20.49.126" port: 44103 } }
I20251209 08:59:08.502137 30082 sys_catalog.cc:565] T 00000000000000000000000000000000 P 28875c92005b46f788bbde9310eb27b5 [sys.catalog]: configured and running, proceeding with master startup.
I20251209 08:59:08.502491 30088 sys_catalog.cc:455] T 00000000000000000000000000000000 P 28875c92005b46f788bbde9310eb27b5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 28875c92005b46f788bbde9310eb27b5. Latest consensus state: current_term: 2 leader_uuid: "28875c92005b46f788bbde9310eb27b5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28875c92005b46f788bbde9310eb27b5" member_type: VOTER last_known_addr { host: "127.20.49.126" port: 44103 } } }
I20251209 08:59:08.502590 30088 sys_catalog.cc:458] T 00000000000000000000000000000000 P 28875c92005b46f788bbde9310eb27b5 [sys.catalog]: This master's current role is: LEADER
I20251209 08:59:08.502849 30087 sys_catalog.cc:455] T 00000000000000000000000000000000 P 28875c92005b46f788bbde9310eb27b5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "28875c92005b46f788bbde9310eb27b5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28875c92005b46f788bbde9310eb27b5" member_type: VOTER last_known_addr { host: "127.20.49.126" port: 44103 } } }
I20251209 08:59:08.502966 30087 sys_catalog.cc:458] T 00000000000000000000000000000000 P 28875c92005b46f788bbde9310eb27b5 [sys.catalog]: This master's current role is: LEADER
I20251209 08:59:08.504158 30090 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251209 08:59:08.507252 30090 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251209 08:59:08.507884 30090 catalog_manager.cc:1269] Loaded cluster ID: d78cb938e88640a8b342384a6220f4c1
I20251209 08:59:08.507948 30090 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251209 08:59:08.510499 30090 catalog_manager.cc:1514] Loading token signing keys...
I20251209 08:59:08.511415 30090 catalog_manager.cc:6038] T 00000000000000000000000000000000 P 28875c92005b46f788bbde9310eb27b5: Loaded TSK: 0
I20251209 08:59:08.512749 30090 catalog_manager.cc:1524] Initializing in-progress tserver states...
W20251209 08:59:08.579101 30084 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251209 08:59:08.579340 30084 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251209 08:59:08.579409 30084 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251209 08:59:08.582859 30084 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251209 08:59:08.582942 30084 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251209 08:59:08.583045 30084 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.49.65
I20251209 08:59:08.587417 30084 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.49.84:41011
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765270668420570-20677-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765270668420570-20677-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765270668420570-20677-0/krb5kdc/oryx_127.20.49.65.keytab
--principal=oryx/127.20.49.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.49.65:40937
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765270668420570-20677-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.49.65
--webserver_port=37609
--webserver_require_spnego=true
--tserver_master_addrs=127.20.49.126:44103
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30084
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.49.65
--log_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765270668420570-20677-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 468897755969d686fb08386f42f7835685e7953a
build type DEBUG
built by None at 09 Dec 2025 08:43:15 UTC on 24a791456cd2
build id 9426
I20251209 08:59:08.588519 30084 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251209 08:59:08.589371 30084 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251209 08:59:08.595979 30109 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251209 08:59:08.595957 30108 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251209 08:59:08.595952 30111 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251209 08:59:08.596278 30084 server_base.cc:1047] running on GCE node
I20251209 08:59:08.596683 30084 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251209 08:59:08.597313 30084 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251209 08:59:08.598491 30084 hybrid_clock.cc:648] HybridClock initialized: now 1765270748598473 us; error 30 us; skew 500 ppm
Dec 09 08:59:08 dist-test-slave-8w6c krb5kdc[29492](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765270748, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.49.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251209 08:59:08.601778 30084 init.cc:377] Logged in from keytab as oryx/127.20.49.65@KRBTEST.COM (short username oryx)
I20251209 08:59:08.602838 30084 webserver.cc:492] Webserver started at http://127.20.49.65:37609/ using document root <none> and password file <none>
I20251209 08:59:08.603488 30084 fs_manager.cc:362] Metadata directory not provided
I20251209 08:59:08.603560 30084 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251209 08:59:08.607287 30084 fs_manager.cc:714] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20251209 08:59:08.609318 30118 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251209 08:59:08.610455 30084 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20251209 08:59:08.610569 30084 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765270668420570-20677-0/minicluster-data/ts-0/data,/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765270668420570-20677-0/minicluster-data/ts-0/wal
uuid: "e9a726712f134b10b7d49758a1b94e5d"
format_stamp: "Formatted at 2025-12-09 08:59:07 on dist-test-slave-8w6c"
server_key: "3e45b0bed1c2fd5cd0bf3b86e688260b"
server_key_iv: "7e2222127b71da878c594f90784b0676"
server_key_version: "encryptionkey@0"
I20251209 08:59:08.611044 30084 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765270668420570-20677-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765270668420570-20677-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765270668420570-20677-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251209 08:59:08.639590 30084 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251209 08:59:08.642803 30084 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251209 08:59:08.642995 30084 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251209 08:59:08.643563 30084 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251209 08:59:08.644505 30084 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251209 08:59:08.644557 30084 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251209 08:59:08.644620 30084 ts_tablet_manager.cc:616] Registered 0 tablets
I20251209 08:59:08.644649 30084 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251209 08:59:08.655478 30084 rpc_server.cc:307] RPC server started. Bound to: 127.20.49.65:40937
I20251209 08:59:08.655524 30231 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.49.65:40937 every 8 connection(s)
I20251209 08:59:08.656551 30084 server_base.cc:1179] Dumped server information to /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765270668420570-20677-0/minicluster-data/ts-0/data/info.pb
I20251209 08:59:08.661939 20677 external_mini_cluster.cc:1428] Started /tmp/dist-test-task8XuDZV/build/debug/bin/kudu as pid 30084
I20251209 08:59:08.662400 20677 external_mini_cluster.cc:1366] Running /tmp/dist-test-task8XuDZV/build/debug/bin/kudu
/tmp/dist-test-task8XuDZV/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765270668420570-20677-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765270668420570-20677-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765270668420570-20677-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765270668420570-20677-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.49.66:44559
--local_ip_for_outbound_sockets=127.20.49.66
--tserver_master_addrs=127.20.49.126:44103
--webserver_port=42547
--webserver_interface=127.20.49.66
--keytab_file=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765270668420570-20677-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.49.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.20.49.84:41011
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.20.49.66
--keytab_file=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765270668420570-20677-0/krb5kdc/oryx_127.20.49.66.keytab with env {KRB5CCNAME=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765270668420570-20677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765270668420570-20677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765270668420570-20677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Dec 09 08:59:08 dist-test-slave-8w6c krb5kdc[29492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.49.65@KRBTEST.COM for oryx/127.20.49.126@KRBTEST.COM, Server not found in Kerberos database
Dec 09 08:59:08 dist-test-slave-8w6c krb5kdc[29492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.49.65@KRBTEST.COM for oryx/127.20.49.126@KRBTEST.COM, Server not found in Kerberos database
I20251209 08:59:08.669868 30235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:59:08.658913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.49.126:44103 (local address 127.20.49.65:35827)
1209 08:59:08.659694 (+ 781us) negotiation.cc:107] Waiting for socket to connect
1209 08:59:08.659737 (+ 43us) client_negotiation.cc:174] Beginning negotiation
1209 08:59:08.660699 (+ 962us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1209 08:59:08.661919 (+ 1220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1209 08:59:08.661933 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
1209 08:59:08.662492 (+ 559us) client_negotiation.cc:189] Negotiated authn=SASL
1209 08:59:08.663845 (+ 1353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:59:08.663866 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:59:08.665318 (+ 1452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1209 08:59:08.665323 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1209 08:59:08.665804 (+ 481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:59:08.665813 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:59:08.666067 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:59:08.666773 (+ 706us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1209 08:59:08.666801 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
1209 08:59:08.669031 (+ 2230us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.49.126:44103: Server oryx/127.20.49.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":570,"thread_start_us":135,"threads_started":1}
W20251209 08:59:08.669864 30234 negotiation.cc:337] Failed RPC negotiation. Trace:
1209 08:59:08.658902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.49.65:35827 (local address 127.20.49.126:44103)
1209 08:59:08.659373 (+ 471us) server_negotiation.cc:207] Beginning negotiation
1209 08:59:08.659386 (+ 13us) server_negotiation.cc:400] Waiting for connection header
1209 08:59:08.659866 (+ 480us) server_negotiation.cc:408] Connection header received
1209 08:59:08.661105 (+ 1239us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1209 08:59:08.661116 (+ 11us) server_negotiation.cc:462] Received NEGOTIATE request from client
1209 08:59:08.661385 (+ 269us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1209 08:59:08.661729 (+ 344us) server_negotiation.cc:227] Negotiated authn=SASL
1209 08:59:08.664111 (+ 2382us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:59:08.665173 (+ 1062us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1209 08:59:08.665926 (+ 753us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:59:08.666172 (+ 246us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:59:08.669228 (+ 3056us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.49.65:35827: BlockingRecv error: recv got EOF from 127.20.49.65:35827 (error 108)
Metrics: {"server-negotiator.queue_time_us":306,"thread_start_us":125,"threads_started":1}
W20251209 08:59:08.671100 30232 heartbeater.cc:646] Failed to heartbeat to 127.20.49.126:44103 (0 consecutive failures): Not authorized: Failed to ping master at 127.20.49.126:44103: Client connection negotiation failed: client connection to 127.20.49.126:44103: Server oryx/127.20.49.126@KRBTEST.COM not found in Kerberos database
Dec 09 08:59:08 dist-test-slave-8w6c krb5kdc[29492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.49.65@KRBTEST.COM for oryx/127.20.49.126@KRBTEST.COM, Server not found in Kerberos database
Dec 09 08:59:08 dist-test-slave-8w6c krb5kdc[29492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.49.65@KRBTEST.COM for oryx/127.20.49.126@KRBTEST.COM, Server not found in Kerberos database
I20251209 08:59:08.676066 30235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:59:08.671553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.49.126:44103 (local address 127.20.49.65:33031)
1209 08:59:08.671742 (+ 189us) negotiation.cc:107] Waiting for socket to connect
1209 08:59:08.671756 (+ 14us) client_negotiation.cc:174] Beginning negotiation
1209 08:59:08.671843 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1209 08:59:08.672086 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1209 08:59:08.672089 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1209 08:59:08.672244 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
1209 08:59:08.672435 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:59:08.672441 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:59:08.673152 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1209 08:59:08.673155 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1209 08:59:08.673520 (+ 365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:59:08.673526 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:59:08.673632 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:59:08.674202 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1209 08:59:08.674219 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
1209 08:59:08.675896 (+ 1677us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.49.126:44103: Server oryx/127.20.49.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":109}
W20251209 08:59:08.676139 30234 negotiation.cc:337] Failed RPC negotiation. Trace:
1209 08:59:08.671622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.49.65:33031 (local address 127.20.49.126:44103)
1209 08:59:08.671788 (+ 166us) server_negotiation.cc:207] Beginning negotiation
1209 08:59:08.671794 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1209 08:59:08.671810 (+ 16us) server_negotiation.cc:408] Connection header received
1209 08:59:08.671938 (+ 128us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1209 08:59:08.671941 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1209 08:59:08.671986 (+ 45us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1209 08:59:08.672048 (+ 62us) server_negotiation.cc:227] Negotiated authn=SASL
1209 08:59:08.672551 (+ 503us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:59:08.673044 (+ 493us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1209 08:59:08.673626 (+ 582us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:59:08.674225 (+ 599us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:59:08.676029 (+ 1804us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.49.65:33031: BlockingRecv error: recv got EOF from 127.20.49.65:33031 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Dec 09 08:59:08 dist-test-slave-8w6c krb5kdc[29492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.49.65@KRBTEST.COM for oryx/127.20.49.126@KRBTEST.COM, Server not found in Kerberos database
Dec 09 08:59:08 dist-test-slave-8w6c krb5kdc[29492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.49.65@KRBTEST.COM for oryx/127.20.49.126@KRBTEST.COM, Server not found in Kerberos database
I20251209 08:59:08.681129 30235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:59:08.676849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.49.126:44103 (local address 127.20.49.65:52645)
1209 08:59:08.676966 (+ 117us) negotiation.cc:107] Waiting for socket to connect
1209 08:59:08.676981 (+ 15us) client_negotiation.cc:174] Beginning negotiation
1209 08:59:08.677061 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1209 08:59:08.677366 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1209 08:59:08.677369 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1209 08:59:08.677511 (+ 142us) client_negotiation.cc:189] Negotiated authn=SASL
1209 08:59:08.677658 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:59:08.677663 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:59:08.678367 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1209 08:59:08.678371 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1209 08:59:08.678738 (+ 367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:59:08.678744 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:59:08.678848 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:59:08.679341 (+ 493us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1209 08:59:08.679357 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
1209 08:59:08.680984 (+ 1627us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.49.126:44103: Server oryx/127.20.49.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":34}
W20251209 08:59:08.681226 30234 negotiation.cc:337] Failed RPC negotiation. Trace:
1209 08:59:08.677084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.49.65:52645 (local address 127.20.49.126:44103)
1209 08:59:08.677189 (+ 105us) server_negotiation.cc:207] Beginning negotiation
1209 08:59:08.677192 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1209 08:59:08.677202 (+ 10us) server_negotiation.cc:408] Connection header received
1209 08:59:08.677232 (+ 30us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1209 08:59:08.677234 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1209 08:59:08.677271 (+ 37us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1209 08:59:08.677339 (+ 68us) server_negotiation.cc:227] Negotiated authn=SASL
1209 08:59:08.677801 (+ 462us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:59:08.678268 (+ 467us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1209 08:59:08.678847 (+ 579us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:59:08.679349 (+ 502us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:59:08.681106 (+ 1757us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.49.65:52645: BlockingRecv error: recv got EOF from 127.20.49.65:52645 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
W20251209 08:59:08.681578 30232 heartbeater.cc:412] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20251209 08:59:08.778415 30236 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251209 08:59:08.778694 30236 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251209 08:59:08.778764 30236 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251209 08:59:08.782064 30236 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251209 08:59:08.782130 30236 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251209 08:59:08.782208 30236 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.49.66
I20251209 08:59:08.786715 30236 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.49.84:41011
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765270668420570-20677-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765270668420570-20677-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765270668420570-20677-0/krb5kdc/oryx_127.20.49.66.keytab
--principal=oryx/127.20.49.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.49.66:44559
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765270668420570-20677-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.49.66
--webserver_port=42547
--webserver_require_spnego=true
--tserver_master_addrs=127.20.49.126:44103
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30236
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.49.66
--log_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765270668420570-20677-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 468897755969d686fb08386f42f7835685e7953a
build type DEBUG
built by None at 09 Dec 2025 08:43:15 UTC on 24a791456cd2
build id 9426
I20251209 08:59:08.787784 30236 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251209 08:59:08.788618 30236 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251209 08:59:08.795168 30242 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251209 08:59:08.795150 30241 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251209 08:59:08.795149 30244 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251209 08:59:08.795406 30236 server_base.cc:1047] running on GCE node
I20251209 08:59:08.795800 30236 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251209 08:59:08.796414 30236 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251209 08:59:08.797592 30236 hybrid_clock.cc:648] HybridClock initialized: now 1765270748797577 us; error 30 us; skew 500 ppm
Dec 09 08:59:08 dist-test-slave-8w6c krb5kdc[29492](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765270748, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.49.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251209 08:59:08.800315 30236 init.cc:377] Logged in from keytab as oryx/127.20.49.66@KRBTEST.COM (short username oryx)
I20251209 08:59:08.801411 30236 webserver.cc:492] Webserver started at http://127.20.49.66:42547/ using document root <none> and password file <none>
I20251209 08:59:08.802142 30236 fs_manager.cc:362] Metadata directory not provided
I20251209 08:59:08.802227 30236 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251209 08:59:08.806581 30236 fs_manager.cc:714] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20251209 08:59:08.808561 30251 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251209 08:59:08.809640 30236 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20251209 08:59:08.809789 30236 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765270668420570-20677-0/minicluster-data/ts-1/data,/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765270668420570-20677-0/minicluster-data/ts-1/wal
uuid: "e1df360180a247ce867f5c22eaf9cfb4"
format_stamp: "Formatted at 2025-12-09 08:59:07 on dist-test-slave-8w6c"
server_key: "e0fa2f988bc92fd80ee260638beb70e1"
server_key_iv: "f3cc25b37281be03451eb190c9dc3ad9"
server_key_version: "encryptionkey@0"
I20251209 08:59:08.810233 30236 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765270668420570-20677-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765270668420570-20677-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765270668420570-20677-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251209 08:59:08.826879 30236 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251209 08:59:08.830185 30236 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251209 08:59:08.830351 30236 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251209 08:59:08.830880 30236 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251209 08:59:08.831758 30236 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251209 08:59:08.831807 30236 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251209 08:59:08.831848 30236 ts_tablet_manager.cc:616] Registered 0 tablets
I20251209 08:59:08.831871 30236 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251209 08:59:08.842096 30236 rpc_server.cc:307] RPC server started. Bound to: 127.20.49.66:44559
I20251209 08:59:08.842126 30364 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.49.66:44559 every 8 connection(s)
I20251209 08:59:08.843135 30236 server_base.cc:1179] Dumped server information to /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765270668420570-20677-0/minicluster-data/ts-1/data/info.pb
I20251209 08:59:08.849774 20677 external_mini_cluster.cc:1428] Started /tmp/dist-test-task8XuDZV/build/debug/bin/kudu as pid 30236
I20251209 08:59:08.850281 20677 external_mini_cluster.cc:1366] Running /tmp/dist-test-task8XuDZV/build/debug/bin/kudu
/tmp/dist-test-task8XuDZV/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765270668420570-20677-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765270668420570-20677-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765270668420570-20677-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765270668420570-20677-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.49.67:40601
--local_ip_for_outbound_sockets=127.20.49.67
--tserver_master_addrs=127.20.49.126:44103
--webserver_port=38479
--webserver_interface=127.20.49.67
--keytab_file=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765270668420570-20677-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.49.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.20.49.84:41011
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.20.49.67
--keytab_file=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765270668420570-20677-0/krb5kdc/oryx_127.20.49.67.keytab with env {KRB5CCNAME=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765270668420570-20677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765270668420570-20677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765270668420570-20677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Dec 09 08:59:08 dist-test-slave-8w6c krb5kdc[29492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.49.66@KRBTEST.COM for oryx/127.20.49.126@KRBTEST.COM, Server not found in Kerberos database
Dec 09 08:59:08 dist-test-slave-8w6c krb5kdc[29492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.49.66@KRBTEST.COM for oryx/127.20.49.126@KRBTEST.COM, Server not found in Kerberos database
W20251209 08:59:08.853776 30234 negotiation.cc:337] Failed RPC negotiation. Trace:
1209 08:59:08.845084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.49.66:53215 (local address 127.20.49.126:44103)
1209 08:59:08.845246 (+ 162us) server_negotiation.cc:207] Beginning negotiation
1209 08:59:08.845251 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1209 08:59:08.846032 (+ 781us) server_negotiation.cc:408] Connection header received
1209 08:59:08.846867 (+ 835us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1209 08:59:08.846870 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1209 08:59:08.846925 (+ 55us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1209 08:59:08.847005 (+ 80us) server_negotiation.cc:227] Negotiated authn=SASL
1209 08:59:08.848790 (+ 1785us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:59:08.849299 (+ 509us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1209 08:59:08.850104 (+ 805us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:59:08.850320 (+ 216us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:59:08.853612 (+ 3292us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.49.66:53215: BlockingRecv error: recv got EOF from 127.20.49.66:53215 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
I20251209 08:59:08.854254 30367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:59:08.845388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.49.126:44103 (local address 127.20.49.66:53215)
1209 08:59:08.845870 (+ 482us) negotiation.cc:107] Waiting for socket to connect
1209 08:59:08.845916 (+ 46us) client_negotiation.cc:174] Beginning negotiation
1209 08:59:08.846661 (+ 745us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1209 08:59:08.847165 (+ 504us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1209 08:59:08.847193 (+ 28us) client_negotiation.cc:394] Received NEGOTIATE response from server
1209 08:59:08.847644 (+ 451us) client_negotiation.cc:189] Negotiated authn=SASL
1209 08:59:08.848612 (+ 968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:59:08.848629 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:59:08.849455 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1209 08:59:08.849458 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1209 08:59:08.849975 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:59:08.849986 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:59:08.850180 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:59:08.850934 (+ 754us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1209 08:59:08.850961 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
1209 08:59:08.853463 (+ 2502us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.49.126:44103: Server oryx/127.20.49.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":286,"thread_start_us":114,"threads_started":1}
W20251209 08:59:08.854658 30365 heartbeater.cc:646] Failed to heartbeat to 127.20.49.126:44103 (0 consecutive failures): Not authorized: Failed to ping master at 127.20.49.126:44103: Client connection negotiation failed: client connection to 127.20.49.126:44103: Server oryx/127.20.49.126@KRBTEST.COM not found in Kerberos database
Dec 09 08:59:08 dist-test-slave-8w6c krb5kdc[29492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.49.66@KRBTEST.COM for oryx/127.20.49.126@KRBTEST.COM, Server not found in Kerberos database
Dec 09 08:59:08 dist-test-slave-8w6c krb5kdc[29492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.49.66@KRBTEST.COM for oryx/127.20.49.126@KRBTEST.COM, Server not found in Kerberos database
I20251209 08:59:08.859902 30367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:59:08.855169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.49.126:44103 (local address 127.20.49.66:55371)
1209 08:59:08.855303 (+ 134us) negotiation.cc:107] Waiting for socket to connect
1209 08:59:08.855318 (+ 15us) client_negotiation.cc:174] Beginning negotiation
1209 08:59:08.855408 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1209 08:59:08.855657 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1209 08:59:08.855659 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1209 08:59:08.855820 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
1209 08:59:08.856024 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:59:08.856029 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:59:08.856750 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1209 08:59:08.856753 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1209 08:59:08.857126 (+ 373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:59:08.857133 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:59:08.857240 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:59:08.857901 (+ 661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1209 08:59:08.857914 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
1209 08:59:08.859743 (+ 1829us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.49.126:44103: Server oryx/127.20.49.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":45}
W20251209 08:59:08.860000 30234 negotiation.cc:337] Failed RPC negotiation. Trace:
1209 08:59:08.855234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.49.66:55371 (local address 127.20.49.126:44103)
1209 08:59:08.855367 (+ 133us) server_negotiation.cc:207] Beginning negotiation
1209 08:59:08.855372 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1209 08:59:08.855386 (+ 14us) server_negotiation.cc:408] Connection header received
1209 08:59:08.855504 (+ 118us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1209 08:59:08.855508 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1209 08:59:08.855556 (+ 48us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1209 08:59:08.855633 (+ 77us) server_negotiation.cc:227] Negotiated authn=SASL
1209 08:59:08.856143 (+ 510us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:59:08.856623 (+ 480us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1209 08:59:08.857234 (+ 611us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:59:08.857447 (+ 213us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:59:08.859893 (+ 2446us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.49.66:55371: BlockingRecv error: recv got EOF from 127.20.49.66:55371 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Dec 09 08:59:08 dist-test-slave-8w6c krb5kdc[29492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.49.66@KRBTEST.COM for oryx/127.20.49.126@KRBTEST.COM, Server not found in Kerberos database
Dec 09 08:59:08 dist-test-slave-8w6c krb5kdc[29492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.49.66@KRBTEST.COM for oryx/127.20.49.126@KRBTEST.COM, Server not found in Kerberos database
I20251209 08:59:08.865310 30367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:59:08.860534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.49.126:44103 (local address 127.20.49.66:59937)
1209 08:59:08.860767 (+ 233us) negotiation.cc:107] Waiting for socket to connect
1209 08:59:08.860779 (+ 12us) client_negotiation.cc:174] Beginning negotiation
1209 08:59:08.860855 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1209 08:59:08.861104 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1209 08:59:08.861106 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1209 08:59:08.861231 (+ 125us) client_negotiation.cc:189] Negotiated authn=SASL
1209 08:59:08.861381 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:59:08.861386 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:59:08.862454 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1209 08:59:08.862457 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1209 08:59:08.862826 (+ 369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:59:08.862833 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:59:08.862933 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:59:08.863429 (+ 496us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1209 08:59:08.863442 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
1209 08:59:08.865211 (+ 1769us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.49.126:44103: Server oryx/127.20.49.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":43}
W20251209 08:59:08.865492 30234 negotiation.cc:337] Failed RPC negotiation. Trace:
1209 08:59:08.860625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.49.66:59937 (local address 127.20.49.126:44103)
1209 08:59:08.860733 (+ 108us) server_negotiation.cc:207] Beginning negotiation
1209 08:59:08.860737 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1209 08:59:08.860845 (+ 108us) server_negotiation.cc:408] Connection header received
1209 08:59:08.860949 (+ 104us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1209 08:59:08.860955 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
1209 08:59:08.861012 (+ 57us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1209 08:59:08.861095 (+ 83us) server_negotiation.cc:227] Negotiated authn=SASL
1209 08:59:08.861490 (+ 395us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:59:08.862338 (+ 848us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1209 08:59:08.862936 (+ 598us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:59:08.863143 (+ 207us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:59:08.865373 (+ 2230us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.49.66:59937: BlockingRecv error: recv got EOF from 127.20.49.66:59937 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
W20251209 08:59:08.865793 30365 heartbeater.cc:412] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20251209 08:59:08.962867 30368 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251209 08:59:08.963109 30368 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251209 08:59:08.963203 30368 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251209 08:59:08.966516 30368 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251209 08:59:08.966586 30368 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251209 08:59:08.966704 30368 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.49.67
I20251209 08:59:08.970949 30368 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.49.84:41011
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765270668420570-20677-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765270668420570-20677-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765270668420570-20677-0/krb5kdc/oryx_127.20.49.67.keytab
--principal=oryx/127.20.49.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.49.67:40601
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765270668420570-20677-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.49.67
--webserver_port=38479
--webserver_require_spnego=true
--tserver_master_addrs=127.20.49.126:44103
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30368
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.49.67
--log_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765270668420570-20677-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 468897755969d686fb08386f42f7835685e7953a
build type DEBUG
built by None at 09 Dec 2025 08:43:15 UTC on 24a791456cd2
build id 9426
I20251209 08:59:08.972049 30368 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251209 08:59:08.972916 30368 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251209 08:59:08.979470 30374 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251209 08:59:08.979449 30376 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251209 08:59:08.979449 30373 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251209 08:59:08.979748 30368 server_base.cc:1047] running on GCE node
I20251209 08:59:08.980113 30368 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251209 08:59:08.980661 30368 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251209 08:59:08.981810 30368 hybrid_clock.cc:648] HybridClock initialized: now 1765270748981794 us; error 30 us; skew 500 ppm
Dec 09 08:59:08 dist-test-slave-8w6c krb5kdc[29492](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765270748, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.49.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251209 08:59:08.984314 30368 init.cc:377] Logged in from keytab as oryx/127.20.49.67@KRBTEST.COM (short username oryx)
I20251209 08:59:08.985361 30368 webserver.cc:492] Webserver started at http://127.20.49.67:38479/ using document root <none> and password file <none>
I20251209 08:59:08.985987 30368 fs_manager.cc:362] Metadata directory not provided
I20251209 08:59:08.986059 30368 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251209 08:59:08.989464 30368 fs_manager.cc:714] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20251209 08:59:08.991405 30383 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251209 08:59:08.992388 30368 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20251209 08:59:08.992493 30368 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765270668420570-20677-0/minicluster-data/ts-2/data,/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765270668420570-20677-0/minicluster-data/ts-2/wal
uuid: "6d524d7b0cbb4204ab6d9e9204d5da91"
format_stamp: "Formatted at 2025-12-09 08:59:08 on dist-test-slave-8w6c"
server_key: "303364481f2d019926121332e5e10439"
server_key_iv: "c4e68bc00f1fcb9d16cf0ba1ebcfd935"
server_key_version: "encryptionkey@0"
I20251209 08:59:08.992910 30368 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765270668420570-20677-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765270668420570-20677-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765270668420570-20677-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251209 08:59:09.019207 30368 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251209 08:59:09.022416 30368 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251209 08:59:09.022606 30368 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251209 08:59:09.023187 30368 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251209 08:59:09.024135 30368 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251209 08:59:09.024187 30368 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251209 08:59:09.024247 30368 ts_tablet_manager.cc:616] Registered 0 tablets
I20251209 08:59:09.024274 30368 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251209 08:59:09.033524 30368 rpc_server.cc:307] RPC server started. Bound to: 127.20.49.67:40601
I20251209 08:59:09.033564 30496 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.49.67:40601 every 8 connection(s)
I20251209 08:59:09.034584 30368 server_base.cc:1179] Dumped server information to /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1765270668420570-20677-0/minicluster-data/ts-2/data/info.pb
I20251209 08:59:09.037366 20677 external_mini_cluster.cc:1428] Started /tmp/dist-test-task8XuDZV/build/debug/bin/kudu as pid 30368
Dec 09 08:59:09 dist-test-slave-8w6c krb5kdc[29492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.49.67@KRBTEST.COM for oryx/127.20.49.126@KRBTEST.COM, Server not found in Kerberos database
Dec 09 08:59:09 dist-test-slave-8w6c krb5kdc[29492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.49.67@KRBTEST.COM for oryx/127.20.49.126@KRBTEST.COM, Server not found in Kerberos database
W20251209 08:59:09.045542 30234 negotiation.cc:337] Failed RPC negotiation. Trace:
1209 08:59:09.036361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.49.67:45919 (local address 127.20.49.126:44103)
1209 08:59:09.036491 (+ 130us) server_negotiation.cc:207] Beginning negotiation
1209 08:59:09.036495 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1209 08:59:09.037208 (+ 713us) server_negotiation.cc:408] Connection header received
1209 08:59:09.038306 (+ 1098us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1209 08:59:09.038312 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
1209 08:59:09.038383 (+ 71us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1209 08:59:09.038472 (+ 89us) server_negotiation.cc:227] Negotiated authn=SASL
1209 08:59:09.040438 (+ 1966us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:59:09.041240 (+ 802us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1209 08:59:09.042069 (+ 829us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:59:09.042227 (+ 158us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:59:09.045436 (+ 3209us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.49.67:45919: BlockingRecv error: recv got EOF from 127.20.49.67:45919 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
I20251209 08:59:09.046124 30499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:59:09.036590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.49.126:44103 (local address 127.20.49.67:45919)
1209 08:59:09.037038 (+ 448us) negotiation.cc:107] Waiting for socket to connect
1209 08:59:09.037074 (+ 36us) client_negotiation.cc:174] Beginning negotiation
1209 08:59:09.038084 (+ 1010us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1209 08:59:09.038675 (+ 591us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1209 08:59:09.038686 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
1209 08:59:09.039151 (+ 465us) client_negotiation.cc:189] Negotiated authn=SASL
1209 08:59:09.040251 (+ 1100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:59:09.040268 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:59:09.041369 (+ 1101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1209 08:59:09.041372 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1209 08:59:09.041955 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:59:09.041963 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:59:09.042188 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:59:09.042833 (+ 645us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1209 08:59:09.042856 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
1209 08:59:09.045281 (+ 2425us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.49.126:44103: Server oryx/127.20.49.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":117,"threads_started":1}
W20251209 08:59:09.046567 30497 heartbeater.cc:646] Failed to heartbeat to 127.20.49.126:44103 (0 consecutive failures): Not authorized: Failed to ping master at 127.20.49.126:44103: Client connection negotiation failed: client connection to 127.20.49.126:44103: Server oryx/127.20.49.126@KRBTEST.COM not found in Kerberos database
I20251209 08:59:09.047529 30501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:59:09.038350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34350 (local address 127.20.49.126:44103)
1209 08:59:09.038597 (+ 247us) server_negotiation.cc:207] Beginning negotiation
1209 08:59:09.038616 (+ 19us) server_negotiation.cc:400] Waiting for connection header
1209 08:59:09.038629 (+ 13us) server_negotiation.cc:408] Connection header received
1209 08:59:09.038664 (+ 35us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1209 08:59:09.038667 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1209 08:59:09.038710 (+ 43us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1209 08:59:09.038801 (+ 91us) server_negotiation.cc:227] Negotiated authn=SASL
1209 08:59:09.039675 (+ 874us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:59:09.040484 (+ 809us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1209 08:59:09.041533 (+ 1049us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:59:09.041748 (+ 215us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:59:09.043191 (+ 1443us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1209 08:59:09.043213 (+ 22us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1209 08:59:09.043228 (+ 15us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1209 08:59:09.043284 (+ 56us) server_negotiation.cc:923] calling sasl_server_start()
1209 08:59:09.045321 (+ 2037us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1209 08:59:09.045843 (+ 522us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1209 08:59:09.045851 (+ 8us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1209 08:59:09.045857 (+ 6us) server_negotiation.cc:968] calling sasl_server_step()
1209 08:59:09.045937 (+ 80us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1209 08:59:09.046175 (+ 238us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1209 08:59:09.046177 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1209 08:59:09.046179 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1209 08:59:09.046564 (+ 385us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1209 08:59:09.046750 (+ 186us) server_negotiation.cc:1036] Waiting for connection context
1209 08:59:09.047126 (+ 376us) server_negotiation.cc:300] Negotiation successful
1209 08:59:09.047386 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":180,"thread_start_us":85,"threads_started":1}
Dec 09 08:59:09 dist-test-slave-8w6c krb5kdc[29492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.49.67@KRBTEST.COM for oryx/127.20.49.126@KRBTEST.COM, Server not found in Kerberos database
Dec 09 08:59:09 dist-test-slave-8w6c krb5kdc[29492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.49.67@KRBTEST.COM for oryx/127.20.49.126@KRBTEST.COM, Server not found in Kerberos database
I20251209 08:59:09.051623 30499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:59:09.047004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.49.126:44103 (local address 127.20.49.67:53815)
1209 08:59:09.047178 (+ 174us) negotiation.cc:107] Waiting for socket to connect
1209 08:59:09.047191 (+ 13us) client_negotiation.cc:174] Beginning negotiation
1209 08:59:09.047264 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1209 08:59:09.047501 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1209 08:59:09.047504 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1209 08:59:09.047674 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
1209 08:59:09.047914 (+ 240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:59:09.047921 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:59:09.048646 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1209 08:59:09.048649 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1209 08:59:09.049071 (+ 422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:59:09.049080 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:59:09.049214 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:59:09.049837 (+ 623us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1209 08:59:09.049858 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
1209 08:59:09.051535 (+ 1677us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.49.126:44103: Server oryx/127.20.49.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":103}
W20251209 08:59:09.051767 30234 negotiation.cc:337] Failed RPC negotiation. Trace:
1209 08:59:09.047093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.49.67:53815 (local address 127.20.49.126:44103)
1209 08:59:09.047220 (+ 127us) server_negotiation.cc:207] Beginning negotiation
1209 08:59:09.047225 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1209 08:59:09.047240 (+ 15us) server_negotiation.cc:408] Connection header received
1209 08:59:09.047360 (+ 120us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1209 08:59:09.047362 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1209 08:59:09.047400 (+ 38us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1209 08:59:09.047466 (+ 66us) server_negotiation.cc:227] Negotiated authn=SASL
1209 08:59:09.048027 (+ 561us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:59:09.048515 (+ 488us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1209 08:59:09.049187 (+ 672us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:59:09.049365 (+ 178us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:59:09.051646 (+ 2281us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.49.67:53815: BlockingRecv error: recv got EOF from 127.20.49.67:53815 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Dec 09 08:59:09 dist-test-slave-8w6c krb5kdc[29492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.49.67@KRBTEST.COM for oryx/127.20.49.126@KRBTEST.COM, Server not found in Kerberos database
Dec 09 08:59:09 dist-test-slave-8w6c krb5kdc[29492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.49.67@KRBTEST.COM for oryx/127.20.49.126@KRBTEST.COM, Server not found in Kerberos database
I20251209 08:59:09.056183 30499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:59:09.052158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.49.126:44103 (local address 127.20.49.67:33133)
1209 08:59:09.052275 (+ 117us) negotiation.cc:107] Waiting for socket to connect
1209 08:59:09.052287 (+ 12us) client_negotiation.cc:174] Beginning negotiation
1209 08:59:09.052352 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1209 08:59:09.052577 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1209 08:59:09.052579 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1209 08:59:09.052738 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
1209 08:59:09.052897 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:59:09.052903 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:59:09.053528 (+ 625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1209 08:59:09.053532 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1209 08:59:09.053978 (+ 446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:59:09.053986 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:59:09.054079 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:59:09.054535 (+ 456us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1209 08:59:09.054548 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
1209 08:59:09.056095 (+ 1547us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.49.126:44103: Server oryx/127.20.49.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":49}
W20251209 08:59:09.056344 30234 negotiation.cc:337] Failed RPC negotiation. Trace:
1209 08:59:09.052245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.49.67:33133 (local address 127.20.49.126:44103)
1209 08:59:09.052376 (+ 131us) server_negotiation.cc:207] Beginning negotiation
1209 08:59:09.052379 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1209 08:59:09.052392 (+ 13us) server_negotiation.cc:408] Connection header received
1209 08:59:09.052419 (+ 27us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1209 08:59:09.052422 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1209 08:59:09.052454 (+ 32us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1209 08:59:09.052519 (+ 65us) server_negotiation.cc:227] Negotiated authn=SASL
1209 08:59:09.052994 (+ 475us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:59:09.053430 (+ 436us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1209 08:59:09.054081 (+ 651us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:59:09.054212 (+ 131us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:59:09.056215 (+ 2003us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.49.67:33133: BlockingRecv error: recv got EOF from 127.20.49.67:33133 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
W20251209 08:59:09.056370 30497 heartbeater.cc:412] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Dec 09 08:59:09 dist-test-slave-8w6c krb5kdc[29492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.49.65@KRBTEST.COM for oryx/127.20.49.126@KRBTEST.COM, Server not found in Kerberos database
Dec 09 08:59:09 dist-test-slave-8w6c krb5kdc[29492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.49.65@KRBTEST.COM for oryx/127.20.49.126@KRBTEST.COM, Server not found in Kerberos database
I20251209 08:59:09.687673 30502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:59:09.682242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.49.126:44103 (local address 127.20.49.65:39409)
1209 08:59:09.682564 (+ 322us) negotiation.cc:107] Waiting for socket to connect
1209 08:59:09.682579 (+ 15us) client_negotiation.cc:174] Beginning negotiation
1209 08:59:09.682684 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1209 08:59:09.682981 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1209 08:59:09.682983 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1209 08:59:09.683170 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
1209 08:59:09.683420 (+ 250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:59:09.683426 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:59:09.684415 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1209 08:59:09.684419 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1209 08:59:09.684937 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:59:09.684947 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:59:09.685046 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:59:09.685618 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1209 08:59:09.685631 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
1209 08:59:09.687500 (+ 1869us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.49.126:44103: Server oryx/127.20.49.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":76,"threads_started":1}
W20251209 08:59:09.687796 30503 negotiation.cc:337] Failed RPC negotiation. Trace:
1209 08:59:09.682507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.49.65:39409 (local address 127.20.49.126:44103)
1209 08:59:09.682746 (+ 239us) server_negotiation.cc:207] Beginning negotiation
1209 08:59:09.682749 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1209 08:59:09.682763 (+ 14us) server_negotiation.cc:408] Connection header received
1209 08:59:09.682801 (+ 38us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1209 08:59:09.682804 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1209 08:59:09.682847 (+ 43us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1209 08:59:09.682929 (+ 82us) server_negotiation.cc:227] Negotiated authn=SASL
1209 08:59:09.683531 (+ 602us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:59:09.684295 (+ 764us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1209 08:59:09.685051 (+ 756us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:59:09.685195 (+ 144us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:59:09.687632 (+ 2437us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.49.65:39409: BlockingRecv error: recv got EOF from 127.20.49.65:39409 (error 108)
Metrics: {"server-negotiator.queue_time_us":164,"thread_start_us":76,"threads_started":1}
Dec 09 08:59:09 dist-test-slave-8w6c krb5kdc[29492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.49.66@KRBTEST.COM for oryx/127.20.49.126@KRBTEST.COM, Server not found in Kerberos database
Dec 09 08:59:09 dist-test-slave-8w6c krb5kdc[29492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.49.66@KRBTEST.COM for oryx/127.20.49.126@KRBTEST.COM, Server not found in Kerberos database
I20251209 08:59:09.872155 30504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:59:09.866567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.49.126:44103 (local address 127.20.49.66:47315)
1209 08:59:09.866853 (+ 286us) negotiation.cc:107] Waiting for socket to connect
1209 08:59:09.866867 (+ 14us) client_negotiation.cc:174] Beginning negotiation
1209 08:59:09.866999 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1209 08:59:09.867303 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1209 08:59:09.867306 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1209 08:59:09.867505 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
1209 08:59:09.867748 (+ 243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:59:09.867755 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:59:09.868526 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1209 08:59:09.868529 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1209 08:59:09.868965 (+ 436us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:59:09.868973 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:59:09.869071 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:59:09.869735 (+ 664us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1209 08:59:09.869751 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
1209 08:59:09.871959 (+ 2208us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.49.126:44103: Server oryx/127.20.49.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":90,"threads_started":1}
W20251209 08:59:09.872196 30503 negotiation.cc:337] Failed RPC negotiation. Trace:
1209 08:59:09.866650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.49.66:47315 (local address 127.20.49.126:44103)
1209 08:59:09.866847 (+ 197us) server_negotiation.cc:207] Beginning negotiation
1209 08:59:09.866851 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1209 08:59:09.866965 (+ 114us) server_negotiation.cc:408] Connection header received
1209 08:59:09.867083 (+ 118us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1209 08:59:09.867086 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1209 08:59:09.867143 (+ 57us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1209 08:59:09.867228 (+ 85us) server_negotiation.cc:227] Negotiated authn=SASL
1209 08:59:09.867897 (+ 669us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:59:09.868411 (+ 514us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1209 08:59:09.869069 (+ 658us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:59:09.869200 (+ 131us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:59:09.872080 (+ 2880us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.49.66:47315: BlockingRecv error: recv got EOF from 127.20.49.66:47315 (error 108)
Metrics: {"server-negotiator.queue_time_us":103}
Dec 09 08:59:10 dist-test-slave-8w6c krb5kdc[29492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.49.67@KRBTEST.COM for oryx/127.20.49.126@KRBTEST.COM, Server not found in Kerberos database
Dec 09 08:59:10 dist-test-slave-8w6c krb5kdc[29492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.49.67@KRBTEST.COM for oryx/127.20.49.126@KRBTEST.COM, Server not found in Kerberos database
I20251209 08:59:10.062450 30505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:59:10.057056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.49.126:44103 (local address 127.20.49.67:51719)
1209 08:59:10.057377 (+ 321us) negotiation.cc:107] Waiting for socket to connect
1209 08:59:10.057393 (+ 16us) client_negotiation.cc:174] Beginning negotiation
1209 08:59:10.057537 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1209 08:59:10.057831 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1209 08:59:10.057834 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1209 08:59:10.058034 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
1209 08:59:10.058258 (+ 224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:59:10.058264 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:59:10.059022 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1209 08:59:10.059026 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1209 08:59:10.059426 (+ 400us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:59:10.059433 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:59:10.059545 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:59:10.060188 (+ 643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1209 08:59:10.060200 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
1209 08:59:10.062318 (+ 2118us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.49.126:44103: Server oryx/127.20.49.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":100,"threads_started":1}
W20251209 08:59:10.062551 30503 negotiation.cc:337] Failed RPC negotiation. Trace:
1209 08:59:10.057199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.49.67:51719 (local address 127.20.49.126:44103)
1209 08:59:10.057380 (+ 181us) server_negotiation.cc:207] Beginning negotiation
1209 08:59:10.057387 (+ 7us) server_negotiation.cc:400] Waiting for connection header
1209 08:59:10.057476 (+ 89us) server_negotiation.cc:408] Connection header received
1209 08:59:10.057635 (+ 159us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1209 08:59:10.057638 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1209 08:59:10.057718 (+ 80us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1209 08:59:10.057795 (+ 77us) server_negotiation.cc:227] Negotiated authn=SASL
1209 08:59:10.058387 (+ 592us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:59:10.058891 (+ 504us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1209 08:59:10.059540 (+ 649us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:59:10.059751 (+ 211us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:59:10.062451 (+ 2700us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.49.67:51719: BlockingRecv error: recv got EOF from 127.20.49.67:51719 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
Dec 09 08:59:10 dist-test-slave-8w6c krb5kdc[29492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.49.65@KRBTEST.COM for oryx/127.20.49.126@KRBTEST.COM, Server not found in Kerberos database
Dec 09 08:59:10 dist-test-slave-8w6c krb5kdc[29492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.49.65@KRBTEST.COM for oryx/127.20.49.126@KRBTEST.COM, Server not found in Kerberos database
I20251209 08:59:10.693329 30506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:59:10.688466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.49.126:44103 (local address 127.20.49.65:49397)
1209 08:59:10.688739 (+ 273us) negotiation.cc:107] Waiting for socket to connect
1209 08:59:10.688752 (+ 13us) client_negotiation.cc:174] Beginning negotiation
1209 08:59:10.688844 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1209 08:59:10.689074 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1209 08:59:10.689076 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1209 08:59:10.689258 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
1209 08:59:10.689453 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:59:10.689459 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:59:10.690261 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1209 08:59:10.690263 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1209 08:59:10.690636 (+ 373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:59:10.690642 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:59:10.690744 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:59:10.691315 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1209 08:59:10.691329 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
1209 08:59:10.693171 (+ 1842us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.49.126:44103: Server oryx/127.20.49.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":77,"threads_started":1}
W20251209 08:59:10.693451 30507 negotiation.cc:337] Failed RPC negotiation. Trace:
1209 08:59:10.688596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.49.65:49397 (local address 127.20.49.126:44103)
1209 08:59:10.688855 (+ 259us) server_negotiation.cc:207] Beginning negotiation
1209 08:59:10.688858 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1209 08:59:10.688875 (+ 17us) server_negotiation.cc:408] Connection header received
1209 08:59:10.688910 (+ 35us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1209 08:59:10.688912 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1209 08:59:10.688958 (+ 46us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1209 08:59:10.689043 (+ 85us) server_negotiation.cc:227] Negotiated authn=SASL
1209 08:59:10.689597 (+ 554us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:59:10.690163 (+ 566us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1209 08:59:10.690768 (+ 605us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:59:10.690919 (+ 151us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:59:10.693324 (+ 2405us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.49.65:49397: BlockingRecv error: recv got EOF from 127.20.49.65:49397 (error 108)
Metrics: {"server-negotiator.queue_time_us":190,"thread_start_us":70,"threads_started":1}
Dec 09 08:59:10 dist-test-slave-8w6c krb5kdc[29492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.49.66@KRBTEST.COM for oryx/127.20.49.126@KRBTEST.COM, Server not found in Kerberos database
Dec 09 08:59:10 dist-test-slave-8w6c krb5kdc[29492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.49.66@KRBTEST.COM for oryx/127.20.49.126@KRBTEST.COM, Server not found in Kerberos database
I20251209 08:59:10.878786 30508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:59:10.873122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.49.126:44103 (local address 127.20.49.66:43773)
1209 08:59:10.873477 (+ 355us) negotiation.cc:107] Waiting for socket to connect
1209 08:59:10.873491 (+ 14us) client_negotiation.cc:174] Beginning negotiation
1209 08:59:10.873618 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1209 08:59:10.874101 (+ 483us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1209 08:59:10.874106 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
1209 08:59:10.874337 (+ 231us) client_negotiation.cc:189] Negotiated authn=SASL
1209 08:59:10.874556 (+ 219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:59:10.874563 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:59:10.875425 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1209 08:59:10.875429 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1209 08:59:10.875827 (+ 398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:59:10.875834 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:59:10.875931 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:59:10.876511 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1209 08:59:10.876525 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
1209 08:59:10.878662 (+ 2137us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.49.126:44103: Server oryx/127.20.49.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":152,"threads_started":1}
W20251209 08:59:10.879004 30507 negotiation.cc:337] Failed RPC negotiation. Trace:
1209 08:59:10.873259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.49.66:43773 (local address 127.20.49.126:44103)
1209 08:59:10.873445 (+ 186us) server_negotiation.cc:207] Beginning negotiation
1209 08:59:10.873453 (+ 8us) server_negotiation.cc:400] Waiting for connection header
1209 08:59:10.873822 (+ 369us) server_negotiation.cc:408] Connection header received
1209 08:59:10.873880 (+ 58us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1209 08:59:10.873884 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1209 08:59:10.873976 (+ 92us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1209 08:59:10.874054 (+ 78us) server_negotiation.cc:227] Negotiated authn=SASL
1209 08:59:10.874680 (+ 626us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:59:10.875330 (+ 650us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1209 08:59:10.875976 (+ 646us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:59:10.876125 (+ 149us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:59:10.878863 (+ 2738us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.49.66:43773: BlockingRecv error: recv got EOF from 127.20.49.66:43773 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
Dec 09 08:59:11 dist-test-slave-8w6c krb5kdc[29492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.49.67@KRBTEST.COM for oryx/127.20.49.126@KRBTEST.COM, Server not found in Kerberos database
Dec 09 08:59:11 dist-test-slave-8w6c krb5kdc[29492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.49.67@KRBTEST.COM for oryx/127.20.49.126@KRBTEST.COM, Server not found in Kerberos database
I20251209 08:59:11.068579 30509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:59:11.063437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.49.126:44103 (local address 127.20.49.67:57169)
1209 08:59:11.063716 (+ 279us) negotiation.cc:107] Waiting for socket to connect
1209 08:59:11.063732 (+ 16us) client_negotiation.cc:174] Beginning negotiation
1209 08:59:11.063841 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1209 08:59:11.064127 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1209 08:59:11.064130 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1209 08:59:11.064311 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
1209 08:59:11.064559 (+ 248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:59:11.064567 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:59:11.065311 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1209 08:59:11.065315 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1209 08:59:11.065747 (+ 432us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:59:11.065756 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:59:11.065848 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:59:11.066441 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1209 08:59:11.066454 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
1209 08:59:11.068425 (+ 1971us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.49.126:44103: Server oryx/127.20.49.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":81,"threads_started":1}
W20251209 08:59:11.068686 30507 negotiation.cc:337] Failed RPC negotiation. Trace:
1209 08:59:11.063590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.49.67:57169 (local address 127.20.49.126:44103)
1209 08:59:11.063782 (+ 192us) server_negotiation.cc:207] Beginning negotiation
1209 08:59:11.063788 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1209 08:59:11.063805 (+ 17us) server_negotiation.cc:408] Connection header received
1209 08:59:11.063936 (+ 131us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1209 08:59:11.063939 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1209 08:59:11.064003 (+ 64us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1209 08:59:11.064081 (+ 78us) server_negotiation.cc:227] Negotiated authn=SASL
1209 08:59:11.064674 (+ 593us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:59:11.065181 (+ 507us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1209 08:59:11.065848 (+ 667us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:59:11.065985 (+ 137us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:59:11.068579 (+ 2594us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.49.67:57169: BlockingRecv error: recv got EOF from 127.20.49.67:57169 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Dec 09 08:59:11 dist-test-slave-8w6c krb5kdc[29492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.49.65@KRBTEST.COM for oryx/127.20.49.126@KRBTEST.COM, Server not found in Kerberos database
Dec 09 08:59:11 dist-test-slave-8w6c krb5kdc[29492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.49.65@KRBTEST.COM for oryx/127.20.49.126@KRBTEST.COM, Server not found in Kerberos database
I20251209 08:59:11.700366 30510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:59:11.694311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.49.126:44103 (local address 127.20.49.65:53805)
1209 08:59:11.694699 (+ 388us) negotiation.cc:107] Waiting for socket to connect
1209 08:59:11.694742 (+ 43us) client_negotiation.cc:174] Beginning negotiation
1209 08:59:11.694838 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1209 08:59:11.695142 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1209 08:59:11.695144 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1209 08:59:11.695410 (+ 266us) client_negotiation.cc:189] Negotiated authn=SASL
1209 08:59:11.695652 (+ 242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:59:11.695660 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:59:11.696683 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1209 08:59:11.696686 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1209 08:59:11.697134 (+ 448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:59:11.697145 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:59:11.697259 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:59:11.698016 (+ 757us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1209 08:59:11.698035 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
1209 08:59:11.700203 (+ 2168us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.49.126:44103: Server oryx/127.20.49.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":290,"thread_start_us":95,"threads_started":1}
W20251209 08:59:11.700486 30511 negotiation.cc:337] Failed RPC negotiation. Trace:
1209 08:59:11.694439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.49.65:53805 (local address 127.20.49.126:44103)
1209 08:59:11.694771 (+ 332us) server_negotiation.cc:207] Beginning negotiation
1209 08:59:11.694777 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1209 08:59:11.694797 (+ 20us) server_negotiation.cc:408] Connection header received
1209 08:59:11.694937 (+ 140us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1209 08:59:11.694940 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1209 08:59:11.694996 (+ 56us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1209 08:59:11.695095 (+ 99us) server_negotiation.cc:227] Negotiated authn=SASL
1209 08:59:11.695828 (+ 733us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:59:11.696535 (+ 707us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1209 08:59:11.697245 (+ 710us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:59:11.697452 (+ 207us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:59:11.700345 (+ 2893us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.49.65:53805: BlockingRecv error: recv got EOF from 127.20.49.65:53805 (error 108)
Metrics: {"server-negotiator.queue_time_us":162,"thread_start_us":59,"threads_started":1}
Dec 09 08:59:11 dist-test-slave-8w6c krb5kdc[29492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.49.66@KRBTEST.COM for oryx/127.20.49.126@KRBTEST.COM, Server not found in Kerberos database
Dec 09 08:59:11 dist-test-slave-8w6c krb5kdc[29492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.49.66@KRBTEST.COM for oryx/127.20.49.126@KRBTEST.COM, Server not found in Kerberos database
I20251209 08:59:11.884768 30512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:59:11.879519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.49.126:44103 (local address 127.20.49.66:43159)
1209 08:59:11.879842 (+ 323us) negotiation.cc:107] Waiting for socket to connect
1209 08:59:11.879860 (+ 18us) client_negotiation.cc:174] Beginning negotiation
1209 08:59:11.880016 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1209 08:59:11.880283 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1209 08:59:11.880286 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1209 08:59:11.880502 (+ 216us) client_negotiation.cc:189] Negotiated authn=SASL
1209 08:59:11.880700 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:59:11.880707 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:59:11.881453 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1209 08:59:11.881457 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1209 08:59:11.881957 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:59:11.881970 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:59:11.882086 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:59:11.882687 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1209 08:59:11.882701 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
1209 08:59:11.884608 (+ 1907us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.49.126:44103: Server oryx/127.20.49.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":97,"threads_started":1}
W20251209 08:59:11.884884 30511 negotiation.cc:337] Failed RPC negotiation. Trace:
1209 08:59:11.879656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.49.66:43159 (local address 127.20.49.126:44103)
1209 08:59:11.879843 (+ 187us) server_negotiation.cc:207] Beginning negotiation
1209 08:59:11.879849 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1209 08:59:11.879949 (+ 100us) server_negotiation.cc:408] Connection header received
1209 08:59:11.880127 (+ 178us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1209 08:59:11.880130 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1209 08:59:11.880176 (+ 46us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1209 08:59:11.880254 (+ 78us) server_negotiation.cc:227] Negotiated authn=SASL
1209 08:59:11.880828 (+ 574us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:59:11.881339 (+ 511us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1209 08:59:11.882090 (+ 751us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:59:11.882311 (+ 221us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:59:11.884773 (+ 2462us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.49.66:43159: BlockingRecv error: recv got EOF from 127.20.49.66:43159 (error 108)
Metrics: {"server-negotiator.queue_time_us":99}
Dec 09 08:59:12 dist-test-slave-8w6c krb5kdc[29492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.49.67@KRBTEST.COM for oryx/127.20.49.126@KRBTEST.COM, Server not found in Kerberos database
Dec 09 08:59:12 dist-test-slave-8w6c krb5kdc[29492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.49.67@KRBTEST.COM for oryx/127.20.49.126@KRBTEST.COM, Server not found in Kerberos database
I20251209 08:59:12.074710 30513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:59:12.069476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.49.126:44103 (local address 127.20.49.67:42693)
1209 08:59:12.069828 (+ 352us) negotiation.cc:107] Waiting for socket to connect
1209 08:59:12.069844 (+ 16us) client_negotiation.cc:174] Beginning negotiation
1209 08:59:12.069975 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1209 08:59:12.070237 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1209 08:59:12.070240 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1209 08:59:12.070441 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
1209 08:59:12.070632 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:59:12.070638 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:59:12.071374 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1209 08:59:12.071378 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1209 08:59:12.071760 (+ 382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:59:12.071767 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:59:12.071875 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:59:12.072637 (+ 762us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1209 08:59:12.072651 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
1209 08:59:12.074511 (+ 1860us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.49.126:44103: Server oryx/127.20.49.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":112,"threads_started":1}
W20251209 08:59:12.074774 30511 negotiation.cc:337] Failed RPC negotiation. Trace:
1209 08:59:12.069604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.49.67:42693 (local address 127.20.49.126:44103)
1209 08:59:12.069811 (+ 207us) server_negotiation.cc:207] Beginning negotiation
1209 08:59:12.069815 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1209 08:59:12.069923 (+ 108us) server_negotiation.cc:408] Connection header received
1209 08:59:12.070077 (+ 154us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1209 08:59:12.070080 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1209 08:59:12.070128 (+ 48us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1209 08:59:12.070200 (+ 72us) server_negotiation.cc:227] Negotiated authn=SASL
1209 08:59:12.070763 (+ 563us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:59:12.071260 (+ 497us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1209 08:59:12.071868 (+ 608us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:59:12.072076 (+ 208us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:59:12.074658 (+ 2582us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.49.67:42693: BlockingRecv error: recv got EOF from 127.20.49.67:42693 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Dec 09 08:59:12 dist-test-slave-8w6c krb5kdc[29492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.49.65@KRBTEST.COM for oryx/127.20.49.126@KRBTEST.COM, Server not found in Kerberos database
Dec 09 08:59:12 dist-test-slave-8w6c krb5kdc[29492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.49.65@KRBTEST.COM for oryx/127.20.49.126@KRBTEST.COM, Server not found in Kerberos database
I20251209 08:59:12.706705 30514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:59:12.701341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.49.126:44103 (local address 127.20.49.65:43535)
1209 08:59:12.701661 (+ 320us) negotiation.cc:107] Waiting for socket to connect
1209 08:59:12.701742 (+ 81us) client_negotiation.cc:174] Beginning negotiation
1209 08:59:12.701870 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1209 08:59:12.702170 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1209 08:59:12.702173 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1209 08:59:12.702510 (+ 337us) client_negotiation.cc:189] Negotiated authn=SASL
1209 08:59:12.702727 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:59:12.702735 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:59:12.703536 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1209 08:59:12.703541 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1209 08:59:12.704004 (+ 463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:59:12.704013 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:59:12.704113 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:59:12.704690 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1209 08:59:12.704708 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
1209 08:59:12.706587 (+ 1879us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.49.126:44103: Server oryx/127.20.49.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":98,"threads_started":1}
W20251209 08:59:12.706861 30515 negotiation.cc:337] Failed RPC negotiation. Trace:
1209 08:59:12.701409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.49.65:43535 (local address 127.20.49.126:44103)
1209 08:59:12.701770 (+ 361us) server_negotiation.cc:207] Beginning negotiation
1209 08:59:12.701781 (+ 11us) server_negotiation.cc:400] Waiting for connection header
1209 08:59:12.701800 (+ 19us) server_negotiation.cc:408] Connection header received
1209 08:59:12.701976 (+ 176us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1209 08:59:12.701982 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
1209 08:59:12.702038 (+ 56us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1209 08:59:12.702134 (+ 96us) server_negotiation.cc:227] Negotiated authn=SASL
1209 08:59:12.702855 (+ 721us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:59:12.703374 (+ 519us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1209 08:59:12.704106 (+ 732us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:59:12.704245 (+ 139us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:59:12.706724 (+ 2479us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.49.65:43535: BlockingRecv error: recv got EOF from 127.20.49.65:43535 (error 108)
Metrics: {"server-negotiator.queue_time_us":253,"thread_start_us":95,"threads_started":1}
Dec 09 08:59:12 dist-test-slave-8w6c krb5kdc[29492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.49.66@KRBTEST.COM for oryx/127.20.49.126@KRBTEST.COM, Server not found in Kerberos database
Dec 09 08:59:12 dist-test-slave-8w6c krb5kdc[29492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.49.66@KRBTEST.COM for oryx/127.20.49.126@KRBTEST.COM, Server not found in Kerberos database
I20251209 08:59:12.891019 30516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:59:12.885773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.49.126:44103 (local address 127.20.49.66:37463)
1209 08:59:12.886066 (+ 293us) negotiation.cc:107] Waiting for socket to connect
1209 08:59:12.886080 (+ 14us) client_negotiation.cc:174] Beginning negotiation
1209 08:59:12.886188 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1209 08:59:12.886474 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1209 08:59:12.886476 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1209 08:59:12.886666 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
1209 08:59:12.886888 (+ 222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:59:12.886895 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:59:12.887602 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1209 08:59:12.887605 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1209 08:59:12.888007 (+ 402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:59:12.888014 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:59:12.888113 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:59:12.888703 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1209 08:59:12.888716 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
1209 08:59:12.890822 (+ 2106us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.49.126:44103: Server oryx/127.20.49.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":93,"threads_started":1}
W20251209 08:59:12.891093 30515 negotiation.cc:337] Failed RPC negotiation. Trace:
1209 08:59:12.885886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.49.66:37463 (local address 127.20.49.126:44103)
1209 08:59:12.886035 (+ 149us) server_negotiation.cc:207] Beginning negotiation
1209 08:59:12.886038 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1209 08:59:12.886152 (+ 114us) server_negotiation.cc:408] Connection header received
1209 08:59:12.886311 (+ 159us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1209 08:59:12.886314 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1209 08:59:12.886368 (+ 54us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1209 08:59:12.886453 (+ 85us) server_negotiation.cc:227] Negotiated authn=SASL
1209 08:59:12.887007 (+ 554us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:59:12.887508 (+ 501us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1209 08:59:12.888125 (+ 617us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:59:12.888269 (+ 144us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:59:12.890960 (+ 2691us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.49.66:37463: BlockingRecv error: recv got EOF from 127.20.49.66:37463 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Dec 09 08:59:13 dist-test-slave-8w6c krb5kdc[29492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.49.67@KRBTEST.COM for oryx/127.20.49.126@KRBTEST.COM, Server not found in Kerberos database
Dec 09 08:59:13 dist-test-slave-8w6c krb5kdc[29492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.49.67@KRBTEST.COM for oryx/127.20.49.126@KRBTEST.COM, Server not found in Kerberos database
I20251209 08:59:13.085450 30517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:59:13.075600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.49.126:44103 (local address 127.20.49.67:55029)
1209 08:59:13.075923 (+ 323us) negotiation.cc:107] Waiting for socket to connect
1209 08:59:13.075939 (+ 16us) client_negotiation.cc:174] Beginning negotiation
1209 08:59:13.076078 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1209 08:59:13.076337 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1209 08:59:13.076340 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1209 08:59:13.076538 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
1209 08:59:13.076724 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:59:13.076730 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:59:13.077455 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1209 08:59:13.077458 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1209 08:59:13.077950 (+ 492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:59:13.077959 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:59:13.078079 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:59:13.083526 (+ 5447us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1209 08:59:13.083541 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
1209 08:59:13.085300 (+ 1759us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.49.126:44103: Server oryx/127.20.49.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":118,"threads_started":1}
W20251209 08:59:13.085561 30515 negotiation.cc:337] Failed RPC negotiation. Trace:
1209 08:59:13.075778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.49.67:55029 (local address 127.20.49.126:44103)
1209 08:59:13.075925 (+ 147us) server_negotiation.cc:207] Beginning negotiation
1209 08:59:13.075931 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1209 08:59:13.076016 (+ 85us) server_negotiation.cc:408] Connection header received
1209 08:59:13.076187 (+ 171us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1209 08:59:13.076190 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1209 08:59:13.076237 (+ 47us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1209 08:59:13.076306 (+ 69us) server_negotiation.cc:227] Negotiated authn=SASL
1209 08:59:13.076849 (+ 543us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:59:13.077343 (+ 494us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1209 08:59:13.078071 (+ 728us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:59:13.078278 (+ 207us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:59:13.085447 (+ 7169us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.49.67:55029: BlockingRecv error: recv got EOF from 127.20.49.67:55029 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Dec 09 08:59:13 dist-test-slave-8w6c krb5kdc[29492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.49.65@KRBTEST.COM for oryx/127.20.49.126@KRBTEST.COM, Server not found in Kerberos database
Dec 09 08:59:13 dist-test-slave-8w6c krb5kdc[29492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.49.65@KRBTEST.COM for oryx/127.20.49.126@KRBTEST.COM, Server not found in Kerberos database
I20251209 08:59:13.712924 30518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:59:13.707559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.49.126:44103 (local address 127.20.49.65:52547)
1209 08:59:13.707864 (+ 305us) negotiation.cc:107] Waiting for socket to connect
1209 08:59:13.707877 (+ 13us) client_negotiation.cc:174] Beginning negotiation
1209 08:59:13.707967 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1209 08:59:13.708271 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1209 08:59:13.708274 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1209 08:59:13.708491 (+ 217us) client_negotiation.cc:189] Negotiated authn=SASL
1209 08:59:13.708699 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:59:13.708705 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:59:13.709530 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1209 08:59:13.709534 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1209 08:59:13.709992 (+ 458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:59:13.710003 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:59:13.710106 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:59:13.710701 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1209 08:59:13.710715 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
1209 08:59:13.712759 (+ 2044us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.49.126:44103: Server oryx/127.20.49.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":84,"threads_started":1}
W20251209 08:59:13.713038 30519 negotiation.cc:337] Failed RPC negotiation. Trace:
1209 08:59:13.707648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.49.65:52547 (local address 127.20.49.126:44103)
1209 08:59:13.707907 (+ 259us) server_negotiation.cc:207] Beginning negotiation
1209 08:59:13.707910 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1209 08:59:13.707925 (+ 15us) server_negotiation.cc:408] Connection header received
1209 08:59:13.708057 (+ 132us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1209 08:59:13.708060 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1209 08:59:13.708108 (+ 48us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1209 08:59:13.708193 (+ 85us) server_negotiation.cc:227] Negotiated authn=SASL
1209 08:59:13.708867 (+ 674us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:59:13.709406 (+ 539us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1209 08:59:13.710101 (+ 695us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:59:13.710246 (+ 145us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:59:13.712902 (+ 2656us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.49.65:52547: BlockingRecv error: recv got EOF from 127.20.49.65:52547 (error 108)
Metrics: {"server-negotiator.queue_time_us":188,"thread_start_us":106,"threads_started":1}
Dec 09 08:59:13 dist-test-slave-8w6c krb5kdc[29492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.49.66@KRBTEST.COM for oryx/127.20.49.126@KRBTEST.COM, Server not found in Kerberos database
Dec 09 08:59:13 dist-test-slave-8w6c krb5kdc[29492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.49.66@KRBTEST.COM for oryx/127.20.49.126@KRBTEST.COM, Server not found in Kerberos database
I20251209 08:59:13.897142 30520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:59:13.891904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.49.126:44103 (local address 127.20.49.66:37979)
1209 08:59:13.892232 (+ 328us) negotiation.cc:107] Waiting for socket to connect
1209 08:59:13.892250 (+ 18us) client_negotiation.cc:174] Beginning negotiation
1209 08:59:13.892376 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1209 08:59:13.892638 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1209 08:59:13.892641 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1209 08:59:13.892878 (+ 237us) client_negotiation.cc:189] Negotiated authn=SASL
1209 08:59:13.893071 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:59:13.893077 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:59:13.893905 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1209 08:59:13.893908 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1209 08:59:13.894363 (+ 455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:59:13.894371 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:59:13.894481 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:59:13.895160 (+ 679us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1209 08:59:13.895174 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
1209 08:59:13.896967 (+ 1793us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.49.126:44103: Server oryx/127.20.49.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":108,"threads_started":1}
W20251209 08:59:13.897142 30519 negotiation.cc:337] Failed RPC negotiation. Trace:
1209 08:59:13.892117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.49.66:37979 (local address 127.20.49.126:44103)
1209 08:59:13.892287 (+ 170us) server_negotiation.cc:207] Beginning negotiation
1209 08:59:13.892294 (+ 7us) server_negotiation.cc:400] Waiting for connection header
1209 08:59:13.892317 (+ 23us) server_negotiation.cc:408] Connection header received
1209 08:59:13.892479 (+ 162us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1209 08:59:13.892482 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1209 08:59:13.892531 (+ 49us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1209 08:59:13.892604 (+ 73us) server_negotiation.cc:227] Negotiated authn=SASL
1209 08:59:13.893212 (+ 608us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:59:13.893773 (+ 561us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1209 08:59:13.894477 (+ 704us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:59:13.894696 (+ 219us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:59:13.897048 (+ 2352us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.49.66:37979: BlockingRecv error: recv got EOF from 127.20.49.66:37979 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
Dec 09 08:59:14 dist-test-slave-8w6c krb5kdc[29492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.49.67@KRBTEST.COM for oryx/127.20.49.126@KRBTEST.COM, Server not found in Kerberos database
Dec 09 08:59:14 dist-test-slave-8w6c krb5kdc[29492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.49.67@KRBTEST.COM for oryx/127.20.49.126@KRBTEST.COM, Server not found in Kerberos database
I20251209 08:59:14.091264 30521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:59:14.086359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.49.126:44103 (local address 127.20.49.67:56575)
1209 08:59:14.086655 (+ 296us) negotiation.cc:107] Waiting for socket to connect
1209 08:59:14.086672 (+ 17us) client_negotiation.cc:174] Beginning negotiation
1209 08:59:14.086803 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1209 08:59:14.087055 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1209 08:59:14.087058 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1209 08:59:14.087284 (+ 226us) client_negotiation.cc:189] Negotiated authn=SASL
1209 08:59:14.087487 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:59:14.087493 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:59:14.088268 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1209 08:59:14.088272 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1209 08:59:14.088647 (+ 375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:59:14.088654 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:59:14.088761 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:59:14.089342 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1209 08:59:14.089354 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
1209 08:59:14.091093 (+ 1739us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.49.126:44103: Server oryx/127.20.49.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":93,"threads_started":1}
W20251209 08:59:14.091328 30519 negotiation.cc:337] Failed RPC negotiation. Trace:
1209 08:59:14.086475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.49.67:56575 (local address 127.20.49.126:44103)
1209 08:59:14.086657 (+ 182us) server_negotiation.cc:207] Beginning negotiation
1209 08:59:14.086663 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1209 08:59:14.086763 (+ 100us) server_negotiation.cc:408] Connection header received
1209 08:59:14.086897 (+ 134us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1209 08:59:14.086900 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1209 08:59:14.086949 (+ 49us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1209 08:59:14.087022 (+ 73us) server_negotiation.cc:227] Negotiated authn=SASL
1209 08:59:14.087627 (+ 605us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:59:14.088155 (+ 528us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1209 08:59:14.088768 (+ 613us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:59:14.088984 (+ 216us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:59:14.091222 (+ 2238us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.49.67:56575: BlockingRecv error: recv got EOF from 127.20.49.67:56575 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
Dec 09 08:59:14 dist-test-slave-8w6c krb5kdc[29492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.49.65@KRBTEST.COM for oryx/127.20.49.126@KRBTEST.COM, Server not found in Kerberos database
Dec 09 08:59:14 dist-test-slave-8w6c krb5kdc[29492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.49.65@KRBTEST.COM for oryx/127.20.49.126@KRBTEST.COM, Server not found in Kerberos database
I20251209 08:59:14.719172 30522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:59:14.713909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.49.126:44103 (local address 127.20.49.65:37563)
1209 08:59:14.714267 (+ 358us) negotiation.cc:107] Waiting for socket to connect
1209 08:59:14.714281 (+ 14us) client_negotiation.cc:174] Beginning negotiation
1209 08:59:14.714389 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1209 08:59:14.714747 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1209 08:59:14.714749 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1209 08:59:14.714998 (+ 249us) client_negotiation.cc:189] Negotiated authn=SASL
1209 08:59:14.715214 (+ 216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:59:14.715221 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:59:14.716027 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1209 08:59:14.716031 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1209 08:59:14.716420 (+ 389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:59:14.716427 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:59:14.716516 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:59:14.717152 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1209 08:59:14.717165 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
1209 08:59:14.719056 (+ 1891us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.49.126:44103: Server oryx/127.20.49.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":111,"threads_started":1}
W20251209 08:59:14.719334 30523 negotiation.cc:337] Failed RPC negotiation. Trace:
1209 08:59:14.713982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.49.65:37563 (local address 127.20.49.126:44103)
1209 08:59:14.714271 (+ 289us) server_negotiation.cc:207] Beginning negotiation
1209 08:59:14.714275 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1209 08:59:14.714349 (+ 74us) server_negotiation.cc:408] Connection header received
1209 08:59:14.714546 (+ 197us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1209 08:59:14.714549 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1209 08:59:14.714599 (+ 50us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1209 08:59:14.714696 (+ 97us) server_negotiation.cc:227] Negotiated authn=SASL
1209 08:59:14.715348 (+ 652us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:59:14.715914 (+ 566us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1209 08:59:14.716534 (+ 620us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:59:14.716730 (+ 196us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:59:14.719185 (+ 2455us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.49.65:37563: BlockingRecv error: recv got EOF from 127.20.49.65:37563 (error 108)
Metrics: {"server-negotiator.queue_time_us":193,"thread_start_us":90,"threads_started":1}
Dec 09 08:59:14 dist-test-slave-8w6c krb5kdc[29492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.49.66@KRBTEST.COM for oryx/127.20.49.126@KRBTEST.COM, Server not found in Kerberos database
Dec 09 08:59:14 dist-test-slave-8w6c krb5kdc[29492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.49.66@KRBTEST.COM for oryx/127.20.49.126@KRBTEST.COM, Server not found in Kerberos database
I20251209 08:59:14.902896 30524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:59:14.897995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.49.126:44103 (local address 127.20.49.66:41867)
1209 08:59:14.898313 (+ 318us) negotiation.cc:107] Waiting for socket to connect
1209 08:59:14.898331 (+ 18us) client_negotiation.cc:174] Beginning negotiation
1209 08:59:14.898440 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1209 08:59:14.898675 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1209 08:59:14.898677 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1209 08:59:14.898890 (+ 213us) client_negotiation.cc:189] Negotiated authn=SASL
1209 08:59:14.899091 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:59:14.899097 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:59:14.899864 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1209 08:59:14.899866 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1209 08:59:14.900244 (+ 378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:59:14.900250 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:59:14.900343 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:59:14.900912 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1209 08:59:14.900926 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
1209 08:59:14.902737 (+ 1811us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.49.126:44103: Server oryx/127.20.49.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":121,"threads_started":1}
W20251209 08:59:14.903092 30523 negotiation.cc:337] Failed RPC negotiation. Trace:
1209 08:59:14.898106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.49.66:41867 (local address 127.20.49.126:44103)
1209 08:59:14.898243 (+ 137us) server_negotiation.cc:207] Beginning negotiation
1209 08:59:14.898247 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1209 08:59:14.898429 (+ 182us) server_negotiation.cc:408] Connection header received
1209 08:59:14.898530 (+ 101us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1209 08:59:14.898533 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1209 08:59:14.898584 (+ 51us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1209 08:59:14.898649 (+ 65us) server_negotiation.cc:227] Negotiated authn=SASL
1209 08:59:14.899219 (+ 570us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:59:14.899762 (+ 543us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1209 08:59:14.900368 (+ 606us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:59:14.900525 (+ 157us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:59:14.902947 (+ 2422us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.49.66:41867: BlockingRecv error: recv got EOF from 127.20.49.66:41867 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Dec 09 08:59:15 dist-test-slave-8w6c krb5kdc[29492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.49.67@KRBTEST.COM for oryx/127.20.49.126@KRBTEST.COM, Server not found in Kerberos database
Dec 09 08:59:15 dist-test-slave-8w6c krb5kdc[29492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.49.67@KRBTEST.COM for oryx/127.20.49.126@KRBTEST.COM, Server not found in Kerberos database
I20251209 08:59:15.097326 30525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:59:15.092145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.49.126:44103 (local address 127.20.49.67:49623)
1209 08:59:15.092432 (+ 287us) negotiation.cc:107] Waiting for socket to connect
1209 08:59:15.092446 (+ 14us) client_negotiation.cc:174] Beginning negotiation
1209 08:59:15.092549 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1209 08:59:15.092833 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1209 08:59:15.092836 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1209 08:59:15.093082 (+ 246us) client_negotiation.cc:189] Negotiated authn=SASL
1209 08:59:15.093323 (+ 241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:59:15.093335 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:59:15.094186 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1209 08:59:15.094190 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1209 08:59:15.094590 (+ 400us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:59:15.094598 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:59:15.094693 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:59:15.095275 (+ 582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1209 08:59:15.095288 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
1209 08:59:15.097208 (+ 1920us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.49.126:44103: Server oryx/127.20.49.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":100,"threads_started":1}
W20251209 08:59:15.097476 30523 negotiation.cc:337] Failed RPC negotiation. Trace:
1209 08:59:15.092265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.49.67:49623 (local address 127.20.49.126:44103)
1209 08:59:15.092405 (+ 140us) server_negotiation.cc:207] Beginning negotiation
1209 08:59:15.092409 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1209 08:59:15.092515 (+ 106us) server_negotiation.cc:408] Connection header received
1209 08:59:15.092630 (+ 115us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1209 08:59:15.092633 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1209 08:59:15.092678 (+ 45us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1209 08:59:15.092753 (+ 75us) server_negotiation.cc:227] Negotiated authn=SASL
1209 08:59:15.093455 (+ 702us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:59:15.094054 (+ 599us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1209 08:59:15.094691 (+ 637us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:59:15.094820 (+ 129us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:59:15.097341 (+ 2521us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.49.67:49623: BlockingRecv error: recv got EOF from 127.20.49.67:49623 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Dec 09 08:59:15 dist-test-slave-8w6c krb5kdc[29492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.49.65@KRBTEST.COM for oryx/127.20.49.126@KRBTEST.COM, Server not found in Kerberos database
Dec 09 08:59:15 dist-test-slave-8w6c krb5kdc[29492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.49.65@KRBTEST.COM for oryx/127.20.49.126@KRBTEST.COM, Server not found in Kerberos database
I20251209 08:59:15.724937 30526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:59:15.719994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.49.126:44103 (local address 127.20.49.65:54089)
1209 08:59:15.720334 (+ 340us) negotiation.cc:107] Waiting for socket to connect
1209 08:59:15.720349 (+ 15us) client_negotiation.cc:174] Beginning negotiation
1209 08:59:15.720467 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1209 08:59:15.720765 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1209 08:59:15.720768 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1209 08:59:15.720960 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
1209 08:59:15.721175 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:59:15.721182 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:59:15.721993 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1209 08:59:15.721997 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1209 08:59:15.722412 (+ 415us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:59:15.722420 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:59:15.722522 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:59:15.723119 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1209 08:59:15.723132 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
1209 08:59:15.724820 (+ 1688us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.49.126:44103: Server oryx/127.20.49.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":96,"threads_started":1}
W20251209 08:59:15.725119 30527 negotiation.cc:337] Failed RPC negotiation. Trace:
1209 08:59:15.720132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.49.65:54089 (local address 127.20.49.126:44103)
1209 08:59:15.720372 (+ 240us) server_negotiation.cc:207] Beginning negotiation
1209 08:59:15.720376 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1209 08:59:15.720432 (+ 56us) server_negotiation.cc:408] Connection header received
1209 08:59:15.720545 (+ 113us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1209 08:59:15.720548 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1209 08:59:15.720592 (+ 44us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1209 08:59:15.720671 (+ 79us) server_negotiation.cc:227] Negotiated authn=SASL
1209 08:59:15.721286 (+ 615us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:59:15.721885 (+ 599us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1209 08:59:15.722521 (+ 636us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:59:15.722654 (+ 133us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:59:15.724969 (+ 2315us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.49.65:54089: BlockingRecv error: recv got EOF from 127.20.49.65:54089 (error 108)
Metrics: {"server-negotiator.queue_time_us":170,"thread_start_us":62,"threads_started":1}
Dec 09 08:59:15 dist-test-slave-8w6c krb5kdc[29492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.49.66@KRBTEST.COM for oryx/127.20.49.126@KRBTEST.COM, Server not found in Kerberos database
Dec 09 08:59:15 dist-test-slave-8w6c krb5kdc[29492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.49.66@KRBTEST.COM for oryx/127.20.49.126@KRBTEST.COM, Server not found in Kerberos database
I20251209 08:59:15.909116 30528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:59:15.903924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.49.126:44103 (local address 127.20.49.66:55445)
1209 08:59:15.904253 (+ 329us) negotiation.cc:107] Waiting for socket to connect
1209 08:59:15.904270 (+ 17us) client_negotiation.cc:174] Beginning negotiation
1209 08:59:15.904447 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1209 08:59:15.904731 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1209 08:59:15.904734 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1209 08:59:15.904946 (+ 212us) client_negotiation.cc:189] Negotiated authn=SASL
1209 08:59:15.905157 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:59:15.905163 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:59:15.906013 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1209 08:59:15.906016 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1209 08:59:15.906461 (+ 445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:59:15.906469 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:59:15.906580 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:59:15.907201 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1209 08:59:15.907216 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
1209 08:59:15.908987 (+ 1771us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.49.126:44103: Server oryx/127.20.49.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":112,"threads_started":1}
W20251209 08:59:15.909323 30527 negotiation.cc:337] Failed RPC negotiation. Trace:
1209 08:59:15.904052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.49.66:55445 (local address 127.20.49.126:44103)
1209 08:59:15.904242 (+ 190us) server_negotiation.cc:207] Beginning negotiation
1209 08:59:15.904247 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1209 08:59:15.904377 (+ 130us) server_negotiation.cc:408] Connection header received
1209 08:59:15.904556 (+ 179us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1209 08:59:15.904559 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1209 08:59:15.904613 (+ 54us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1209 08:59:15.904690 (+ 77us) server_negotiation.cc:227] Negotiated authn=SASL
1209 08:59:15.905286 (+ 596us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:59:15.905888 (+ 602us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1209 08:59:15.906570 (+ 682us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:59:15.906790 (+ 220us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:59:15.909123 (+ 2333us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.49.66:55445: BlockingRecv error: recv got EOF from 127.20.49.66:55445 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
Dec 09 08:59:16 dist-test-slave-8w6c krb5kdc[29492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.49.67@KRBTEST.COM for oryx/127.20.49.126@KRBTEST.COM, Server not found in Kerberos database
Dec 09 08:59:16 dist-test-slave-8w6c krb5kdc[29492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.49.67@KRBTEST.COM for oryx/127.20.49.126@KRBTEST.COM, Server not found in Kerberos database
I20251209 08:59:16.103000 30529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:59:16.098064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.49.126:44103 (local address 127.20.49.67:38099)
1209 08:59:16.098369 (+ 305us) negotiation.cc:107] Waiting for socket to connect
1209 08:59:16.098384 (+ 15us) client_negotiation.cc:174] Beginning negotiation
1209 08:59:16.098498 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1209 08:59:16.098754 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1209 08:59:16.098757 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1209 08:59:16.098972 (+ 215us) client_negotiation.cc:189] Negotiated authn=SASL
1209 08:59:16.099170 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:59:16.099177 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:59:16.099919 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1209 08:59:16.099922 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1209 08:59:16.100304 (+ 382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:59:16.100311 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:59:16.100422 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:59:16.100999 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1209 08:59:16.101012 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
1209 08:59:16.102806 (+ 1794us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.49.126:44103: Server oryx/127.20.49.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":82,"threads_started":1}
W20251209 08:59:16.103094 30527 negotiation.cc:337] Failed RPC negotiation. Trace:
1209 08:59:16.098161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.49.67:38099 (local address 127.20.49.126:44103)
1209 08:59:16.098299 (+ 138us) server_negotiation.cc:207] Beginning negotiation
1209 08:59:16.098304 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1209 08:59:16.098447 (+ 143us) server_negotiation.cc:408] Connection header received
1209 08:59:16.098588 (+ 141us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1209 08:59:16.098592 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1209 08:59:16.098641 (+ 49us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1209 08:59:16.098711 (+ 70us) server_negotiation.cc:227] Negotiated authn=SASL
1209 08:59:16.099310 (+ 599us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:59:16.099795 (+ 485us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1209 08:59:16.100413 (+ 618us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:59:16.100621 (+ 208us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:59:16.102977 (+ 2356us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.49.67:38099: BlockingRecv error: recv got EOF from 127.20.49.67:38099 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Dec 09 08:59:16 dist-test-slave-8w6c krb5kdc[29492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.49.65@KRBTEST.COM for oryx/127.20.49.126@KRBTEST.COM, Server not found in Kerberos database
Dec 09 08:59:16 dist-test-slave-8w6c krb5kdc[29492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.49.65@KRBTEST.COM for oryx/127.20.49.126@KRBTEST.COM, Server not found in Kerberos database
I20251209 08:59:16.731238 30530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:59:16.725786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.49.126:44103 (local address 127.20.49.65:56469)
1209 08:59:16.726109 (+ 323us) negotiation.cc:107] Waiting for socket to connect
1209 08:59:16.726128 (+ 19us) client_negotiation.cc:174] Beginning negotiation
1209 08:59:16.726240 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1209 08:59:16.726478 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1209 08:59:16.726481 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1209 08:59:16.726667 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
1209 08:59:16.726918 (+ 251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:59:16.726924 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:59:16.728013 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1209 08:59:16.728019 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1209 08:59:16.728466 (+ 447us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:59:16.728473 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:59:16.728587 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:59:16.729296 (+ 709us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1209 08:59:16.729310 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
1209 08:59:16.731088 (+ 1778us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.49.126:44103: Server oryx/127.20.49.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":92,"threads_started":1}
W20251209 08:59:16.731355 30531 negotiation.cc:337] Failed RPC negotiation. Trace:
1209 08:59:16.725978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.49.65:56469 (local address 127.20.49.126:44103)
1209 08:59:16.726220 (+ 242us) server_negotiation.cc:207] Beginning negotiation
1209 08:59:16.726224 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1209 08:59:16.726239 (+ 15us) server_negotiation.cc:408] Connection header received
1209 08:59:16.726322 (+ 83us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1209 08:59:16.726325 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1209 08:59:16.726370 (+ 45us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1209 08:59:16.726458 (+ 88us) server_negotiation.cc:227] Negotiated authn=SASL
1209 08:59:16.727099 (+ 641us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:59:16.727901 (+ 802us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1209 08:59:16.728606 (+ 705us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:59:16.728828 (+ 222us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:59:16.731232 (+ 2404us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.49.65:56469: BlockingRecv error: recv got EOF from 127.20.49.65:56469 (error 108)
Metrics: {"server-negotiator.queue_time_us":168,"thread_start_us":77,"threads_started":1}
Dec 09 08:59:16 dist-test-slave-8w6c krb5kdc[29492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.49.66@KRBTEST.COM for oryx/127.20.49.126@KRBTEST.COM, Server not found in Kerberos database
Dec 09 08:59:16 dist-test-slave-8w6c krb5kdc[29492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.49.66@KRBTEST.COM for oryx/127.20.49.126@KRBTEST.COM, Server not found in Kerberos database
I20251209 08:59:16.914999 30532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:59:16.909859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.49.126:44103 (local address 127.20.49.66:55593)
1209 08:59:16.910167 (+ 308us) negotiation.cc:107] Waiting for socket to connect
1209 08:59:16.910185 (+ 18us) client_negotiation.cc:174] Beginning negotiation
1209 08:59:16.910311 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1209 08:59:16.910564 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1209 08:59:16.910568 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
1209 08:59:16.910803 (+ 235us) client_negotiation.cc:189] Negotiated authn=SASL
1209 08:59:16.910999 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:59:16.911004 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:59:16.911929 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1209 08:59:16.911934 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1209 08:59:16.912353 (+ 419us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:59:16.912361 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:59:16.912476 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:59:16.913075 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1209 08:59:16.913088 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
1209 08:59:16.914880 (+ 1792us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.49.126:44103: Server oryx/127.20.49.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":94,"threads_started":1}
W20251209 08:59:16.915139 30531 negotiation.cc:337] Failed RPC negotiation. Trace:
1209 08:59:16.910002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.49.66:55593 (local address 127.20.49.126:44103)
1209 08:59:16.910168 (+ 166us) server_negotiation.cc:207] Beginning negotiation
1209 08:59:16.910174 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1209 08:59:16.910262 (+ 88us) server_negotiation.cc:408] Connection header received
1209 08:59:16.910403 (+ 141us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1209 08:59:16.910407 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1209 08:59:16.910457 (+ 50us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1209 08:59:16.910528 (+ 71us) server_negotiation.cc:227] Negotiated authn=SASL
1209 08:59:16.911130 (+ 602us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:59:16.911815 (+ 685us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1209 08:59:16.912475 (+ 660us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:59:16.912692 (+ 217us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:59:16.915021 (+ 2329us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.49.66:55593: BlockingRecv error: recv got EOF from 127.20.49.66:55593 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
Dec 09 08:59:17 dist-test-slave-8w6c krb5kdc[29492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.49.67@KRBTEST.COM for oryx/127.20.49.126@KRBTEST.COM, Server not found in Kerberos database
Dec 09 08:59:17 dist-test-slave-8w6c krb5kdc[29492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.49.67@KRBTEST.COM for oryx/127.20.49.126@KRBTEST.COM, Server not found in Kerberos database
I20251209 08:59:17.108969 30533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:59:17.103998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.49.126:44103 (local address 127.20.49.67:52341)
1209 08:59:17.104297 (+ 299us) negotiation.cc:107] Waiting for socket to connect
1209 08:59:17.104314 (+ 17us) client_negotiation.cc:174] Beginning negotiation
1209 08:59:17.104456 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1209 08:59:17.104731 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1209 08:59:17.104733 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1209 08:59:17.104931 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
1209 08:59:17.105133 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:59:17.105140 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:59:17.106008 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1209 08:59:17.106012 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1209 08:59:17.106436 (+ 424us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:59:17.106445 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:59:17.106542 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:59:17.107123 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1209 08:59:17.107137 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
1209 08:59:17.108855 (+ 1718us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.49.126:44103: Server oryx/127.20.49.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":100,"threads_started":1}
W20251209 08:59:17.109205 30531 negotiation.cc:337] Failed RPC negotiation. Trace:
1209 08:59:17.104039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.49.67:52341 (local address 127.20.49.126:44103)
1209 08:59:17.104217 (+ 178us) server_negotiation.cc:207] Beginning negotiation
1209 08:59:17.104222 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1209 08:59:17.104406 (+ 184us) server_negotiation.cc:408] Connection header received
1209 08:59:17.104557 (+ 151us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1209 08:59:17.104561 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1209 08:59:17.104617 (+ 56us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1209 08:59:17.104701 (+ 84us) server_negotiation.cc:227] Negotiated authn=SASL
1209 08:59:17.105259 (+ 558us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:59:17.105881 (+ 622us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1209 08:59:17.106579 (+ 698us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:59:17.106771 (+ 192us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:59:17.109002 (+ 2231us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.49.67:52341: BlockingRecv error: recv got EOF from 127.20.49.67:52341 (error 108)
Metrics: {"server-negotiator.queue_time_us":89}
Dec 09 08:59:17 dist-test-slave-8w6c krb5kdc[29492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.49.65@KRBTEST.COM for oryx/127.20.49.126@KRBTEST.COM, Server not found in Kerberos database
Dec 09 08:59:17 dist-test-slave-8w6c krb5kdc[29492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.49.65@KRBTEST.COM for oryx/127.20.49.126@KRBTEST.COM, Server not found in Kerberos database
I20251209 08:59:17.737044 30534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:59:17.732102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.49.126:44103 (local address 127.20.49.65:35405)
1209 08:59:17.732482 (+ 380us) negotiation.cc:107] Waiting for socket to connect
1209 08:59:17.732496 (+ 14us) client_negotiation.cc:174] Beginning negotiation
1209 08:59:17.732605 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1209 08:59:17.732853 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1209 08:59:17.732855 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1209 08:59:17.733057 (+ 202us) client_negotiation.cc:189] Negotiated authn=SASL
1209 08:59:17.733258 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:59:17.733264 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:59:17.734006 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1209 08:59:17.734010 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1209 08:59:17.734411 (+ 401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:59:17.734418 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:59:17.734511 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:59:17.735151 (+ 640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1209 08:59:17.735164 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
1209 08:59:17.736929 (+ 1765us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.49.126:44103: Server oryx/127.20.49.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":305,"thread_start_us":82,"threads_started":1}
W20251209 08:59:17.737179 30535 negotiation.cc:337] Failed RPC negotiation. Trace:
1209 08:59:17.732178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.49.65:35405 (local address 127.20.49.126:44103)
1209 08:59:17.732422 (+ 244us) server_negotiation.cc:207] Beginning negotiation
1209 08:59:17.732426 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1209 08:59:17.732563 (+ 137us) server_negotiation.cc:408] Connection header received
1209 08:59:17.732699 (+ 136us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1209 08:59:17.732701 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1209 08:59:17.732748 (+ 47us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1209 08:59:17.732825 (+ 77us) server_negotiation.cc:227] Negotiated authn=SASL
1209 08:59:17.733355 (+ 530us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:59:17.733900 (+ 545us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1209 08:59:17.734504 (+ 604us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:59:17.734648 (+ 144us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:59:17.737057 (+ 2409us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.49.65:35405: BlockingRecv error: recv got EOF from 127.20.49.65:35405 (error 108)
Metrics: {"server-negotiator.queue_time_us":172,"thread_start_us":73,"threads_started":1}
Dec 09 08:59:17 dist-test-slave-8w6c krb5kdc[29492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.49.66@KRBTEST.COM for oryx/127.20.49.126@KRBTEST.COM, Server not found in Kerberos database
Dec 09 08:59:17 dist-test-slave-8w6c krb5kdc[29492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.49.66@KRBTEST.COM for oryx/127.20.49.126@KRBTEST.COM, Server not found in Kerberos database
I20251209 08:59:17.920650 30536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:59:17.915778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.49.126:44103 (local address 127.20.49.66:40015)
1209 08:59:17.916047 (+ 269us) negotiation.cc:107] Waiting for socket to connect
1209 08:59:17.916060 (+ 13us) client_negotiation.cc:174] Beginning negotiation
1209 08:59:17.916169 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1209 08:59:17.916383 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1209 08:59:17.916385 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1209 08:59:17.916579 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
1209 08:59:17.916785 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:59:17.916792 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:59:17.917513 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1209 08:59:17.917515 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1209 08:59:17.917922 (+ 407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:59:17.917929 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:59:17.918015 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:59:17.918605 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1209 08:59:17.918620 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
1209 08:59:17.920532 (+ 1912us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.49.126:44103: Server oryx/127.20.49.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":86,"threads_started":1}
W20251209 08:59:17.920833 30535 negotiation.cc:337] Failed RPC negotiation. Trace:
1209 08:59:17.915875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.49.66:40015 (local address 127.20.49.126:44103)
1209 08:59:17.916003 (+ 128us) server_negotiation.cc:207] Beginning negotiation
1209 08:59:17.916007 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1209 08:59:17.916125 (+ 118us) server_negotiation.cc:408] Connection header received
1209 08:59:17.916243 (+ 118us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1209 08:59:17.916245 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1209 08:59:17.916289 (+ 44us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1209 08:59:17.916359 (+ 70us) server_negotiation.cc:227] Negotiated authn=SASL
1209 08:59:17.916909 (+ 550us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:59:17.917420 (+ 511us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1209 08:59:17.918061 (+ 641us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:59:17.918221 (+ 160us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:59:17.920687 (+ 2466us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.49.66:40015: BlockingRecv error: recv got EOF from 127.20.49.66:40015 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Dec 09 08:59:18 dist-test-slave-8w6c krb5kdc[29492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.49.67@KRBTEST.COM for oryx/127.20.49.126@KRBTEST.COM, Server not found in Kerberos database
Dec 09 08:59:18 dist-test-slave-8w6c krb5kdc[29492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.49.67@KRBTEST.COM for oryx/127.20.49.126@KRBTEST.COM, Server not found in Kerberos database
I20251209 08:59:18.119638 30537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:59:18.110005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.49.126:44103 (local address 127.20.49.67:47373)
1209 08:59:18.110321 (+ 316us) negotiation.cc:107] Waiting for socket to connect
1209 08:59:18.110334 (+ 13us) client_negotiation.cc:174] Beginning negotiation
1209 08:59:18.110438 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1209 08:59:18.110688 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1209 08:59:18.110690 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1209 08:59:18.110890 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
1209 08:59:18.111085 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:59:18.111091 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:59:18.111753 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1209 08:59:18.111755 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1209 08:59:18.112153 (+ 398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:59:18.112160 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:59:18.112250 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:59:18.117513 (+ 5263us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1209 08:59:18.117529 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
1209 08:59:18.119519 (+ 1990us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.49.126:44103: Server oryx/127.20.49.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":111,"threads_started":1}
W20251209 08:59:18.119791 30535 negotiation.cc:337] Failed RPC negotiation. Trace:
1209 08:59:18.110082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.49.67:47373 (local address 127.20.49.126:44103)
1209 08:59:18.110233 (+ 151us) server_negotiation.cc:207] Beginning negotiation
1209 08:59:18.110236 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1209 08:59:18.110400 (+ 164us) server_negotiation.cc:408] Connection header received
1209 08:59:18.110523 (+ 123us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1209 08:59:18.110525 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1209 08:59:18.110572 (+ 47us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1209 08:59:18.110648 (+ 76us) server_negotiation.cc:227] Negotiated authn=SASL
1209 08:59:18.111185 (+ 537us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:59:18.111663 (+ 478us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1209 08:59:18.112249 (+ 586us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:59:18.112381 (+ 132us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:59:18.119668 (+ 7287us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.49.67:47373: BlockingRecv error: recv got EOF from 127.20.49.67:47373 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Dec 09 08:59:18 dist-test-slave-8w6c krb5kdc[29492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.49.65@KRBTEST.COM for oryx/127.20.49.126@KRBTEST.COM, Server not found in Kerberos database
Dec 09 08:59:18 dist-test-slave-8w6c krb5kdc[29492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.49.65@KRBTEST.COM for oryx/127.20.49.126@KRBTEST.COM, Server not found in Kerberos database
I20251209 08:59:18.742851 30538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:59:18.737828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.49.126:44103 (local address 127.20.49.65:50557)
1209 08:59:18.738139 (+ 311us) negotiation.cc:107] Waiting for socket to connect
1209 08:59:18.738154 (+ 15us) client_negotiation.cc:174] Beginning negotiation
1209 08:59:18.738252 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1209 08:59:18.738488 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1209 08:59:18.738492 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
1209 08:59:18.738683 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
1209 08:59:18.738937 (+ 254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:59:18.738944 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:59:18.739681 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1209 08:59:18.739685 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1209 08:59:18.740117 (+ 432us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:59:18.740125 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:59:18.740224 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:59:18.740786 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1209 08:59:18.740809 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
1209 08:59:18.742711 (+ 1902us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.49.126:44103: Server oryx/127.20.49.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":95,"threads_started":1}
W20251209 08:59:18.742988 30539 negotiation.cc:337] Failed RPC negotiation. Trace:
1209 08:59:18.737952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.49.65:50557 (local address 127.20.49.126:44103)
1209 08:59:18.738231 (+ 279us) server_negotiation.cc:207] Beginning negotiation
1209 08:59:18.738235 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1209 08:59:18.738248 (+ 13us) server_negotiation.cc:408] Connection header received
1209 08:59:18.738334 (+ 86us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1209 08:59:18.738337 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1209 08:59:18.738379 (+ 42us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1209 08:59:18.738457 (+ 78us) server_negotiation.cc:227] Negotiated authn=SASL
1209 08:59:18.739051 (+ 594us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:59:18.739564 (+ 513us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1209 08:59:18.740218 (+ 654us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:59:18.740348 (+ 130us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:59:18.742861 (+ 2513us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.49.65:50557: BlockingRecv error: recv got EOF from 127.20.49.65:50557 (error 108)
Metrics: {"server-negotiator.queue_time_us":203,"thread_start_us":97,"threads_started":1}
Dec 09 08:59:18 dist-test-slave-8w6c krb5kdc[29492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.49.66@KRBTEST.COM for oryx/127.20.49.126@KRBTEST.COM, Server not found in Kerberos database
Dec 09 08:59:18 dist-test-slave-8w6c krb5kdc[29492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.49.66@KRBTEST.COM for oryx/127.20.49.126@KRBTEST.COM, Server not found in Kerberos database
I20251209 08:59:18.926702 30540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:59:18.921573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.49.126:44103 (local address 127.20.49.66:47121)
1209 08:59:18.921918 (+ 345us) negotiation.cc:107] Waiting for socket to connect
1209 08:59:18.921932 (+ 14us) client_negotiation.cc:174] Beginning negotiation
1209 08:59:18.922054 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1209 08:59:18.922369 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1209 08:59:18.922373 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
1209 08:59:18.922581 (+ 208us) client_negotiation.cc:189] Negotiated authn=SASL
1209 08:59:18.922821 (+ 240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:59:18.922830 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:59:18.923650 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1209 08:59:18.923653 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1209 08:59:18.924062 (+ 409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:59:18.924070 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:59:18.924167 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:59:18.924747 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1209 08:59:18.924760 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
1209 08:59:18.926582 (+ 1822us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.49.126:44103: Server oryx/127.20.49.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":133,"threads_started":1}
W20251209 08:59:18.926818 30539 negotiation.cc:337] Failed RPC negotiation. Trace:
1209 08:59:18.921724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.49.66:47121 (local address 127.20.49.126:44103)
1209 08:59:18.921883 (+ 159us) server_negotiation.cc:207] Beginning negotiation
1209 08:59:18.921887 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1209 08:59:18.922012 (+ 125us) server_negotiation.cc:408] Connection header received
1209 08:59:18.922168 (+ 156us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1209 08:59:18.922174 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
1209 08:59:18.922241 (+ 67us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1209 08:59:18.922331 (+ 90us) server_negotiation.cc:227] Negotiated authn=SASL
1209 08:59:18.922964 (+ 633us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:59:18.923547 (+ 583us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1209 08:59:18.924213 (+ 666us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:59:18.924378 (+ 165us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:59:18.926717 (+ 2339us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.49.66:47121: BlockingRecv error: recv got EOF from 127.20.49.66:47121 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Dec 09 08:59:19 dist-test-slave-8w6c krb5kdc[29492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.49.67@KRBTEST.COM for oryx/127.20.49.126@KRBTEST.COM, Server not found in Kerberos database
Dec 09 08:59:19 dist-test-slave-8w6c krb5kdc[29492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.49.67@KRBTEST.COM for oryx/127.20.49.126@KRBTEST.COM, Server not found in Kerberos database
I20251209 08:59:19.125770 30541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:59:19.120524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.49.126:44103 (local address 127.20.49.67:53365)
1209 08:59:19.120823 (+ 299us) negotiation.cc:107] Waiting for socket to connect
1209 08:59:19.120837 (+ 14us) client_negotiation.cc:174] Beginning negotiation
1209 08:59:19.120939 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1209 08:59:19.121212 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1209 08:59:19.121214 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1209 08:59:19.121397 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
1209 08:59:19.121609 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:59:19.121614 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:59:19.122376 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1209 08:59:19.122379 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1209 08:59:19.122758 (+ 379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:59:19.122764 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:59:19.122860 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:59:19.123487 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1209 08:59:19.123500 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
1209 08:59:19.125573 (+ 2073us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.49.126:44103: Server oryx/127.20.49.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":85,"threads_started":1}
W20251209 08:59:19.125835 30539 negotiation.cc:337] Failed RPC negotiation. Trace:
1209 08:59:19.120696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.49.67:53365 (local address 127.20.49.126:44103)
1209 08:59:19.120829 (+ 133us) server_negotiation.cc:207] Beginning negotiation
1209 08:59:19.120832 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1209 08:59:19.120904 (+ 72us) server_negotiation.cc:408] Connection header received
1209 08:59:19.121039 (+ 135us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1209 08:59:19.121042 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1209 08:59:19.121099 (+ 57us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1209 08:59:19.121165 (+ 66us) server_negotiation.cc:227] Negotiated authn=SASL
1209 08:59:19.121735 (+ 570us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:59:19.122267 (+ 532us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1209 08:59:19.122868 (+ 601us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:59:19.123007 (+ 139us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:59:19.125725 (+ 2718us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.49.67:53365: BlockingRecv error: recv got EOF from 127.20.49.67:53365 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Dec 09 08:59:19 dist-test-slave-8w6c krb5kdc[29492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.49.65@KRBTEST.COM for oryx/127.20.49.126@KRBTEST.COM, Server not found in Kerberos database
Dec 09 08:59:19 dist-test-slave-8w6c krb5kdc[29492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.49.65@KRBTEST.COM for oryx/127.20.49.126@KRBTEST.COM, Server not found in Kerberos database
I20251209 08:59:19.748944 30542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:59:19.743734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.49.126:44103 (local address 127.20.49.65:33871)
1209 08:59:19.744032 (+ 298us) negotiation.cc:107] Waiting for socket to connect
1209 08:59:19.744046 (+ 14us) client_negotiation.cc:174] Beginning negotiation
1209 08:59:19.744147 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1209 08:59:19.744408 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1209 08:59:19.744410 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1209 08:59:19.744606 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
1209 08:59:19.744834 (+ 228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:59:19.744840 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:59:19.745576 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1209 08:59:19.745579 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1209 08:59:19.746033 (+ 454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:59:19.746041 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:59:19.746140 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:59:19.746736 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1209 08:59:19.746755 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
1209 08:59:19.748807 (+ 2052us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.49.126:44103: Server oryx/127.20.49.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":98,"threads_started":1}
W20251209 08:59:19.749122 30543 negotiation.cc:337] Failed RPC negotiation. Trace:
1209 08:59:19.743911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.49.65:33871 (local address 127.20.49.126:44103)
1209 08:59:19.744176 (+ 265us) server_negotiation.cc:207] Beginning negotiation
1209 08:59:19.744180 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1209 08:59:19.744196 (+ 16us) server_negotiation.cc:408] Connection header received
1209 08:59:19.744236 (+ 40us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1209 08:59:19.744238 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1209 08:59:19.744282 (+ 44us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1209 08:59:19.744361 (+ 79us) server_negotiation.cc:227] Negotiated authn=SASL
1209 08:59:19.744947 (+ 586us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:59:19.745478 (+ 531us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1209 08:59:19.746152 (+ 674us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:59:19.746290 (+ 138us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:59:19.748958 (+ 2668us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.49.65:33871: BlockingRecv error: recv got EOF from 127.20.49.65:33871 (error 108)
Metrics: {"server-negotiator.queue_time_us":181,"thread_start_us":85,"threads_started":1}
Dec 09 08:59:19 dist-test-slave-8w6c krb5kdc[29492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.49.66@KRBTEST.COM for oryx/127.20.49.126@KRBTEST.COM, Server not found in Kerberos database
Dec 09 08:59:19 dist-test-slave-8w6c krb5kdc[29492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.49.66@KRBTEST.COM for oryx/127.20.49.126@KRBTEST.COM, Server not found in Kerberos database
I20251209 08:59:19.932435 30544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:59:19.927509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.49.126:44103 (local address 127.20.49.66:48673)
1209 08:59:19.927793 (+ 284us) negotiation.cc:107] Waiting for socket to connect
1209 08:59:19.927809 (+ 16us) client_negotiation.cc:174] Beginning negotiation
1209 08:59:19.927947 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1209 08:59:19.928223 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1209 08:59:19.928226 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1209 08:59:19.928438 (+ 212us) client_negotiation.cc:189] Negotiated authn=SASL
1209 08:59:19.928632 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:59:19.928638 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:59:19.929389 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1209 08:59:19.929392 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1209 08:59:19.929804 (+ 412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:59:19.929811 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:59:19.929933 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:59:19.930526 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1209 08:59:19.930539 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
1209 08:59:19.932320 (+ 1781us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.49.126:44103: Server oryx/127.20.49.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":90,"threads_started":1}
W20251209 08:59:19.932605 30543 negotiation.cc:337] Failed RPC negotiation. Trace:
1209 08:59:19.927592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.49.66:48673 (local address 127.20.49.126:44103)
1209 08:59:19.927744 (+ 152us) server_negotiation.cc:207] Beginning negotiation
1209 08:59:19.927749 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1209 08:59:19.927883 (+ 134us) server_negotiation.cc:408] Connection header received
1209 08:59:19.928051 (+ 168us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1209 08:59:19.928054 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1209 08:59:19.928103 (+ 49us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1209 08:59:19.928191 (+ 88us) server_negotiation.cc:227] Negotiated authn=SASL
1209 08:59:19.928765 (+ 574us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:59:19.929274 (+ 509us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1209 08:59:19.929924 (+ 650us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:59:19.930140 (+ 216us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:59:19.932465 (+ 2325us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.49.66:48673: BlockingRecv error: recv got EOF from 127.20.49.66:48673 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Dec 09 08:59:20 dist-test-slave-8w6c krb5kdc[29492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.49.67@KRBTEST.COM for oryx/127.20.49.126@KRBTEST.COM, Server not found in Kerberos database
Dec 09 08:59:20 dist-test-slave-8w6c krb5kdc[29492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.49.67@KRBTEST.COM for oryx/127.20.49.126@KRBTEST.COM, Server not found in Kerberos database
I20251209 08:59:20.131795 30545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:59:20.126660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.49.126:44103 (local address 127.20.49.67:52369)
1209 08:59:20.127062 (+ 402us) negotiation.cc:107] Waiting for socket to connect
1209 08:59:20.127079 (+ 17us) client_negotiation.cc:174] Beginning negotiation
1209 08:59:20.127192 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1209 08:59:20.127466 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1209 08:59:20.127469 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1209 08:59:20.127674 (+ 205us) client_negotiation.cc:189] Negotiated authn=SASL
1209 08:59:20.127864 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:59:20.127870 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:59:20.128611 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1209 08:59:20.128615 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1209 08:59:20.129003 (+ 388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:59:20.129010 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:59:20.129124 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:59:20.129837 (+ 713us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1209 08:59:20.129850 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
1209 08:59:20.131643 (+ 1793us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.49.126:44103: Server oryx/127.20.49.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":312,"thread_start_us":108,"threads_started":1}
W20251209 08:59:20.131903 30543 negotiation.cc:337] Failed RPC negotiation. Trace:
1209 08:59:20.126737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.49.67:52369 (local address 127.20.49.126:44103)
1209 08:59:20.126916 (+ 179us) server_negotiation.cc:207] Beginning negotiation
1209 08:59:20.126921 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1209 08:59:20.127135 (+ 214us) server_negotiation.cc:408] Connection header received
1209 08:59:20.127299 (+ 164us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1209 08:59:20.127302 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1209 08:59:20.127354 (+ 52us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1209 08:59:20.127423 (+ 69us) server_negotiation.cc:227] Negotiated authn=SASL
1209 08:59:20.127993 (+ 570us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:59:20.128496 (+ 503us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1209 08:59:20.129124 (+ 628us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:59:20.129359 (+ 235us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:59:20.131791 (+ 2432us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.49.67:52369: BlockingRecv error: recv got EOF from 127.20.49.67:52369 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Dec 09 08:59:20 dist-test-slave-8w6c krb5kdc[29492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.49.65@KRBTEST.COM for oryx/127.20.49.126@KRBTEST.COM, Server not found in Kerberos database
Dec 09 08:59:20 dist-test-slave-8w6c krb5kdc[29492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.49.65@KRBTEST.COM for oryx/127.20.49.126@KRBTEST.COM, Server not found in Kerberos database
I20251209 08:59:20.754979 30546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:59:20.749878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.49.126:44103 (local address 127.20.49.65:45835)
1209 08:59:20.750170 (+ 292us) negotiation.cc:107] Waiting for socket to connect
1209 08:59:20.750188 (+ 18us) client_negotiation.cc:174] Beginning negotiation
1209 08:59:20.750320 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1209 08:59:20.750655 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1209 08:59:20.750658 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1209 08:59:20.750910 (+ 252us) client_negotiation.cc:189] Negotiated authn=SASL
1209 08:59:20.751144 (+ 234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:59:20.751152 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:59:20.751877 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1209 08:59:20.751880 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1209 08:59:20.752289 (+ 409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:59:20.752296 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:59:20.752390 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:59:20.752979 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1209 08:59:20.752993 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
1209 08:59:20.754827 (+ 1834us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.49.126:44103: Server oryx/127.20.49.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":87,"threads_started":1}
W20251209 08:59:20.755110 30547 negotiation.cc:337] Failed RPC negotiation. Trace:
1209 08:59:20.749917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.49.65:45835 (local address 127.20.49.126:44103)
1209 08:59:20.750194 (+ 277us) server_negotiation.cc:207] Beginning negotiation
1209 08:59:20.750199 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1209 08:59:20.750253 (+ 54us) server_negotiation.cc:408] Connection header received
1209 08:59:20.750454 (+ 201us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1209 08:59:20.750458 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1209 08:59:20.750514 (+ 56us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1209 08:59:20.750622 (+ 108us) server_negotiation.cc:227] Negotiated authn=SASL
1209 08:59:20.751257 (+ 635us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:59:20.751771 (+ 514us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1209 08:59:20.752378 (+ 607us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:59:20.752503 (+ 125us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:59:20.754987 (+ 2484us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.49.65:45835: BlockingRecv error: recv got EOF from 127.20.49.65:45835 (error 108)
Metrics: {"server-negotiator.queue_time_us":186,"thread_start_us":77,"threads_started":1}
Dec 09 08:59:20 dist-test-slave-8w6c krb5kdc[29492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.49.66@KRBTEST.COM for oryx/127.20.49.126@KRBTEST.COM, Server not found in Kerberos database
Dec 09 08:59:20 dist-test-slave-8w6c krb5kdc[29492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.49.66@KRBTEST.COM for oryx/127.20.49.126@KRBTEST.COM, Server not found in Kerberos database
I20251209 08:59:20.938438 30548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:59:20.933236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.49.126:44103 (local address 127.20.49.66:55123)
1209 08:59:20.933551 (+ 315us) negotiation.cc:107] Waiting for socket to connect
1209 08:59:20.933564 (+ 13us) client_negotiation.cc:174] Beginning negotiation
1209 08:59:20.933667 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1209 08:59:20.933951 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1209 08:59:20.933954 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1209 08:59:20.934159 (+ 205us) client_negotiation.cc:189] Negotiated authn=SASL
1209 08:59:20.934353 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:59:20.934358 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:59:20.935113 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1209 08:59:20.935117 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1209 08:59:20.935515 (+ 398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:59:20.935523 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:59:20.935628 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:59:20.936260 (+ 632us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1209 08:59:20.936274 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
1209 08:59:20.938322 (+ 2048us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.49.126:44103: Server oryx/127.20.49.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":118,"threads_started":1}
W20251209 08:59:20.938627 30547 negotiation.cc:337] Failed RPC negotiation. Trace:
1209 08:59:20.933400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.49.66:55123 (local address 127.20.49.126:44103)
1209 08:59:20.933575 (+ 175us) server_negotiation.cc:207] Beginning negotiation
1209 08:59:20.933579 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1209 08:59:20.933640 (+ 61us) server_negotiation.cc:408] Connection header received
1209 08:59:20.933781 (+ 141us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1209 08:59:20.933783 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1209 08:59:20.933829 (+ 46us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1209 08:59:20.933906 (+ 77us) server_negotiation.cc:227] Negotiated authn=SASL
1209 08:59:20.934458 (+ 552us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:59:20.934966 (+ 508us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1209 08:59:20.935630 (+ 664us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:59:20.935772 (+ 142us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:59:20.938505 (+ 2733us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.49.66:55123: BlockingRecv error: recv got EOF from 127.20.49.66:55123 (error 108)
Metrics: {"server-negotiator.queue_time_us":87}
Dec 09 08:59:21 dist-test-slave-8w6c krb5kdc[29492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.49.67@KRBTEST.COM for oryx/127.20.49.126@KRBTEST.COM, Server not found in Kerberos database
Dec 09 08:59:21 dist-test-slave-8w6c krb5kdc[29492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.49.67@KRBTEST.COM for oryx/127.20.49.126@KRBTEST.COM, Server not found in Kerberos database
I20251209 08:59:21.137717 30549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:59:21.132693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.49.126:44103 (local address 127.20.49.67:48011)
1209 08:59:21.133020 (+ 327us) negotiation.cc:107] Waiting for socket to connect
1209 08:59:21.133033 (+ 13us) client_negotiation.cc:174] Beginning negotiation
1209 08:59:21.133139 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1209 08:59:21.133368 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1209 08:59:21.133370 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1209 08:59:21.133563 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
1209 08:59:21.133851 (+ 288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:59:21.133860 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:59:21.134578 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1209 08:59:21.134581 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1209 08:59:21.134995 (+ 414us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:59:21.135002 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:59:21.135098 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:59:21.135690 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1209 08:59:21.135703 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
1209 08:59:21.137563 (+ 1860us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.49.126:44103: Server oryx/127.20.49.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":95,"threads_started":1}
W20251209 08:59:21.137869 30547 negotiation.cc:337] Failed RPC negotiation. Trace:
1209 08:59:21.132756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.49.67:48011 (local address 127.20.49.126:44103)
1209 08:59:21.132894 (+ 138us) server_negotiation.cc:207] Beginning negotiation
1209 08:59:21.132898 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1209 08:59:21.133098 (+ 200us) server_negotiation.cc:408] Connection header received
1209 08:59:21.133222 (+ 124us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1209 08:59:21.133225 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1209 08:59:21.133272 (+ 47us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1209 08:59:21.133341 (+ 69us) server_negotiation.cc:227] Negotiated authn=SASL
1209 08:59:21.133979 (+ 638us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:59:21.134476 (+ 497us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1209 08:59:21.135093 (+ 617us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:59:21.135225 (+ 132us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:59:21.137731 (+ 2506us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.49.67:48011: BlockingRecv error: recv got EOF from 127.20.49.67:48011 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Dec 09 08:59:21 dist-test-slave-8w6c krb5kdc[29492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.49.65@KRBTEST.COM for oryx/127.20.49.126@KRBTEST.COM, Server not found in Kerberos database
Dec 09 08:59:21 dist-test-slave-8w6c krb5kdc[29492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.49.65@KRBTEST.COM for oryx/127.20.49.126@KRBTEST.COM, Server not found in Kerberos database
I20251209 08:59:21.761193 30550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:59:21.755894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.49.126:44103 (local address 127.20.49.65:40611)
1209 08:59:21.756180 (+ 286us) negotiation.cc:107] Waiting for socket to connect
1209 08:59:21.756194 (+ 14us) client_negotiation.cc:174] Beginning negotiation
1209 08:59:21.756291 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1209 08:59:21.756666 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1209 08:59:21.756669 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1209 08:59:21.756867 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
1209 08:59:21.757072 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:59:21.757078 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:59:21.757892 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1209 08:59:21.757896 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1209 08:59:21.758303 (+ 407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:59:21.758312 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:59:21.758418 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:59:21.759059 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1209 08:59:21.759073 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
1209 08:59:21.761062 (+ 1989us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.49.126:44103: Server oryx/127.20.49.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":83,"threads_started":1}
W20251209 08:59:21.761327 30551 negotiation.cc:337] Failed RPC negotiation. Trace:
1209 08:59:21.756008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.49.65:40611 (local address 127.20.49.126:44103)
1209 08:59:21.756249 (+ 241us) server_negotiation.cc:207] Beginning negotiation
1209 08:59:21.756253 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1209 08:59:21.756268 (+ 15us) server_negotiation.cc:408] Connection header received
1209 08:59:21.756422 (+ 154us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1209 08:59:21.756425 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1209 08:59:21.756480 (+ 55us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1209 08:59:21.756581 (+ 101us) server_negotiation.cc:227] Negotiated authn=SASL
1209 08:59:21.757206 (+ 625us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:59:21.757772 (+ 566us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1209 08:59:21.758422 (+ 650us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:59:21.758572 (+ 150us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:59:21.761211 (+ 2639us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.49.65:40611: BlockingRecv error: recv got EOF from 127.20.49.65:40611 (error 108)
Metrics: {"server-negotiator.queue_time_us":171,"thread_start_us":71,"threads_started":1}
Dec 09 08:59:21 dist-test-slave-8w6c krb5kdc[29492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.49.66@KRBTEST.COM for oryx/127.20.49.126@KRBTEST.COM, Server not found in Kerberos database
Dec 09 08:59:21 dist-test-slave-8w6c krb5kdc[29492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.49.66@KRBTEST.COM for oryx/127.20.49.126@KRBTEST.COM, Server not found in Kerberos database
I20251209 08:59:21.944212 30552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:59:21.939515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.49.126:44103 (local address 127.20.49.66:51137)
1209 08:59:21.939834 (+ 319us) negotiation.cc:107] Waiting for socket to connect
1209 08:59:21.939848 (+ 14us) client_negotiation.cc:174] Beginning negotiation
1209 08:59:21.939967 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1209 08:59:21.940214 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1209 08:59:21.940216 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1209 08:59:21.940400 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
1209 08:59:21.940593 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:59:21.940599 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:59:21.941282 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1209 08:59:21.941285 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1209 08:59:21.941653 (+ 368us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:59:21.941659 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:59:21.941800 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:59:21.942353 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1209 08:59:21.942366 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
1209 08:59:21.944084 (+ 1718us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.49.126:44103: Server oryx/127.20.49.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":92,"threads_started":1}
W20251209 08:59:21.944341 30551 negotiation.cc:337] Failed RPC negotiation. Trace:
1209 08:59:21.939674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.49.66:51137 (local address 127.20.49.126:44103)
1209 08:59:21.939831 (+ 157us) server_negotiation.cc:207] Beginning negotiation
1209 08:59:21.939835 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1209 08:59:21.939922 (+ 87us) server_negotiation.cc:408] Connection header received
1209 08:59:21.940047 (+ 125us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1209 08:59:21.940049 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1209 08:59:21.940094 (+ 45us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1209 08:59:21.940170 (+ 76us) server_negotiation.cc:227] Negotiated authn=SASL
1209 08:59:21.940687 (+ 517us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:59:21.941197 (+ 510us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1209 08:59:21.941792 (+ 595us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:59:21.941942 (+ 150us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:59:21.944232 (+ 2290us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.49.66:51137: BlockingRecv error: recv got EOF from 127.20.49.66:51137 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Dec 09 08:59:22 dist-test-slave-8w6c krb5kdc[29492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.49.67@KRBTEST.COM for oryx/127.20.49.126@KRBTEST.COM, Server not found in Kerberos database
Dec 09 08:59:22 dist-test-slave-8w6c krb5kdc[29492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.49.67@KRBTEST.COM for oryx/127.20.49.126@KRBTEST.COM, Server not found in Kerberos database
I20251209 08:59:22.143165 30553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:59:22.138496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.49.126:44103 (local address 127.20.49.67:58187)
1209 08:59:22.138783 (+ 287us) negotiation.cc:107] Waiting for socket to connect
1209 08:59:22.138797 (+ 14us) client_negotiation.cc:174] Beginning negotiation
1209 08:59:22.138921 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1209 08:59:22.139148 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1209 08:59:22.139150 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1209 08:59:22.139342 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
1209 08:59:22.139535 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:59:22.139541 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:59:22.140235 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1209 08:59:22.140238 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1209 08:59:22.140601 (+ 363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:59:22.140607 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:59:22.140691 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:59:22.141253 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1209 08:59:22.141265 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
1209 08:59:22.143049 (+ 1784us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.49.126:44103: Server oryx/127.20.49.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":80,"threads_started":1}
W20251209 08:59:22.143294 30551 negotiation.cc:337] Failed RPC negotiation. Trace:
1209 08:59:22.138561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.49.67:58187 (local address 127.20.49.126:44103)
1209 08:59:22.138695 (+ 134us) server_negotiation.cc:207] Beginning negotiation
1209 08:59:22.138699 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1209 08:59:22.138886 (+ 187us) server_negotiation.cc:408] Connection header received
1209 08:59:22.138998 (+ 112us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1209 08:59:22.139001 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1209 08:59:22.139047 (+ 46us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1209 08:59:22.139120 (+ 73us) server_negotiation.cc:227] Negotiated authn=SASL
1209 08:59:22.139632 (+ 512us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:59:22.140130 (+ 498us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1209 08:59:22.140679 (+ 549us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:59:22.140801 (+ 122us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:59:22.143184 (+ 2383us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.49.67:58187: BlockingRecv error: recv got EOF from 127.20.49.67:58187 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Dec 09 08:59:22 dist-test-slave-8w6c krb5kdc[29492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.49.65@KRBTEST.COM for oryx/127.20.49.126@KRBTEST.COM, Server not found in Kerberos database
Dec 09 08:59:22 dist-test-slave-8w6c krb5kdc[29492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.49.65@KRBTEST.COM for oryx/127.20.49.126@KRBTEST.COM, Server not found in Kerberos database
I20251209 08:59:22.766625 30554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:59:22.761991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.49.126:44103 (local address 127.20.49.65:57863)
1209 08:59:22.762306 (+ 315us) negotiation.cc:107] Waiting for socket to connect
1209 08:59:22.762320 (+ 14us) client_negotiation.cc:174] Beginning negotiation
1209 08:59:22.762413 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1209 08:59:22.762658 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1209 08:59:22.762660 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1209 08:59:22.762846 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
1209 08:59:22.763046 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:59:22.763051 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:59:22.763775 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1209 08:59:22.763777 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1209 08:59:22.764161 (+ 384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:59:22.764167 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:59:22.764254 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:59:22.764780 (+ 526us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1209 08:59:22.764793 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
1209 08:59:22.766510 (+ 1717us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.49.126:44103: Server oryx/127.20.49.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":123,"threads_started":1}
W20251209 08:59:22.766780 30555 negotiation.cc:337] Failed RPC negotiation. Trace:
1209 08:59:22.762095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.49.65:57863 (local address 127.20.49.126:44103)
1209 08:59:22.762370 (+ 275us) server_negotiation.cc:207] Beginning negotiation
1209 08:59:22.762373 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1209 08:59:22.762387 (+ 14us) server_negotiation.cc:408] Connection header received
1209 08:59:22.762498 (+ 111us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1209 08:59:22.762501 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1209 08:59:22.762546 (+ 45us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1209 08:59:22.762625 (+ 79us) server_negotiation.cc:227] Negotiated authn=SASL
1209 08:59:22.763157 (+ 532us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:59:22.763694 (+ 537us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1209 08:59:22.764242 (+ 548us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:59:22.764372 (+ 130us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:59:22.766650 (+ 2278us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.49.65:57863: BlockingRecv error: recv got EOF from 127.20.49.65:57863 (error 108)
Metrics: {"server-negotiator.queue_time_us":207,"thread_start_us":61,"threads_started":1}
Dec 09 08:59:22 dist-test-slave-8w6c krb5kdc[29492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.49.66@KRBTEST.COM for oryx/127.20.49.126@KRBTEST.COM, Server not found in Kerberos database
Dec 09 08:59:22 dist-test-slave-8w6c krb5kdc[29492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.49.66@KRBTEST.COM for oryx/127.20.49.126@KRBTEST.COM, Server not found in Kerberos database
I20251209 08:59:22.950178 30556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:59:22.945185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.49.126:44103 (local address 127.20.49.66:43613)
1209 08:59:22.945448 (+ 263us) negotiation.cc:107] Waiting for socket to connect
1209 08:59:22.945461 (+ 13us) client_negotiation.cc:174] Beginning negotiation
1209 08:59:22.945558 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1209 08:59:22.945836 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1209 08:59:22.945838 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1209 08:59:22.946028 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
1209 08:59:22.946239 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:59:22.946245 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:59:22.947014 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1209 08:59:22.947018 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1209 08:59:22.947434 (+ 416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:59:22.947442 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:59:22.947554 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:59:22.948182 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1209 08:59:22.948196 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
1209 08:59:22.950057 (+ 1861us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.49.126:44103: Server oryx/127.20.49.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":84,"threads_started":1}
W20251209 08:59:22.950310 30555 negotiation.cc:337] Failed RPC negotiation. Trace:
1209 08:59:22.945248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.49.66:43613 (local address 127.20.49.126:44103)
1209 08:59:22.945378 (+ 130us) server_negotiation.cc:207] Beginning negotiation
1209 08:59:22.945381 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1209 08:59:22.945510 (+ 129us) server_negotiation.cc:408] Connection header received
1209 08:59:22.945652 (+ 142us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1209 08:59:22.945654 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1209 08:59:22.945725 (+ 71us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1209 08:59:22.945785 (+ 60us) server_negotiation.cc:227] Negotiated authn=SASL
1209 08:59:22.946380 (+ 595us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:59:22.946880 (+ 500us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1209 08:59:22.947553 (+ 673us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:59:22.947700 (+ 147us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:59:22.950206 (+ 2506us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.49.66:43613: BlockingRecv error: recv got EOF from 127.20.49.66:43613 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Dec 09 08:59:23 dist-test-slave-8w6c krb5kdc[29492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.49.67@KRBTEST.COM for oryx/127.20.49.126@KRBTEST.COM, Server not found in Kerberos database
Dec 09 08:59:23 dist-test-slave-8w6c krb5kdc[29492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.49.67@KRBTEST.COM for oryx/127.20.49.126@KRBTEST.COM, Server not found in Kerberos database
I20251209 08:59:23.151510 30557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:59:23.143910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.49.126:44103 (local address 127.20.49.67:37297)
1209 08:59:23.144173 (+ 263us) negotiation.cc:107] Waiting for socket to connect
1209 08:59:23.144186 (+ 13us) client_negotiation.cc:174] Beginning negotiation
1209 08:59:23.144288 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1209 08:59:23.144500 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1209 08:59:23.144502 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1209 08:59:23.144683 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
1209 08:59:23.144880 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:59:23.144886 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:59:23.145540 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1209 08:59:23.145542 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1209 08:59:23.146009 (+ 467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:59:23.146018 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:59:23.146114 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:59:23.149520 (+ 3406us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1209 08:59:23.149534 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
1209 08:59:23.151388 (+ 1854us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.49.126:44103: Server oryx/127.20.49.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":85,"threads_started":1}
W20251209 08:59:23.151650 30555 negotiation.cc:337] Failed RPC negotiation. Trace:
1209 08:59:23.143987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.49.67:37297 (local address 127.20.49.126:44103)
1209 08:59:23.144114 (+ 127us) server_negotiation.cc:207] Beginning negotiation
1209 08:59:23.144118 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1209 08:59:23.144250 (+ 132us) server_negotiation.cc:408] Connection header received
1209 08:59:23.144361 (+ 111us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1209 08:59:23.144364 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1209 08:59:23.144409 (+ 45us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1209 08:59:23.144479 (+ 70us) server_negotiation.cc:227] Negotiated authn=SASL
1209 08:59:23.144974 (+ 495us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:59:23.145457 (+ 483us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1209 08:59:23.146115 (+ 658us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:59:23.146254 (+ 139us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:59:23.151536 (+ 5282us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.49.67:37297: BlockingRecv error: recv got EOF from 127.20.49.67:37297 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Dec 09 08:59:23 dist-test-slave-8w6c krb5kdc[29492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.49.65@KRBTEST.COM for oryx/127.20.49.126@KRBTEST.COM, Server not found in Kerberos database
Dec 09 08:59:23 dist-test-slave-8w6c krb5kdc[29492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.49.65@KRBTEST.COM for oryx/127.20.49.126@KRBTEST.COM, Server not found in Kerberos database
I20251209 08:59:23.772468 30558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:59:23.767495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.49.126:44103 (local address 127.20.49.65:46899)
1209 08:59:23.767764 (+ 269us) negotiation.cc:107] Waiting for socket to connect
1209 08:59:23.767778 (+ 14us) client_negotiation.cc:174] Beginning negotiation
1209 08:59:23.767867 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1209 08:59:23.768177 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1209 08:59:23.768179 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1209 08:59:23.768366 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
1209 08:59:23.768565 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:59:23.768571 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:59:23.769366 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1209 08:59:23.769370 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1209 08:59:23.769858 (+ 488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:59:23.769868 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:59:23.769973 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:59:23.770590 (+ 617us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1209 08:59:23.770603 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
1209 08:59:23.772353 (+ 1750us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.49.126:44103: Server oryx/127.20.49.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":81,"threads_started":1}
W20251209 08:59:23.772625 30559 negotiation.cc:337] Failed RPC negotiation. Trace:
1209 08:59:23.767565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.49.65:46899 (local address 127.20.49.126:44103)
1209 08:59:23.767808 (+ 243us) server_negotiation.cc:207] Beginning negotiation
1209 08:59:23.767812 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1209 08:59:23.767827 (+ 15us) server_negotiation.cc:408] Connection header received
1209 08:59:23.767962 (+ 135us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1209 08:59:23.767965 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1209 08:59:23.768012 (+ 47us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1209 08:59:23.768095 (+ 83us) server_negotiation.cc:227] Negotiated authn=SASL
1209 08:59:23.768684 (+ 589us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:59:23.769239 (+ 555us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1209 08:59:23.769994 (+ 755us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:59:23.770144 (+ 150us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:59:23.772497 (+ 2353us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.49.65:46899: BlockingRecv error: recv got EOF from 127.20.49.65:46899 (error 108)
Metrics: {"server-negotiator.queue_time_us":176,"thread_start_us":77,"threads_started":1}
Dec 09 08:59:23 dist-test-slave-8w6c krb5kdc[29492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.49.66@KRBTEST.COM for oryx/127.20.49.126@KRBTEST.COM, Server not found in Kerberos database
Dec 09 08:59:23 dist-test-slave-8w6c krb5kdc[29492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.49.66@KRBTEST.COM for oryx/127.20.49.126@KRBTEST.COM, Server not found in Kerberos database
I20251209 08:59:23.956455 30560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:59:23.951318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.49.126:44103 (local address 127.20.49.66:44037)
1209 08:59:23.951627 (+ 309us) negotiation.cc:107] Waiting for socket to connect
1209 08:59:23.951643 (+ 16us) client_negotiation.cc:174] Beginning negotiation
1209 08:59:23.951766 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1209 08:59:23.952233 (+ 467us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1209 08:59:23.952236 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1209 08:59:23.952421 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
1209 08:59:23.952615 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:59:23.952620 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:59:23.953383 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1209 08:59:23.953387 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1209 08:59:23.953809 (+ 422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:59:23.953817 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:59:23.953934 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:59:23.954502 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1209 08:59:23.954515 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
1209 08:59:23.956343 (+ 1828us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.49.126:44103: Server oryx/127.20.49.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":98,"threads_started":1}
W20251209 08:59:23.956596 30559 negotiation.cc:337] Failed RPC negotiation. Trace:
1209 08:59:23.951471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.49.66:44037 (local address 127.20.49.126:44103)
1209 08:59:23.951630 (+ 159us) server_negotiation.cc:207] Beginning negotiation
1209 08:59:23.951637 (+ 7us) server_negotiation.cc:400] Waiting for connection header
1209 08:59:23.951726 (+ 89us) server_negotiation.cc:408] Connection header received
1209 08:59:23.951867 (+ 141us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1209 08:59:23.951870 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1209 08:59:23.951927 (+ 57us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1209 08:59:23.952008 (+ 81us) server_negotiation.cc:227] Negotiated authn=SASL
1209 08:59:23.952742 (+ 734us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:59:23.953272 (+ 530us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1209 08:59:23.953935 (+ 663us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:59:23.954163 (+ 228us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:59:23.956483 (+ 2320us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.49.66:44037: BlockingRecv error: recv got EOF from 127.20.49.66:44037 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
I20251209 08:59:24.044047 20677 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task8XuDZV/build/debug/bin/kudu with pid 30084
I20251209 08:59:24.050345 20677 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task8XuDZV/build/debug/bin/kudu with pid 30236
I20251209 08:59:24.056250 20677 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task8XuDZV/build/debug/bin/kudu with pid 30368
I20251209 08:59:24.062031 20677 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task8XuDZV/build/debug/bin/kudu with pid 30014
2025-12-09T08:59:24Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (19023 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2025-12-09T08:59:24Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-12-09T08:59:24Z Disabled control of system clock
I20251209 08:59:24.093189 20677 external_mini_cluster.cc:1366] Running /tmp/dist-test-task8XuDZV/build/debug/bin/kudu
/tmp/dist-test-task8XuDZV/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1765270668420570-20677-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1765270668420570-20677-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1765270668420570-20677-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1765270668420570-20677-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.49.126:40409
--webserver_interface=127.20.49.126
--webserver_port=0
--builtin_ntp_servers=127.20.49.84:39353
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.49.126:40409
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20251209 08:59:24.203718 30564 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251209 08:59:24.203997 30564 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251209 08:59:24.204049 30564 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251209 08:59:24.207388 30564 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251209 08:59:24.207458 30564 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251209 08:59:24.207475 30564 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251209 08:59:24.207497 30564 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251209 08:59:24.207533 30564 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251209 08:59:24.211402 30564 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.49.84:39353
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1765270668420570-20677-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1765270668420570-20677-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.49.126:40409
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.49.126:40409
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1765270668420570-20677-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.49.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30564
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1765270668420570-20677-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 468897755969d686fb08386f42f7835685e7953a
build type DEBUG
built by None at 09 Dec 2025 08:43:15 UTC on 24a791456cd2
build id 9426
I20251209 08:59:24.212548 30564 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251209 08:59:24.213701 30564 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251209 08:59:24.219695 30569 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251209 08:59:24.219758 30570 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251209 08:59:24.219816 30564 server_base.cc:1047] running on GCE node
W20251209 08:59:24.219695 30572 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251209 08:59:24.220558 30564 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251209 08:59:24.221487 30564 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251209 08:59:24.222637 30564 hybrid_clock.cc:648] HybridClock initialized: now 1765270764222623 us; error 39 us; skew 500 ppm
I20251209 08:59:24.224543 30564 webserver.cc:492] Webserver started at http://127.20.49.126:46461/ using document root <none> and password file <none>
I20251209 08:59:24.225142 30564 fs_manager.cc:362] Metadata directory not provided
I20251209 08:59:24.225203 30564 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251209 08:59:24.225399 30564 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251209 08:59:24.227200 30564 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1765270668420570-20677-0/minicluster-data/master-0/data/instance:
uuid: "b592c544e00b46308c3818c0899f7aec"
format_stamp: "Formatted at 2025-12-09 08:59:24 on dist-test-slave-8w6c"
server_key: "6b8bf1a7b1205b77936448c1253d082b"
server_key_iv: "3385b37445e9d7a104cb46bf752b31fa"
server_key_version: "encryptionkey@0"
I20251209 08:59:24.227710 30564 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1765270668420570-20677-0/minicluster-data/master-0/wal/instance:
uuid: "b592c544e00b46308c3818c0899f7aec"
format_stamp: "Formatted at 2025-12-09 08:59:24 on dist-test-slave-8w6c"
server_key: "6b8bf1a7b1205b77936448c1253d082b"
server_key_iv: "3385b37445e9d7a104cb46bf752b31fa"
server_key_version: "encryptionkey@0"
I20251209 08:59:24.231195 30564 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20251209 08:59:24.233464 30578 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251209 08:59:24.234556 30564 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20251209 08:59:24.234681 30564 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1765270668420570-20677-0/minicluster-data/master-0/data,/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1765270668420570-20677-0/minicluster-data/master-0/wal
uuid: "b592c544e00b46308c3818c0899f7aec"
format_stamp: "Formatted at 2025-12-09 08:59:24 on dist-test-slave-8w6c"
server_key: "6b8bf1a7b1205b77936448c1253d082b"
server_key_iv: "3385b37445e9d7a104cb46bf752b31fa"
server_key_version: "encryptionkey@0"
I20251209 08:59:24.234781 30564 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1765270668420570-20677-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1765270668420570-20677-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1765270668420570-20677-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251209 08:59:24.246910 30564 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251209 08:59:24.247570 30564 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251209 08:59:24.247748 30564 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251209 08:59:24.255273 30630 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.49.126:40409 every 8 connection(s)
I20251209 08:59:24.255270 30564 rpc_server.cc:307] RPC server started. Bound to: 127.20.49.126:40409
I20251209 08:59:24.256317 30564 server_base.cc:1179] Dumped server information to /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1765270668420570-20677-0/minicluster-data/master-0/data/info.pb
I20251209 08:59:24.258522 20677 external_mini_cluster.cc:1428] Started /tmp/dist-test-task8XuDZV/build/debug/bin/kudu as pid 30564
I20251209 08:59:24.258666 20677 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1765270668420570-20677-0/minicluster-data/master-0/wal/instance
I20251209 08:59:24.258931 20677 external_mini_cluster.cc:1468] Setting key 41a1db8d9b0a715db94e62eb0f172201
I20251209 08:59:24.259407 30631 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251209 08:59:24.265954 30631 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b592c544e00b46308c3818c0899f7aec: Bootstrap starting.
I20251209 08:59:24.266860 30634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:59:24.260134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59974 (local address 127.20.49.126:40409)
1209 08:59:24.260712 (+ 578us) server_negotiation.cc:207] Beginning negotiation
1209 08:59:24.260721 (+ 9us) server_negotiation.cc:400] Waiting for connection header
1209 08:59:24.260755 (+ 34us) server_negotiation.cc:408] Connection header received
1209 08:59:24.261390 (+ 635us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1209 08:59:24.261406 (+ 16us) server_negotiation.cc:462] Received NEGOTIATE request from client
1209 08:59:24.261788 (+ 382us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1209 08:59:24.262075 (+ 287us) server_negotiation.cc:227] Negotiated authn=SASL
1209 08:59:24.262695 (+ 620us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:59:24.263566 (+ 871us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1209 08:59:24.264479 (+ 913us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:59:24.264683 (+ 204us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:59:24.265297 (+ 614us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1209 08:59:24.265340 (+ 43us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1209 08:59:24.265349 (+ 9us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1209 08:59:24.265764 (+ 415us) server_negotiation.cc:923] calling sasl_server_start()
1209 08:59:24.265805 (+ 41us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1209 08:59:24.265819 (+ 14us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1209 08:59:24.265904 (+ 85us) server_negotiation.cc:1036] Waiting for connection context
1209 08:59:24.266081 (+ 177us) server_negotiation.cc:300] Negotiation successful
1209 08:59:24.266300 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":253,"thread_start_us":120,"threads_started":1}
I20251209 08:59:24.268114 30631 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b592c544e00b46308c3818c0899f7aec: Neither blocks nor log segments found. Creating new log.
I20251209 08:59:24.268819 30631 log.cc:826] T 00000000000000000000000000000000 P b592c544e00b46308c3818c0899f7aec: Log is configured to *not* fsync() on all Append() calls
I20251209 08:59:24.270722 30631 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b592c544e00b46308c3818c0899f7aec: No bootstrap required, opened a new log
I20251209 08:59:24.273170 30631 raft_consensus.cc:359] T 00000000000000000000000000000000 P b592c544e00b46308c3818c0899f7aec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b592c544e00b46308c3818c0899f7aec" member_type: VOTER last_known_addr { host: "127.20.49.126" port: 40409 } }
I20251209 08:59:24.273411 30631 raft_consensus.cc:385] T 00000000000000000000000000000000 P b592c544e00b46308c3818c0899f7aec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251209 08:59:24.273511 30631 raft_consensus.cc:740] T 00000000000000000000000000000000 P b592c544e00b46308c3818c0899f7aec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b592c544e00b46308c3818c0899f7aec, State: Initialized, Role: FOLLOWER
I20251209 08:59:24.273999 30631 consensus_queue.cc:260] T 00000000000000000000000000000000 P b592c544e00b46308c3818c0899f7aec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b592c544e00b46308c3818c0899f7aec" member_type: VOTER last_known_addr { host: "127.20.49.126" port: 40409 } }
I20251209 08:59:24.274133 30631 raft_consensus.cc:399] T 00000000000000000000000000000000 P b592c544e00b46308c3818c0899f7aec [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251209 08:59:24.274195 30631 raft_consensus.cc:493] T 00000000000000000000000000000000 P b592c544e00b46308c3818c0899f7aec [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251209 08:59:24.274266 30631 raft_consensus.cc:3060] T 00000000000000000000000000000000 P b592c544e00b46308c3818c0899f7aec [term 0 FOLLOWER]: Advancing to term 1
I20251209 08:59:24.275081 30631 raft_consensus.cc:515] T 00000000000000000000000000000000 P b592c544e00b46308c3818c0899f7aec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b592c544e00b46308c3818c0899f7aec" member_type: VOTER last_known_addr { host: "127.20.49.126" port: 40409 } }
I20251209 08:59:24.275386 30631 leader_election.cc:304] T 00000000000000000000000000000000 P b592c544e00b46308c3818c0899f7aec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b592c544e00b46308c3818c0899f7aec; no voters:
I20251209 08:59:24.275632 30631 leader_election.cc:290] T 00000000000000000000000000000000 P b592c544e00b46308c3818c0899f7aec [CANDIDATE]: Term 1 election: Requested vote from peers
I20251209 08:59:24.275758 30636 raft_consensus.cc:2804] T 00000000000000000000000000000000 P b592c544e00b46308c3818c0899f7aec [term 1 FOLLOWER]: Leader election won for term 1
I20251209 08:59:24.275982 30636 raft_consensus.cc:697] T 00000000000000000000000000000000 P b592c544e00b46308c3818c0899f7aec [term 1 LEADER]: Becoming Leader. State: Replica: b592c544e00b46308c3818c0899f7aec, State: Running, Role: LEADER
I20251209 08:59:24.276336 30636 consensus_queue.cc:237] T 00000000000000000000000000000000 P b592c544e00b46308c3818c0899f7aec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b592c544e00b46308c3818c0899f7aec" member_type: VOTER last_known_addr { host: "127.20.49.126" port: 40409 } }
I20251209 08:59:24.276717 30631 sys_catalog.cc:565] T 00000000000000000000000000000000 P b592c544e00b46308c3818c0899f7aec [sys.catalog]: configured and running, proceeding with master startup.
I20251209 08:59:24.277971 30638 sys_catalog.cc:455] T 00000000000000000000000000000000 P b592c544e00b46308c3818c0899f7aec [sys.catalog]: SysCatalogTable state changed. Reason: New leader b592c544e00b46308c3818c0899f7aec. Latest consensus state: current_term: 1 leader_uuid: "b592c544e00b46308c3818c0899f7aec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b592c544e00b46308c3818c0899f7aec" member_type: VOTER last_known_addr { host: "127.20.49.126" port: 40409 } } }
I20251209 08:59:24.278115 30638 sys_catalog.cc:458] T 00000000000000000000000000000000 P b592c544e00b46308c3818c0899f7aec [sys.catalog]: This master's current role is: LEADER
I20251209 08:59:24.278462 30637 sys_catalog.cc:455] T 00000000000000000000000000000000 P b592c544e00b46308c3818c0899f7aec [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b592c544e00b46308c3818c0899f7aec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b592c544e00b46308c3818c0899f7aec" member_type: VOTER last_known_addr { host: "127.20.49.126" port: 40409 } } }
I20251209 08:59:24.278520 30645 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251209 08:59:24.278534 30637 sys_catalog.cc:458] T 00000000000000000000000000000000 P b592c544e00b46308c3818c0899f7aec [sys.catalog]: This master's current role is: LEADER
I20251209 08:59:24.281643 30645 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251209 08:59:24.286785 30645 catalog_manager.cc:1357] Generated new cluster ID: 5840adf790234511922756a8d67ba789
I20251209 08:59:24.286899 30645 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251209 08:59:24.299749 30645 catalog_manager.cc:1380] Generated new certificate authority record
I20251209 08:59:24.300915 30645 catalog_manager.cc:1514] Loading token signing keys...
I20251209 08:59:24.309103 30645 catalog_manager.cc:6027] T 00000000000000000000000000000000 P b592c544e00b46308c3818c0899f7aec: Generated new TSK 0
I20251209 08:59:24.309837 30645 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251209 08:59:24.313658 20677 external_mini_cluster.cc:1366] Running /tmp/dist-test-task8XuDZV/build/debug/bin/kudu
/tmp/dist-test-task8XuDZV/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1765270668420570-20677-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1765270668420570-20677-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1765270668420570-20677-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1765270668420570-20677-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.49.65:0
--local_ip_for_outbound_sockets=127.20.49.65
--webserver_interface=127.20.49.65
--webserver_port=0
--tserver_master_addrs=127.20.49.126:40409
--builtin_ntp_servers=127.20.49.84:39353
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20251209 08:59:24.423520 30655 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251209 08:59:24.423738 30655 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251209 08:59:24.423786 30655 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251209 08:59:24.427151 30655 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251209 08:59:24.427209 30655 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251209 08:59:24.427300 30655 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.49.65
I20251209 08:59:24.431219 30655 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.49.84:39353
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1765270668420570-20677-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1765270668420570-20677-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.49.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1765270668420570-20677-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.49.65
--webserver_port=0
--tserver_master_addrs=127.20.49.126:40409
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30655
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.49.65
--log_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1765270668420570-20677-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 468897755969d686fb08386f42f7835685e7953a
build type DEBUG
built by None at 09 Dec 2025 08:43:15 UTC on 24a791456cd2
build id 9426
I20251209 08:59:24.432344 30655 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251209 08:59:24.433511 30655 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251209 08:59:24.440382 30663 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251209 08:59:24.440475 30660 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251209 08:59:24.440387 30661 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251209 08:59:24.440810 30655 server_base.cc:1047] running on GCE node
I20251209 08:59:24.441300 30655 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251209 08:59:24.442035 30655 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251209 08:59:24.443226 30655 hybrid_clock.cc:648] HybridClock initialized: now 1765270764443205 us; error 36 us; skew 500 ppm
I20251209 08:59:24.445449 30655 webserver.cc:492] Webserver started at http://127.20.49.65:37427/ using document root <none> and password file <none>
I20251209 08:59:24.446149 30655 fs_manager.cc:362] Metadata directory not provided
I20251209 08:59:24.446211 30655 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251209 08:59:24.446377 30655 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251209 08:59:24.448020 30655 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1765270668420570-20677-0/minicluster-data/ts-0/data/instance:
uuid: "764d4bd6629b44428625a63a1137a384"
format_stamp: "Formatted at 2025-12-09 08:59:24 on dist-test-slave-8w6c"
server_key: "fe01420c0a40e29abe17744d256595ff"
server_key_iv: "ab3f873556b091351365a6b76ec855c0"
server_key_version: "encryptionkey@0"
I20251209 08:59:24.448455 30655 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1765270668420570-20677-0/minicluster-data/ts-0/wal/instance:
uuid: "764d4bd6629b44428625a63a1137a384"
format_stamp: "Formatted at 2025-12-09 08:59:24 on dist-test-slave-8w6c"
server_key: "fe01420c0a40e29abe17744d256595ff"
server_key_iv: "ab3f873556b091351365a6b76ec855c0"
server_key_version: "encryptionkey@0"
I20251209 08:59:24.451720 30655 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20251209 08:59:24.453917 30669 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251209 08:59:24.454900 30655 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20251209 08:59:24.454999 30655 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1765270668420570-20677-0/minicluster-data/ts-0/data,/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1765270668420570-20677-0/minicluster-data/ts-0/wal
uuid: "764d4bd6629b44428625a63a1137a384"
format_stamp: "Formatted at 2025-12-09 08:59:24 on dist-test-slave-8w6c"
server_key: "fe01420c0a40e29abe17744d256595ff"
server_key_iv: "ab3f873556b091351365a6b76ec855c0"
server_key_version: "encryptionkey@0"
I20251209 08:59:24.455106 30655 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1765270668420570-20677-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1765270668420570-20677-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1765270668420570-20677-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251209 08:59:24.475188 30655 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251209 08:59:24.475865 30655 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251209 08:59:24.476051 30655 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251209 08:59:24.476645 30655 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251209 08:59:24.477650 30655 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251209 08:59:24.477730 30655 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251209 08:59:24.477788 30655 ts_tablet_manager.cc:616] Registered 0 tablets
I20251209 08:59:24.477818 30655 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251209 08:59:24.487107 30655 rpc_server.cc:307] RPC server started. Bound to: 127.20.49.65:36291
I20251209 08:59:24.487160 30782 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.49.65:36291 every 8 connection(s)
I20251209 08:59:24.488068 30655 server_base.cc:1179] Dumped server information to /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1765270668420570-20677-0/minicluster-data/ts-0/data/info.pb
I20251209 08:59:24.490033 20677 external_mini_cluster.cc:1428] Started /tmp/dist-test-task8XuDZV/build/debug/bin/kudu as pid 30655
I20251209 08:59:24.490180 20677 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1765270668420570-20677-0/minicluster-data/ts-0/wal/instance
I20251209 08:59:24.490438 20677 external_mini_cluster.cc:1468] Setting key d42b6826206ac8b0943d5e670f4fbfd5
I20251209 08:59:24.492515 20677 external_mini_cluster.cc:1366] Running /tmp/dist-test-task8XuDZV/build/debug/bin/kudu
/tmp/dist-test-task8XuDZV/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1765270668420570-20677-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1765270668420570-20677-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1765270668420570-20677-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1765270668420570-20677-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.49.66:0
--local_ip_for_outbound_sockets=127.20.49.66
--webserver_interface=127.20.49.66
--webserver_port=0
--tserver_master_addrs=127.20.49.126:40409
--builtin_ntp_servers=127.20.49.84:39353
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20251209 08:59:24.497001 30634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:59:24.489932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.49.65:33795 (local address 127.20.49.126:40409)
1209 08:59:24.490133 (+ 201us) server_negotiation.cc:207] Beginning negotiation
1209 08:59:24.490138 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1209 08:59:24.491161 (+ 1023us) server_negotiation.cc:408] Connection header received
1209 08:59:24.492098 (+ 937us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1209 08:59:24.492101 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1209 08:59:24.492148 (+ 47us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1209 08:59:24.492217 (+ 69us) server_negotiation.cc:227] Negotiated authn=SASL
1209 08:59:24.493482 (+ 1265us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:59:24.494031 (+ 549us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1209 08:59:24.494700 (+ 669us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:59:24.494885 (+ 185us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:59:24.496479 (+ 1594us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1209 08:59:24.496496 (+ 17us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1209 08:59:24.496499 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1209 08:59:24.496512 (+ 13us) server_negotiation.cc:923] calling sasl_server_start()
1209 08:59:24.496532 (+ 20us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1209 08:59:24.496540 (+ 8us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1209 08:59:24.496609 (+ 69us) server_negotiation.cc:1036] Waiting for connection context
1209 08:59:24.496820 (+ 211us) server_negotiation.cc:300] Negotiation successful
1209 08:59:24.496880 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20251209 08:59:24.498085 30785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:59:24.490256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.49.126:40409 (local address 127.20.49.65:33795)
1209 08:59:24.491005 (+ 749us) negotiation.cc:107] Waiting for socket to connect
1209 08:59:24.491055 (+ 50us) client_negotiation.cc:174] Beginning negotiation
1209 08:59:24.491891 (+ 836us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1209 08:59:24.492408 (+ 517us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1209 08:59:24.492419 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
1209 08:59:24.492521 (+ 102us) client_negotiation.cc:189] Negotiated authn=SASL
1209 08:59:24.493335 (+ 814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:59:24.493354 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:59:24.494168 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1209 08:59:24.494172 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1209 08:59:24.494570 (+ 398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:59:24.494577 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:59:24.494808 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:59:24.496177 (+ 1369us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
1209 08:59:24.496182 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
1209 08:59:24.496230 (+ 48us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
1209 08:59:24.496315 (+ 85us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1209 08:59:24.496634 (+ 319us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1209 08:59:24.496644 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1209 08:59:24.496650 (+ 6us) client_negotiation.cc:769] Sending connection context
1209 08:59:24.496826 (+ 176us) client_negotiation.cc:240] Negotiation successful
1209 08:59:24.497123 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":104,"threads_started":1}
I20251209 08:59:24.499375 30783 heartbeater.cc:344] Connected to a master server at 127.20.49.126:40409
I20251209 08:59:24.499619 30783 heartbeater.cc:461] Registering TS with master...
I20251209 08:59:24.500170 30783 heartbeater.cc:507] Master 127.20.49.126:40409 requested a full tablet report, sending...
I20251209 08:59:24.501616 30595 ts_manager.cc:194] Registered new tserver with Master: 764d4bd6629b44428625a63a1137a384 (127.20.49.65:36291)
I20251209 08:59:24.502811 30595 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.20.49.65:33795
W20251209 08:59:24.604187 30786 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251209 08:59:24.604445 30786 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251209 08:59:24.604533 30786 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251209 08:59:24.607904 30786 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251209 08:59:24.607980 30786 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251209 08:59:24.608101 30786 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.49.66
I20251209 08:59:24.612057 30786 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.49.84:39353
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1765270668420570-20677-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1765270668420570-20677-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.49.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1765270668420570-20677-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.49.66
--webserver_port=0
--tserver_master_addrs=127.20.49.126:40409
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30786
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.49.66
--log_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1765270668420570-20677-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 468897755969d686fb08386f42f7835685e7953a
build type DEBUG
built by None at 09 Dec 2025 08:43:15 UTC on 24a791456cd2
build id 9426
I20251209 08:59:24.613195 30786 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251209 08:59:24.614409 30786 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251209 08:59:24.621127 30791 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251209 08:59:24.621161 30794 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251209 08:59:24.621172 30792 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251209 08:59:24.621366 30786 server_base.cc:1047] running on GCE node
I20251209 08:59:24.621776 30786 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251209 08:59:24.622362 30786 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251209 08:59:24.623510 30786 hybrid_clock.cc:648] HybridClock initialized: now 1765270764623493 us; error 42 us; skew 500 ppm
I20251209 08:59:24.625334 30786 webserver.cc:492] Webserver started at http://127.20.49.66:37911/ using document root <none> and password file <none>
I20251209 08:59:24.625962 30786 fs_manager.cc:362] Metadata directory not provided
I20251209 08:59:24.626030 30786 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251209 08:59:24.626242 30786 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251209 08:59:24.627947 30786 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1765270668420570-20677-0/minicluster-data/ts-1/data/instance:
uuid: "0a876b26ec7f4e8599e2e8514140a573"
format_stamp: "Formatted at 2025-12-09 08:59:24 on dist-test-slave-8w6c"
server_key: "b81a260fe85c1ae938b27e4da644dd59"
server_key_iv: "be63006fd2fdd1020ca7cd358c58cb61"
server_key_version: "encryptionkey@0"
I20251209 08:59:24.628448 30786 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1765270668420570-20677-0/minicluster-data/ts-1/wal/instance:
uuid: "0a876b26ec7f4e8599e2e8514140a573"
format_stamp: "Formatted at 2025-12-09 08:59:24 on dist-test-slave-8w6c"
server_key: "b81a260fe85c1ae938b27e4da644dd59"
server_key_iv: "be63006fd2fdd1020ca7cd358c58cb61"
server_key_version: "encryptionkey@0"
I20251209 08:59:24.631939 30786 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20251209 08:59:24.634172 30800 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251209 08:59:24.635172 30786 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20251209 08:59:24.635295 30786 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1765270668420570-20677-0/minicluster-data/ts-1/data,/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1765270668420570-20677-0/minicluster-data/ts-1/wal
uuid: "0a876b26ec7f4e8599e2e8514140a573"
format_stamp: "Formatted at 2025-12-09 08:59:24 on dist-test-slave-8w6c"
server_key: "b81a260fe85c1ae938b27e4da644dd59"
server_key_iv: "be63006fd2fdd1020ca7cd358c58cb61"
server_key_version: "encryptionkey@0"
I20251209 08:59:24.635388 30786 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1765270668420570-20677-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1765270668420570-20677-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1765270668420570-20677-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251209 08:59:24.650274 30786 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251209 08:59:24.650856 30786 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251209 08:59:24.651027 30786 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251209 08:59:24.651552 30786 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251209 08:59:24.652518 30786 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251209 08:59:24.652571 30786 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251209 08:59:24.652631 30786 ts_tablet_manager.cc:616] Registered 0 tablets
I20251209 08:59:24.652665 30786 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251209 08:59:24.661664 30786 rpc_server.cc:307] RPC server started. Bound to: 127.20.49.66:37367
I20251209 08:59:24.661713 30913 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.49.66:37367 every 8 connection(s)
I20251209 08:59:24.662670 30786 server_base.cc:1179] Dumped server information to /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1765270668420570-20677-0/minicluster-data/ts-1/data/info.pb
I20251209 08:59:24.669801 20677 external_mini_cluster.cc:1428] Started /tmp/dist-test-task8XuDZV/build/debug/bin/kudu as pid 30786
I20251209 08:59:24.669977 20677 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1765270668420570-20677-0/minicluster-data/ts-1/wal/instance
I20251209 08:59:24.670291 20677 external_mini_cluster.cc:1468] Setting key 92300c25c27630c3129854678c6ef773
I20251209 08:59:24.670909 30634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:59:24.664409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.49.66:42685 (local address 127.20.49.126:40409)
1209 08:59:24.664555 (+ 146us) server_negotiation.cc:207] Beginning negotiation
1209 08:59:24.664559 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1209 08:59:24.665423 (+ 864us) server_negotiation.cc:408] Connection header received
1209 08:59:24.666314 (+ 891us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1209 08:59:24.666318 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1209 08:59:24.666383 (+ 65us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1209 08:59:24.666462 (+ 79us) server_negotiation.cc:227] Negotiated authn=SASL
1209 08:59:24.667446 (+ 984us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:59:24.667958 (+ 512us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1209 08:59:24.668581 (+ 623us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:59:24.668760 (+ 179us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:59:24.670326 (+ 1566us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1209 08:59:24.670351 (+ 25us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1209 08:59:24.670357 (+ 6us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1209 08:59:24.670376 (+ 19us) server_negotiation.cc:923] calling sasl_server_start()
1209 08:59:24.670402 (+ 26us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1209 08:59:24.670411 (+ 9us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1209 08:59:24.670484 (+ 73us) server_negotiation.cc:1036] Waiting for connection context
1209 08:59:24.670669 (+ 185us) server_negotiation.cc:300] Negotiation successful
1209 08:59:24.670744 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20251209 08:59:24.671633 30916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:59:24.664677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.49.126:40409 (local address 127.20.49.66:42685)
1209 08:59:24.665289 (+ 612us) negotiation.cc:107] Waiting for socket to connect
1209 08:59:24.665323 (+ 34us) client_negotiation.cc:174] Beginning negotiation
1209 08:59:24.666106 (+ 783us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1209 08:59:24.666630 (+ 524us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1209 08:59:24.666639 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
1209 08:59:24.666717 (+ 78us) client_negotiation.cc:189] Negotiated authn=SASL
1209 08:59:24.667300 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:59:24.667312 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:59:24.668085 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1209 08:59:24.668088 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1209 08:59:24.668475 (+ 387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:59:24.668482 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:59:24.668693 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:59:24.670050 (+ 1357us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
1209 08:59:24.670052 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
1209 08:59:24.670096 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
1209 08:59:24.670168 (+ 72us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1209 08:59:24.670494 (+ 326us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1209 08:59:24.670503 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1209 08:59:24.670509 (+ 6us) client_negotiation.cc:769] Sending connection context
1209 08:59:24.670672 (+ 163us) client_negotiation.cc:240] Negotiation successful
1209 08:59:24.670898 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":97,"threads_started":1}
I20251209 08:59:24.672323 20677 external_mini_cluster.cc:1366] Running /tmp/dist-test-task8XuDZV/build/debug/bin/kudu
/tmp/dist-test-task8XuDZV/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1765270668420570-20677-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1765270668420570-20677-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1765270668420570-20677-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1765270668420570-20677-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.49.67:0
--local_ip_for_outbound_sockets=127.20.49.67
--webserver_interface=127.20.49.67
--webserver_port=0
--tserver_master_addrs=127.20.49.126:40409
--builtin_ntp_servers=127.20.49.84:39353
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20251209 08:59:24.672750 30914 heartbeater.cc:344] Connected to a master server at 127.20.49.126:40409
I20251209 08:59:24.673018 30914 heartbeater.cc:461] Registering TS with master...
I20251209 08:59:24.673673 30914 heartbeater.cc:507] Master 127.20.49.126:40409 requested a full tablet report, sending...
I20251209 08:59:24.674840 30595 ts_manager.cc:194] Registered new tserver with Master: 0a876b26ec7f4e8599e2e8514140a573 (127.20.49.66:37367)
I20251209 08:59:24.675429 30595 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.20.49.66:42685
W20251209 08:59:24.783792 30917 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251209 08:59:24.784050 30917 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251209 08:59:24.784128 30917 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251209 08:59:24.787545 30917 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251209 08:59:24.787606 30917 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251209 08:59:24.787707 30917 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.49.67
I20251209 08:59:24.791513 30917 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.49.84:39353
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1765270668420570-20677-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1765270668420570-20677-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.49.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1765270668420570-20677-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.49.67
--webserver_port=0
--tserver_master_addrs=127.20.49.126:40409
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30917
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.49.67
--log_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1765270668420570-20677-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 468897755969d686fb08386f42f7835685e7953a
build type DEBUG
built by None at 09 Dec 2025 08:43:15 UTC on 24a791456cd2
build id 9426
I20251209 08:59:24.792598 30917 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251209 08:59:24.793733 30917 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251209 08:59:24.800096 30925 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251209 08:59:24.800107 30922 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251209 08:59:24.800356 30917 server_base.cc:1047] running on GCE node
W20251209 08:59:24.800174 30923 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251209 08:59:24.800808 30917 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251209 08:59:24.801378 30917 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251209 08:59:24.802533 30917 hybrid_clock.cc:648] HybridClock initialized: now 1765270764802522 us; error 26 us; skew 500 ppm
I20251209 08:59:24.804416 30917 webserver.cc:492] Webserver started at http://127.20.49.67:32827/ using document root <none> and password file <none>
I20251209 08:59:24.804984 30917 fs_manager.cc:362] Metadata directory not provided
I20251209 08:59:24.805050 30917 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251209 08:59:24.805256 30917 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251209 08:59:24.807009 30917 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1765270668420570-20677-0/minicluster-data/ts-2/data/instance:
uuid: "b9741ad1f88641bcb989f7ab70c4b6fa"
format_stamp: "Formatted at 2025-12-09 08:59:24 on dist-test-slave-8w6c"
server_key: "48d338b05c14e4b1e7e605fc67a9cad8"
server_key_iv: "ea052b53aea854254916a8e4a1425b8f"
server_key_version: "encryptionkey@0"
I20251209 08:59:24.807503 30917 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1765270668420570-20677-0/minicluster-data/ts-2/wal/instance:
uuid: "b9741ad1f88641bcb989f7ab70c4b6fa"
format_stamp: "Formatted at 2025-12-09 08:59:24 on dist-test-slave-8w6c"
server_key: "48d338b05c14e4b1e7e605fc67a9cad8"
server_key_iv: "ea052b53aea854254916a8e4a1425b8f"
server_key_version: "encryptionkey@0"
I20251209 08:59:24.811017 30917 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20251209 08:59:24.813328 30931 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251209 08:59:24.814368 30917 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20251209 08:59:24.814476 30917 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1765270668420570-20677-0/minicluster-data/ts-2/data,/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1765270668420570-20677-0/minicluster-data/ts-2/wal
uuid: "b9741ad1f88641bcb989f7ab70c4b6fa"
format_stamp: "Formatted at 2025-12-09 08:59:24 on dist-test-slave-8w6c"
server_key: "48d338b05c14e4b1e7e605fc67a9cad8"
server_key_iv: "ea052b53aea854254916a8e4a1425b8f"
server_key_version: "encryptionkey@0"
I20251209 08:59:24.814574 30917 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1765270668420570-20677-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1765270668420570-20677-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1765270668420570-20677-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251209 08:59:24.832365 30917 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251209 08:59:24.833035 30917 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251209 08:59:24.833209 30917 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251209 08:59:24.833842 30917 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251209 08:59:24.834805 30917 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251209 08:59:24.834857 30917 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251209 08:59:24.834919 30917 ts_tablet_manager.cc:616] Registered 0 tablets
I20251209 08:59:24.834960 30917 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251209 08:59:24.843816 30917 rpc_server.cc:307] RPC server started. Bound to: 127.20.49.67:35425
I20251209 08:59:24.843858 31044 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.49.67:35425 every 8 connection(s)
I20251209 08:59:24.844802 30917 server_base.cc:1179] Dumped server information to /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1765270668420570-20677-0/minicluster-data/ts-2/data/info.pb
I20251209 08:59:24.849100 20677 external_mini_cluster.cc:1428] Started /tmp/dist-test-task8XuDZV/build/debug/bin/kudu as pid 30917
I20251209 08:59:24.849243 20677 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1765270668420570-20677-0/minicluster-data/ts-2/wal/instance
I20251209 08:59:24.849606 20677 external_mini_cluster.cc:1468] Setting key 62f9129a763ece9bcdcc2fd64d83e0f2
I20251209 08:59:24.853468 30634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:59:24.846620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.49.67:34139 (local address 127.20.49.126:40409)
1209 08:59:24.846745 (+ 125us) server_negotiation.cc:207] Beginning negotiation
1209 08:59:24.846748 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1209 08:59:24.847656 (+ 908us) server_negotiation.cc:408] Connection header received
1209 08:59:24.848515 (+ 859us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1209 08:59:24.848519 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1209 08:59:24.848584 (+ 65us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1209 08:59:24.848671 (+ 87us) server_negotiation.cc:227] Negotiated authn=SASL
1209 08:59:24.849751 (+ 1080us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:59:24.850348 (+ 597us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1209 08:59:24.851080 (+ 732us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:59:24.851265 (+ 185us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:59:24.852945 (+ 1680us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1209 08:59:24.852961 (+ 16us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1209 08:59:24.852964 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1209 08:59:24.852976 (+ 12us) server_negotiation.cc:923] calling sasl_server_start()
1209 08:59:24.853001 (+ 25us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1209 08:59:24.853009 (+ 8us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1209 08:59:24.853072 (+ 63us) server_negotiation.cc:1036] Waiting for connection context
1209 08:59:24.853300 (+ 228us) server_negotiation.cc:300] Negotiation successful
1209 08:59:24.853354 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20251209 08:59:24.854271 31047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:59:24.846868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.49.126:40409 (local address 127.20.49.67:34139)
1209 08:59:24.847510 (+ 642us) negotiation.cc:107] Waiting for socket to connect
1209 08:59:24.847546 (+ 36us) client_negotiation.cc:174] Beginning negotiation
1209 08:59:24.848309 (+ 763us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1209 08:59:24.848855 (+ 546us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1209 08:59:24.848864 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
1209 08:59:24.848947 (+ 83us) client_negotiation.cc:189] Negotiated authn=SASL
1209 08:59:24.849581 (+ 634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:59:24.849594 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:59:24.850481 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1209 08:59:24.850484 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1209 08:59:24.850971 (+ 487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:59:24.850980 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:59:24.851202 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:59:24.852703 (+ 1501us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
1209 08:59:24.852705 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
1209 08:59:24.852748 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
1209 08:59:24.852817 (+ 69us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1209 08:59:24.853127 (+ 310us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1209 08:59:24.853136 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1209 08:59:24.853140 (+ 4us) client_negotiation.cc:769] Sending connection context
1209 08:59:24.853295 (+ 155us) client_negotiation.cc:240] Negotiation successful
1209 08:59:24.853505 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":103,"threads_started":1}
I20251209 08:59:24.855347 31045 heartbeater.cc:344] Connected to a master server at 127.20.49.126:40409
I20251209 08:59:24.855584 31045 heartbeater.cc:461] Registering TS with master...
I20251209 08:59:24.856196 31045 heartbeater.cc:507] Master 127.20.49.126:40409 requested a full tablet report, sending...
I20251209 08:59:24.857352 30595 ts_manager.cc:194] Registered new tserver with Master: b9741ad1f88641bcb989f7ab70c4b6fa (127.20.49.67:35425)
I20251209 08:59:24.858115 30595 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.20.49.67:34139
I20251209 08:59:24.863365 20677 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
W20251209 08:59:24.866015 30634 negotiation.cc:337] Failed RPC negotiation. Trace:
1209 08:59:24.865162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59982 (local address 127.20.49.126:40409)
1209 08:59:24.865297 (+ 135us) server_negotiation.cc:207] Beginning negotiation
1209 08:59:24.865301 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1209 08:59:24.865427 (+ 126us) server_negotiation.cc:408] Connection header received
1209 08:59:24.865525 (+ 98us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1209 08:59:24.865528 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1209 08:59:24.865570 (+ 42us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1209 08:59:24.865638 (+ 68us) server_negotiation.cc:227] Negotiated authn=SASL
1209 08:59:24.865868 (+ 230us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59982: BlockingRecv error: recv got EOF from 127.0.0.1:59982 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
I20251209 08:59:24.866153 20677 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task8XuDZV/build/debug/bin/kudu with pid 30655
I20251209 08:59:24.872500 20677 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task8XuDZV/build/debug/bin/kudu with pid 30786
I20251209 08:59:24.878616 20677 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task8XuDZV/build/debug/bin/kudu with pid 30917
I20251209 08:59:24.884487 20677 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task8XuDZV/build/debug/bin/kudu with pid 30564
2025-12-09T08:59:24Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (821 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2025-12-09T08:59:24Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-12-09T08:59:24Z Disabled control of system clock
I20251209 08:59:24.915040 20677 external_mini_cluster.cc:1366] Running /tmp/dist-test-task8XuDZV/build/debug/bin/kudu
/tmp/dist-test-task8XuDZV/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1765270668420570-20677-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1765270668420570-20677-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1765270668420570-20677-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1765270668420570-20677-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.49.126:42927
--webserver_interface=127.20.49.126
--webserver_port=0
--builtin_ntp_servers=127.20.49.84:43139
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.49.126:42927
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20251209 08:59:25.025707 31056 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251209 08:59:25.025982 31056 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251209 08:59:25.026034 31056 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251209 08:59:25.029253 31056 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251209 08:59:25.029314 31056 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251209 08:59:25.029332 31056 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251209 08:59:25.029351 31056 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251209 08:59:25.029369 31056 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251209 08:59:25.033366 31056 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.49.84:43139
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1765270668420570-20677-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1765270668420570-20677-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.49.126:42927
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.49.126:42927
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1765270668420570-20677-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.49.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31056
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1765270668420570-20677-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 468897755969d686fb08386f42f7835685e7953a
build type DEBUG
built by None at 09 Dec 2025 08:43:15 UTC on 24a791456cd2
build id 9426
I20251209 08:59:25.034446 31056 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251209 08:59:25.035527 31056 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251209 08:59:25.041292 31061 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251209 08:59:25.041340 31062 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251209 08:59:25.041510 31064 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251209 08:59:25.042128 31056 server_base.cc:1047] running on GCE node
I20251209 08:59:25.042627 31056 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251209 08:59:25.043597 31056 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251209 08:59:25.044759 31056 hybrid_clock.cc:648] HybridClock initialized: now 1765270765044750 us; error 30 us; skew 500 ppm
I20251209 08:59:25.046800 31056 webserver.cc:492] Webserver started at http://127.20.49.126:37417/ using document root <none> and password file <none>
I20251209 08:59:25.047395 31056 fs_manager.cc:362] Metadata directory not provided
I20251209 08:59:25.047458 31056 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251209 08:59:25.047649 31056 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251209 08:59:25.049360 31056 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1765270668420570-20677-0/minicluster-data/master-0/data/instance:
uuid: "dc977d66fadc4b74ac9e44248e2d633e"
format_stamp: "Formatted at 2025-12-09 08:59:25 on dist-test-slave-8w6c"
server_key: "53253612d2f2b9f2d877bf1371622fc3"
server_key_iv: "3dd89c3a9f3abf8c3ad776fd8e014dac"
server_key_version: "encryptionkey@0"
I20251209 08:59:25.049917 31056 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1765270668420570-20677-0/minicluster-data/master-0/wal/instance:
uuid: "dc977d66fadc4b74ac9e44248e2d633e"
format_stamp: "Formatted at 2025-12-09 08:59:25 on dist-test-slave-8w6c"
server_key: "53253612d2f2b9f2d877bf1371622fc3"
server_key_iv: "3dd89c3a9f3abf8c3ad776fd8e014dac"
server_key_version: "encryptionkey@0"
I20251209 08:59:25.053305 31056 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.001s sys 0.002s
I20251209 08:59:25.055676 31070 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251209 08:59:25.056644 31056 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20251209 08:59:25.056761 31056 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1765270668420570-20677-0/minicluster-data/master-0/data,/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1765270668420570-20677-0/minicluster-data/master-0/wal
uuid: "dc977d66fadc4b74ac9e44248e2d633e"
format_stamp: "Formatted at 2025-12-09 08:59:25 on dist-test-slave-8w6c"
server_key: "53253612d2f2b9f2d877bf1371622fc3"
server_key_iv: "3dd89c3a9f3abf8c3ad776fd8e014dac"
server_key_version: "encryptionkey@0"
I20251209 08:59:25.056859 31056 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1765270668420570-20677-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1765270668420570-20677-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1765270668420570-20677-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251209 08:59:25.062011 31056 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251209 08:59:25.062564 31056 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251209 08:59:25.062728 31056 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251209 08:59:25.069914 31056 rpc_server.cc:307] RPC server started. Bound to: 127.20.49.126:42927
I20251209 08:59:25.069936 31122 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.49.126:42927 every 8 connection(s)
I20251209 08:59:25.071033 31056 server_base.cc:1179] Dumped server information to /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1765270668420570-20677-0/minicluster-data/master-0/data/info.pb
I20251209 08:59:25.073944 31123 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251209 08:59:25.079682 31123 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dc977d66fadc4b74ac9e44248e2d633e: Bootstrap starting.
I20251209 08:59:25.080418 20677 external_mini_cluster.cc:1428] Started /tmp/dist-test-task8XuDZV/build/debug/bin/kudu as pid 31056
I20251209 08:59:25.080557 20677 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1765270668420570-20677-0/minicluster-data/master-0/wal/instance
I20251209 08:59:25.080780 20677 external_mini_cluster.cc:1468] Setting key 790f1c38f8d893d8f25d95395b4805e9
I20251209 08:59:25.082396 31123 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dc977d66fadc4b74ac9e44248e2d633e: Neither blocks nor log segments found. Creating new log.
I20251209 08:59:25.083128 31123 log.cc:826] T 00000000000000000000000000000000 P dc977d66fadc4b74ac9e44248e2d633e: Log is configured to *not* fsync() on all Append() calls
I20251209 08:59:25.085071 31123 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dc977d66fadc4b74ac9e44248e2d633e: No bootstrap required, opened a new log
I20251209 08:59:25.085661 31126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:59:25.081825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52254 (local address 127.20.49.126:42927)
1209 08:59:25.082473 (+ 648us) server_negotiation.cc:207] Beginning negotiation
1209 08:59:25.082482 (+ 9us) server_negotiation.cc:400] Waiting for connection header
1209 08:59:25.082524 (+ 42us) server_negotiation.cc:408] Connection header received
1209 08:59:25.083071 (+ 547us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1209 08:59:25.083089 (+ 18us) server_negotiation.cc:462] Received NEGOTIATE request from client
1209 08:59:25.083427 (+ 338us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1209 08:59:25.083742 (+ 315us) server_negotiation.cc:227] Negotiated authn=SASL
1209 08:59:25.084548 (+ 806us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1209 08:59:25.084554 (+ 6us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1209 08:59:25.084564 (+ 10us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1209 08:59:25.084574 (+ 10us) server_negotiation.cc:923] calling sasl_server_start()
1209 08:59:25.084617 (+ 43us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1209 08:59:25.084628 (+ 11us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1209 08:59:25.084737 (+ 109us) server_negotiation.cc:1036] Waiting for connection context
1209 08:59:25.084870 (+ 133us) server_negotiation.cc:300] Negotiation successful
1209 08:59:25.085090 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":299,"thread_start_us":145,"threads_started":1}
I20251209 08:59:25.088025 31123 raft_consensus.cc:359] T 00000000000000000000000000000000 P dc977d66fadc4b74ac9e44248e2d633e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc977d66fadc4b74ac9e44248e2d633e" member_type: VOTER last_known_addr { host: "127.20.49.126" port: 42927 } }
I20251209 08:59:25.088214 31123 raft_consensus.cc:385] T 00000000000000000000000000000000 P dc977d66fadc4b74ac9e44248e2d633e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251209 08:59:25.088281 31123 raft_consensus.cc:740] T 00000000000000000000000000000000 P dc977d66fadc4b74ac9e44248e2d633e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc977d66fadc4b74ac9e44248e2d633e, State: Initialized, Role: FOLLOWER
I20251209 08:59:25.088666 31123 consensus_queue.cc:260] T 00000000000000000000000000000000 P dc977d66fadc4b74ac9e44248e2d633e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc977d66fadc4b74ac9e44248e2d633e" member_type: VOTER last_known_addr { host: "127.20.49.126" port: 42927 } }
I20251209 08:59:25.088799 31123 raft_consensus.cc:399] T 00000000000000000000000000000000 P dc977d66fadc4b74ac9e44248e2d633e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251209 08:59:25.088862 31123 raft_consensus.cc:493] T 00000000000000000000000000000000 P dc977d66fadc4b74ac9e44248e2d633e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251209 08:59:25.088956 31123 raft_consensus.cc:3060] T 00000000000000000000000000000000 P dc977d66fadc4b74ac9e44248e2d633e [term 0 FOLLOWER]: Advancing to term 1
I20251209 08:59:25.089838 31123 raft_consensus.cc:515] T 00000000000000000000000000000000 P dc977d66fadc4b74ac9e44248e2d633e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc977d66fadc4b74ac9e44248e2d633e" member_type: VOTER last_known_addr { host: "127.20.49.126" port: 42927 } }
I20251209 08:59:25.090174 31123 leader_election.cc:304] T 00000000000000000000000000000000 P dc977d66fadc4b74ac9e44248e2d633e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dc977d66fadc4b74ac9e44248e2d633e; no voters:
I20251209 08:59:25.090443 31123 leader_election.cc:290] T 00000000000000000000000000000000 P dc977d66fadc4b74ac9e44248e2d633e [CANDIDATE]: Term 1 election: Requested vote from peers
I20251209 08:59:25.090561 31128 raft_consensus.cc:2804] T 00000000000000000000000000000000 P dc977d66fadc4b74ac9e44248e2d633e [term 1 FOLLOWER]: Leader election won for term 1
I20251209 08:59:25.090781 31128 raft_consensus.cc:697] T 00000000000000000000000000000000 P dc977d66fadc4b74ac9e44248e2d633e [term 1 LEADER]: Becoming Leader. State: Replica: dc977d66fadc4b74ac9e44248e2d633e, State: Running, Role: LEADER
I20251209 08:59:25.091105 31128 consensus_queue.cc:237] T 00000000000000000000000000000000 P dc977d66fadc4b74ac9e44248e2d633e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc977d66fadc4b74ac9e44248e2d633e" member_type: VOTER last_known_addr { host: "127.20.49.126" port: 42927 } }
I20251209 08:59:25.091502 31123 sys_catalog.cc:565] T 00000000000000000000000000000000 P dc977d66fadc4b74ac9e44248e2d633e [sys.catalog]: configured and running, proceeding with master startup.
I20251209 08:59:25.092525 31130 sys_catalog.cc:455] T 00000000000000000000000000000000 P dc977d66fadc4b74ac9e44248e2d633e [sys.catalog]: SysCatalogTable state changed. Reason: New leader dc977d66fadc4b74ac9e44248e2d633e. Latest consensus state: current_term: 1 leader_uuid: "dc977d66fadc4b74ac9e44248e2d633e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc977d66fadc4b74ac9e44248e2d633e" member_type: VOTER last_known_addr { host: "127.20.49.126" port: 42927 } } }
I20251209 08:59:25.092646 31130 sys_catalog.cc:458] T 00000000000000000000000000000000 P dc977d66fadc4b74ac9e44248e2d633e [sys.catalog]: This master's current role is: LEADER
I20251209 08:59:25.093102 31129 sys_catalog.cc:455] T 00000000000000000000000000000000 P dc977d66fadc4b74ac9e44248e2d633e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dc977d66fadc4b74ac9e44248e2d633e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc977d66fadc4b74ac9e44248e2d633e" member_type: VOTER last_known_addr { host: "127.20.49.126" port: 42927 } } }
I20251209 08:59:25.093225 31129 sys_catalog.cc:458] T 00000000000000000000000000000000 P dc977d66fadc4b74ac9e44248e2d633e [sys.catalog]: This master's current role is: LEADER
I20251209 08:59:25.093854 31137 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251209 08:59:25.096297 31137 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251209 08:59:25.100852 31137 catalog_manager.cc:1357] Generated new cluster ID: fb367c0b7f904d60b9667f59ffc26d71
I20251209 08:59:25.100919 31137 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251209 08:59:25.113293 31137 catalog_manager.cc:1380] Generated new certificate authority record
I20251209 08:59:25.113643 31137 catalog_manager.cc:1514] Loading token signing keys...
I20251209 08:59:25.127024 31137 catalog_manager.cc:6027] T 00000000000000000000000000000000 P dc977d66fadc4b74ac9e44248e2d633e: Generated new TSK 0
I20251209 08:59:25.127909 31137 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251209 08:59:25.137873 20677 external_mini_cluster.cc:1366] Running /tmp/dist-test-task8XuDZV/build/debug/bin/kudu
/tmp/dist-test-task8XuDZV/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1765270668420570-20677-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1765270668420570-20677-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1765270668420570-20677-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1765270668420570-20677-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.49.65:0
--local_ip_for_outbound_sockets=127.20.49.65
--webserver_interface=127.20.49.65
--webserver_port=0
--tserver_master_addrs=127.20.49.126:42927
--builtin_ntp_servers=127.20.49.84:43139
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20251209 08:59:25.247117 31147 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251209 08:59:25.247356 31147 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251209 08:59:25.247416 31147 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251209 08:59:25.250754 31147 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251209 08:59:25.250815 31147 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251209 08:59:25.250897 31147 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.49.65
I20251209 08:59:25.254786 31147 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.49.84:43139
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1765270668420570-20677-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1765270668420570-20677-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.49.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1765270668420570-20677-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.49.65
--webserver_port=0
--tserver_master_addrs=127.20.49.126:42927
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31147
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.49.65
--log_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1765270668420570-20677-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 468897755969d686fb08386f42f7835685e7953a
build type DEBUG
built by None at 09 Dec 2025 08:43:15 UTC on 24a791456cd2
build id 9426
I20251209 08:59:25.255860 31147 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251209 08:59:25.256969 31147 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251209 08:59:25.263403 31152 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251209 08:59:25.263530 31147 server_base.cc:1047] running on GCE node
W20251209 08:59:25.263392 31153 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251209 08:59:25.263388 31155 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251209 08:59:25.264031 31147 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251209 08:59:25.264546 31147 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251209 08:59:25.265730 31147 hybrid_clock.cc:648] HybridClock initialized: now 1765270765265711 us; error 31 us; skew 500 ppm
I20251209 08:59:25.267501 31147 webserver.cc:492] Webserver started at http://127.20.49.65:40353/ using document root <none> and password file <none>
I20251209 08:59:25.268051 31147 fs_manager.cc:362] Metadata directory not provided
I20251209 08:59:25.268105 31147 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251209 08:59:25.268285 31147 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251209 08:59:25.270005 31147 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1765270668420570-20677-0/minicluster-data/ts-0/data/instance:
uuid: "ef26a6d4dfab4d2694f4f78efd477d0c"
format_stamp: "Formatted at 2025-12-09 08:59:25 on dist-test-slave-8w6c"
server_key: "6bbab00d488607af297273b0fc29617b"
server_key_iv: "fd925da82356c10932bead44c64c8164"
server_key_version: "encryptionkey@0"
I20251209 08:59:25.270468 31147 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1765270668420570-20677-0/minicluster-data/ts-0/wal/instance:
uuid: "ef26a6d4dfab4d2694f4f78efd477d0c"
format_stamp: "Formatted at 2025-12-09 08:59:25 on dist-test-slave-8w6c"
server_key: "6bbab00d488607af297273b0fc29617b"
server_key_iv: "fd925da82356c10932bead44c64c8164"
server_key_version: "encryptionkey@0"
I20251209 08:59:25.273762 31147 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20251209 08:59:25.275995 31161 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251209 08:59:25.277182 31147 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20251209 08:59:25.277294 31147 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1765270668420570-20677-0/minicluster-data/ts-0/data,/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1765270668420570-20677-0/minicluster-data/ts-0/wal
uuid: "ef26a6d4dfab4d2694f4f78efd477d0c"
format_stamp: "Formatted at 2025-12-09 08:59:25 on dist-test-slave-8w6c"
server_key: "6bbab00d488607af297273b0fc29617b"
server_key_iv: "fd925da82356c10932bead44c64c8164"
server_key_version: "encryptionkey@0"
I20251209 08:59:25.277405 31147 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1765270668420570-20677-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1765270668420570-20677-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1765270668420570-20677-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251209 08:59:25.282200 31147 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251209 08:59:25.282758 31147 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251209 08:59:25.282917 31147 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251209 08:59:25.283442 31147 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251209 08:59:25.284368 31147 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251209 08:59:25.284415 31147 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251209 08:59:25.284524 31147 ts_tablet_manager.cc:616] Registered 0 tablets
I20251209 08:59:25.284590 31147 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251209 08:59:25.293526 31147 rpc_server.cc:307] RPC server started. Bound to: 127.20.49.65:46089
I20251209 08:59:25.293574 31274 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.49.65:46089 every 8 connection(s)
I20251209 08:59:25.294574 31147 server_base.cc:1179] Dumped server information to /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1765270668420570-20677-0/minicluster-data/ts-0/data/info.pb
I20251209 08:59:25.299832 31126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:59:25.296342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.49.65:57231 (local address 127.20.49.126:42927)
1209 08:59:25.296493 (+ 151us) server_negotiation.cc:207] Beginning negotiation
1209 08:59:25.296498 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1209 08:59:25.297414 (+ 916us) server_negotiation.cc:408] Connection header received
1209 08:59:25.298260 (+ 846us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1209 08:59:25.298263 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1209 08:59:25.298306 (+ 43us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1209 08:59:25.298396 (+ 90us) server_negotiation.cc:227] Negotiated authn=SASL
1209 08:59:25.299441 (+ 1045us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1209 08:59:25.299444 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1209 08:59:25.299447 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1209 08:59:25.299459 (+ 12us) server_negotiation.cc:923] calling sasl_server_start()
1209 08:59:25.299481 (+ 22us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1209 08:59:25.299489 (+ 8us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1209 08:59:25.299576 (+ 87us) server_negotiation.cc:1036] Waiting for connection context
1209 08:59:25.299685 (+ 109us) server_negotiation.cc:300] Negotiation successful
1209 08:59:25.299721 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20251209 08:59:25.300555 31277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:59:25.296639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.49.126:42927 (local address 127.20.49.65:57231)
1209 08:59:25.297282 (+ 643us) negotiation.cc:107] Waiting for socket to connect
1209 08:59:25.297316 (+ 34us) client_negotiation.cc:174] Beginning negotiation
1209 08:59:25.298048 (+ 732us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1209 08:59:25.298528 (+ 480us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1209 08:59:25.298536 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
1209 08:59:25.298616 (+ 80us) client_negotiation.cc:189] Negotiated authn=SASL
1209 08:59:25.299216 (+ 600us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
1209 08:59:25.299218 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
1209 08:59:25.299248 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
1209 08:59:25.299300 (+ 52us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1209 08:59:25.299565 (+ 265us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1209 08:59:25.299570 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1209 08:59:25.299574 (+ 4us) client_negotiation.cc:769] Sending connection context
1209 08:59:25.299730 (+ 156us) client_negotiation.cc:240] Negotiation successful
1209 08:59:25.299907 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":110,"threads_started":1}
I20251209 08:59:25.301642 31275 heartbeater.cc:344] Connected to a master server at 127.20.49.126:42927
I20251209 08:59:25.301901 31275 heartbeater.cc:461] Registering TS with master...
I20251209 08:59:25.302310 31275 heartbeater.cc:507] Master 127.20.49.126:42927 requested a full tablet report, sending...
I20251209 08:59:25.303695 31087 ts_manager.cc:194] Registered new tserver with Master: ef26a6d4dfab4d2694f4f78efd477d0c (127.20.49.65:46089)
I20251209 08:59:25.304287 20677 external_mini_cluster.cc:1428] Started /tmp/dist-test-task8XuDZV/build/debug/bin/kudu as pid 31147
I20251209 08:59:25.304404 20677 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1765270668420570-20677-0/minicluster-data/ts-0/wal/instance
I20251209 08:59:25.304670 20677 external_mini_cluster.cc:1468] Setting key 41909a2762ac2d850358599ad6034b51
I20251209 08:59:25.306486 20677 external_mini_cluster.cc:1366] Running /tmp/dist-test-task8XuDZV/build/debug/bin/kudu
/tmp/dist-test-task8XuDZV/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1765270668420570-20677-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1765270668420570-20677-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1765270668420570-20677-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1765270668420570-20677-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.49.66:0
--local_ip_for_outbound_sockets=127.20.49.66
--webserver_interface=127.20.49.66
--webserver_port=0
--tserver_master_addrs=127.20.49.126:42927
--builtin_ntp_servers=127.20.49.84:43139
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20251209 08:59:25.414495 31278 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251209 08:59:25.414729 31278 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251209 08:59:25.414783 31278 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251209 08:59:25.418064 31278 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251209 08:59:25.418126 31278 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251209 08:59:25.418236 31278 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.49.66
I20251209 08:59:25.422134 31278 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.49.84:43139
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1765270668420570-20677-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1765270668420570-20677-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.49.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1765270668420570-20677-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.49.66
--webserver_port=0
--tserver_master_addrs=127.20.49.126:42927
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31278
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.49.66
--log_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1765270668420570-20677-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 468897755969d686fb08386f42f7835685e7953a
build type DEBUG
built by None at 09 Dec 2025 08:43:15 UTC on 24a791456cd2
build id 9426
I20251209 08:59:25.423241 31278 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251209 08:59:25.424340 31278 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251209 08:59:25.430872 31286 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251209 08:59:25.430908 31283 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251209 08:59:25.430872 31284 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251209 08:59:25.431732 31278 server_base.cc:1047] running on GCE node
I20251209 08:59:25.432099 31278 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251209 08:59:25.432641 31278 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251209 08:59:25.433743 31278 hybrid_clock.cc:648] HybridClock initialized: now 1765270765433729 us; error 29 us; skew 500 ppm
I20251209 08:59:25.435539 31278 webserver.cc:492] Webserver started at http://127.20.49.66:46737/ using document root <none> and password file <none>
I20251209 08:59:25.436117 31278 fs_manager.cc:362] Metadata directory not provided
I20251209 08:59:25.436177 31278 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251209 08:59:25.436360 31278 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251209 08:59:25.438133 31278 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1765270668420570-20677-0/minicluster-data/ts-1/data/instance:
uuid: "abea6a2ef0d2413bbb03f34d9e87e9b2"
format_stamp: "Formatted at 2025-12-09 08:59:25 on dist-test-slave-8w6c"
server_key: "43501becf025e3675249c5423be7e9b2"
server_key_iv: "b472c0e9b84b61e0a03a280e7a3dbea8"
server_key_version: "encryptionkey@0"
I20251209 08:59:25.438601 31278 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1765270668420570-20677-0/minicluster-data/ts-1/wal/instance:
uuid: "abea6a2ef0d2413bbb03f34d9e87e9b2"
format_stamp: "Formatted at 2025-12-09 08:59:25 on dist-test-slave-8w6c"
server_key: "43501becf025e3675249c5423be7e9b2"
server_key_iv: "b472c0e9b84b61e0a03a280e7a3dbea8"
server_key_version: "encryptionkey@0"
I20251209 08:59:25.442055 31278 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20251209 08:59:25.444186 31292 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251209 08:59:25.445138 31278 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20251209 08:59:25.445233 31278 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1765270668420570-20677-0/minicluster-data/ts-1/data,/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1765270668420570-20677-0/minicluster-data/ts-1/wal
uuid: "abea6a2ef0d2413bbb03f34d9e87e9b2"
format_stamp: "Formatted at 2025-12-09 08:59:25 on dist-test-slave-8w6c"
server_key: "43501becf025e3675249c5423be7e9b2"
server_key_iv: "b472c0e9b84b61e0a03a280e7a3dbea8"
server_key_version: "encryptionkey@0"
I20251209 08:59:25.445327 31278 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1765270668420570-20677-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1765270668420570-20677-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1765270668420570-20677-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251209 08:59:25.450104 31278 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251209 08:59:25.450670 31278 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251209 08:59:25.450825 31278 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251209 08:59:25.451332 31278 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251209 08:59:25.452240 31278 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251209 08:59:25.452412 31278 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251209 08:59:25.452482 31278 ts_tablet_manager.cc:616] Registered 0 tablets
I20251209 08:59:25.452517 31278 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251209 08:59:25.461472 31278 rpc_server.cc:307] RPC server started. Bound to: 127.20.49.66:41147
I20251209 08:59:25.461570 31405 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.49.66:41147 every 8 connection(s)
I20251209 08:59:25.462667 31278 server_base.cc:1179] Dumped server information to /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1765270668420570-20677-0/minicluster-data/ts-1/data/info.pb
I20251209 08:59:25.467754 31126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:59:25.464406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.49.66:44405 (local address 127.20.49.126:42927)
1209 08:59:25.464543 (+ 137us) server_negotiation.cc:207] Beginning negotiation
1209 08:59:25.464546 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1209 08:59:25.465413 (+ 867us) server_negotiation.cc:408] Connection header received
1209 08:59:25.466239 (+ 826us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1209 08:59:25.466242 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1209 08:59:25.466286 (+ 44us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1209 08:59:25.466364 (+ 78us) server_negotiation.cc:227] Negotiated authn=SASL
1209 08:59:25.467368 (+ 1004us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1209 08:59:25.467371 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1209 08:59:25.467373 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1209 08:59:25.467386 (+ 13us) server_negotiation.cc:923] calling sasl_server_start()
1209 08:59:25.467407 (+ 21us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1209 08:59:25.467414 (+ 7us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1209 08:59:25.467498 (+ 84us) server_negotiation.cc:1036] Waiting for connection context
1209 08:59:25.467593 (+ 95us) server_negotiation.cc:300] Negotiation successful
1209 08:59:25.467627 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20251209 08:59:25.468477 31408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:59:25.464647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.49.126:42927 (local address 127.20.49.66:44405)
1209 08:59:25.465282 (+ 635us) negotiation.cc:107] Waiting for socket to connect
1209 08:59:25.465317 (+ 35us) client_negotiation.cc:174] Beginning negotiation
1209 08:59:25.466049 (+ 732us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1209 08:59:25.466512 (+ 463us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1209 08:59:25.466520 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
1209 08:59:25.466603 (+ 83us) client_negotiation.cc:189] Negotiated authn=SASL
1209 08:59:25.467144 (+ 541us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
1209 08:59:25.467146 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
1209 08:59:25.467177 (+ 31us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
1209 08:59:25.467234 (+ 57us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1209 08:59:25.467484 (+ 250us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1209 08:59:25.467489 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1209 08:59:25.467492 (+ 3us) client_negotiation.cc:769] Sending connection context
1209 08:59:25.467645 (+ 153us) client_negotiation.cc:240] Negotiation successful
1209 08:59:25.467846 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":106,"threads_started":1}
I20251209 08:59:25.469451 31406 heartbeater.cc:344] Connected to a master server at 127.20.49.126:42927
I20251209 08:59:25.469671 31406 heartbeater.cc:461] Registering TS with master...
I20251209 08:59:25.470093 31406 heartbeater.cc:507] Master 127.20.49.126:42927 requested a full tablet report, sending...
I20251209 08:59:25.471038 31087 ts_manager.cc:194] Registered new tserver with Master: abea6a2ef0d2413bbb03f34d9e87e9b2 (127.20.49.66:41147)
I20251209 08:59:25.472868 20677 external_mini_cluster.cc:1428] Started /tmp/dist-test-task8XuDZV/build/debug/bin/kudu as pid 31278
I20251209 08:59:25.472976 20677 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1765270668420570-20677-0/minicluster-data/ts-1/wal/instance
I20251209 08:59:25.473191 20677 external_mini_cluster.cc:1468] Setting key 697a31c6da0fc94d7863ef6811cdc398
I20251209 08:59:25.474988 20677 external_mini_cluster.cc:1366] Running /tmp/dist-test-task8XuDZV/build/debug/bin/kudu
/tmp/dist-test-task8XuDZV/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1765270668420570-20677-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1765270668420570-20677-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1765270668420570-20677-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1765270668420570-20677-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.49.67:0
--local_ip_for_outbound_sockets=127.20.49.67
--webserver_interface=127.20.49.67
--webserver_port=0
--tserver_master_addrs=127.20.49.126:42927
--builtin_ntp_servers=127.20.49.84:43139
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20251209 08:59:25.584723 31409 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251209 08:59:25.584982 31409 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251209 08:59:25.585073 31409 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251209 08:59:25.588407 31409 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251209 08:59:25.588469 31409 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251209 08:59:25.588595 31409 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.49.67
I20251209 08:59:25.592468 31409 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.49.84:43139
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1765270668420570-20677-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1765270668420570-20677-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.49.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1765270668420570-20677-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.49.67
--webserver_port=0
--tserver_master_addrs=127.20.49.126:42927
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31409
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.49.67
--log_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1765270668420570-20677-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 468897755969d686fb08386f42f7835685e7953a
build type DEBUG
built by None at 09 Dec 2025 08:43:15 UTC on 24a791456cd2
build id 9426
I20251209 08:59:25.593636 31409 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251209 08:59:25.594857 31409 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251209 08:59:25.601394 31417 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251209 08:59:25.601383 31414 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251209 08:59:25.601433 31415 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251209 08:59:25.601817 31409 server_base.cc:1047] running on GCE node
I20251209 08:59:25.602214 31409 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251209 08:59:25.602739 31409 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251209 08:59:25.603901 31409 hybrid_clock.cc:648] HybridClock initialized: now 1765270765603865 us; error 43 us; skew 500 ppm
I20251209 08:59:25.605635 31409 webserver.cc:492] Webserver started at http://127.20.49.67:44623/ using document root <none> and password file <none>
I20251209 08:59:25.606223 31409 fs_manager.cc:362] Metadata directory not provided
I20251209 08:59:25.606281 31409 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251209 08:59:25.606446 31409 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251209 08:59:25.608141 31409 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1765270668420570-20677-0/minicluster-data/ts-2/data/instance:
uuid: "6cef490238d74ff7974cf7802765e092"
format_stamp: "Formatted at 2025-12-09 08:59:25 on dist-test-slave-8w6c"
server_key: "e02a9939d0441827b13090bf29775db6"
server_key_iv: "c1c87bd453f76606cea873802fb0a7b0"
server_key_version: "encryptionkey@0"
I20251209 08:59:25.608573 31409 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1765270668420570-20677-0/minicluster-data/ts-2/wal/instance:
uuid: "6cef490238d74ff7974cf7802765e092"
format_stamp: "Formatted at 2025-12-09 08:59:25 on dist-test-slave-8w6c"
server_key: "e02a9939d0441827b13090bf29775db6"
server_key_iv: "c1c87bd453f76606cea873802fb0a7b0"
server_key_version: "encryptionkey@0"
I20251209 08:59:25.612172 31409 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20251209 08:59:25.614439 31423 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251209 08:59:25.615761 31409 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20251209 08:59:25.615895 31409 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1765270668420570-20677-0/minicluster-data/ts-2/data,/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1765270668420570-20677-0/minicluster-data/ts-2/wal
uuid: "6cef490238d74ff7974cf7802765e092"
format_stamp: "Formatted at 2025-12-09 08:59:25 on dist-test-slave-8w6c"
server_key: "e02a9939d0441827b13090bf29775db6"
server_key_iv: "c1c87bd453f76606cea873802fb0a7b0"
server_key_version: "encryptionkey@0"
I20251209 08:59:25.616019 31409 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1765270668420570-20677-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1765270668420570-20677-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1765270668420570-20677-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251209 08:59:25.623175 31409 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251209 08:59:25.623929 31409 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251209 08:59:25.624112 31409 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251209 08:59:25.624780 31409 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251209 08:59:25.625837 31409 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251209 08:59:25.625893 31409 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251209 08:59:25.625941 31409 ts_tablet_manager.cc:616] Registered 0 tablets
I20251209 08:59:25.625962 31409 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251209 08:59:25.634758 31409 rpc_server.cc:307] RPC server started. Bound to: 127.20.49.67:37467
I20251209 08:59:25.634783 31536 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.49.67:37467 every 8 connection(s)
I20251209 08:59:25.635710 31409 server_base.cc:1179] Dumped server information to /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1765270668420570-20677-0/minicluster-data/ts-2/data/info.pb
I20251209 08:59:25.641031 31126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:59:25.637450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.49.67:57195 (local address 127.20.49.126:42927)
1209 08:59:25.637591 (+ 141us) server_negotiation.cc:207] Beginning negotiation
1209 08:59:25.637595 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1209 08:59:25.638525 (+ 930us) server_negotiation.cc:408] Connection header received
1209 08:59:25.639334 (+ 809us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1209 08:59:25.639338 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1209 08:59:25.639385 (+ 47us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1209 08:59:25.639470 (+ 85us) server_negotiation.cc:227] Negotiated authn=SASL
1209 08:59:25.640591 (+ 1121us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1209 08:59:25.640594 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1209 08:59:25.640596 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1209 08:59:25.640607 (+ 11us) server_negotiation.cc:923] calling sasl_server_start()
1209 08:59:25.640626 (+ 19us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1209 08:59:25.640633 (+ 7us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1209 08:59:25.640717 (+ 84us) server_negotiation.cc:1036] Waiting for connection context
1209 08:59:25.640853 (+ 136us) server_negotiation.cc:300] Negotiation successful
1209 08:59:25.640897 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20251209 08:59:25.641222 20677 external_mini_cluster.cc:1428] Started /tmp/dist-test-task8XuDZV/build/debug/bin/kudu as pid 31409
I20251209 08:59:25.641348 20677 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1765270668420570-20677-0/minicluster-data/ts-2/wal/instance
I20251209 08:59:25.641592 20677 external_mini_cluster.cc:1468] Setting key ca00b313fa6e320d9b1aba95035d779c
I20251209 08:59:25.641875 31539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:59:25.637747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.49.126:42927 (local address 127.20.49.67:57195)
1209 08:59:25.638384 (+ 637us) negotiation.cc:107] Waiting for socket to connect
1209 08:59:25.638417 (+ 33us) client_negotiation.cc:174] Beginning negotiation
1209 08:59:25.639133 (+ 716us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1209 08:59:25.639640 (+ 507us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1209 08:59:25.639648 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
1209 08:59:25.639733 (+ 85us) client_negotiation.cc:189] Negotiated authn=SASL
1209 08:59:25.640371 (+ 638us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
1209 08:59:25.640373 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
1209 08:59:25.640403 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
1209 08:59:25.640454 (+ 51us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1209 08:59:25.640708 (+ 254us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1209 08:59:25.640715 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1209 08:59:25.640721 (+ 6us) client_negotiation.cc:769] Sending connection context
1209 08:59:25.640969 (+ 248us) client_negotiation.cc:240] Negotiation successful
1209 08:59:25.641188 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":105,"threads_started":1}
I20251209 08:59:25.642992 31537 heartbeater.cc:344] Connected to a master server at 127.20.49.126:42927
I20251209 08:59:25.643229 31537 heartbeater.cc:461] Registering TS with master...
I20251209 08:59:25.643639 31537 heartbeater.cc:507] Master 127.20.49.126:42927 requested a full tablet report, sending...
I20251209 08:59:25.644562 31087 ts_manager.cc:194] Registered new tserver with Master: 6cef490238d74ff7974cf7802765e092 (127.20.49.67:37467)
I20251209 08:59:25.655421 20677 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
W20251209 08:59:25.658063 31126 negotiation.cc:337] Failed RPC negotiation. Trace:
1209 08:59:25.657252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52262 (local address 127.20.49.126:42927)
1209 08:59:25.657401 (+ 149us) server_negotiation.cc:207] Beginning negotiation
1209 08:59:25.657405 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1209 08:59:25.657507 (+ 102us) server_negotiation.cc:408] Connection header received
1209 08:59:25.657614 (+ 107us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1209 08:59:25.657616 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1209 08:59:25.657649 (+ 33us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1209 08:59:25.657758 (+ 109us) server_negotiation.cc:227] Negotiated authn=SASL
1209 08:59:25.657939 (+ 181us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52262: BlockingRecv error: recv got EOF from 127.0.0.1:52262 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
I20251209 08:59:25.658335 20677 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task8XuDZV/build/debug/bin/kudu with pid 31147
I20251209 08:59:25.664557 20677 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task8XuDZV/build/debug/bin/kudu with pid 31278
I20251209 08:59:25.670120 20677 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task8XuDZV/build/debug/bin/kudu with pid 31409
I20251209 08:59:25.676167 20677 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task8XuDZV/build/debug/bin/kudu with pid 31056
2025-12-09T08:59:25Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (790 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765270668420570-20677-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Dec 09 08:59:25 dist-test-slave-8w6c krb5kdc[31547](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Dec 09 08:59:25 dist-test-slave-8w6c krb5kdc[31547](info): set up 2 sockets
Dec 09 08:59:25 dist-test-slave-8w6c krb5kdc[31547](info): commencing operation
krb5kdc: starting...
W20251209 08:59:27.674732 20677 mini_kdc.cc:121] Time spent starting KDC: real 1.977s user 0.005s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765270668420570-20677-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765270668420570-20677-0/krb5kdc/test-user.keytab.
Dec 09 08:59:27 dist-test-slave-8w6c krb5kdc[31547](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765270767, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-12-09T08:59:27Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-12-09T08:59:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.49.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.49.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.49.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765270668420570-20677-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.20.49.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765270668420570-20677-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.20.49.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.20.49.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.20.49.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765270668420570-20677-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.20.49.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765270668420570-20677-0/krb5kdc/kudu.keytab.
I20251209 08:59:27.821233 20677 external_mini_cluster.cc:1366] Running /tmp/dist-test-task8XuDZV/build/debug/bin/kudu
/tmp/dist-test-task8XuDZV/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765270668420570-20677-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765270668420570-20677-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765270668420570-20677-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765270668420570-20677-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.49.126:39485
--webserver_interface=127.20.49.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765270668420570-20677-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.49.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.20.49.84:38885
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.49.126:39485
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765270668420570-20677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765270668420570-20677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765270668420570-20677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251209 08:59:27.934897 31563 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251209 08:59:27.935190 31563 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251209 08:59:27.935289 31563 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251209 08:59:27.938699 31563 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251209 08:59:27.938778 31563 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251209 08:59:27.938817 31563 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251209 08:59:27.938851 31563 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251209 08:59:27.938892 31563 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251209 08:59:27.943473 31563 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.49.84:38885
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765270668420570-20677-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765270668420570-20677-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.49.126:39485
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765270668420570-20677-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.49.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.49.126:39485
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765270668420570-20677-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.49.126
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31563
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765270668420570-20677-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 468897755969d686fb08386f42f7835685e7953a
build type DEBUG
built by None at 09 Dec 2025 08:43:15 UTC on 24a791456cd2
build id 9426
I20251209 08:59:27.944630 31563 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251209 08:59:27.945516 31563 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251209 08:59:27.951198 31571 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251209 08:59:27.951187 31568 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251209 08:59:27.951185 31569 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251209 08:59:27.951421 31563 server_base.cc:1047] running on GCE node
I20251209 08:59:27.951920 31563 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251209 08:59:27.952845 31563 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251209 08:59:27.953992 31563 hybrid_clock.cc:648] HybridClock initialized: now 1765270767953972 us; error 39 us; skew 500 ppm
Dec 09 08:59:27 dist-test-slave-8w6c krb5kdc[31547](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765270767, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.49.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251209 08:59:27.956732 31563 init.cc:377] Logged in from keytab as kudu/127.20.49.126@KRBTEST.COM (short username kudu)
I20251209 08:59:27.957876 31563 webserver.cc:492] Webserver started at http://127.20.49.126:38487/ using document root <none> and password file <none>
I20251209 08:59:27.958443 31563 fs_manager.cc:362] Metadata directory not provided
I20251209 08:59:27.958503 31563 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251209 08:59:27.958689 31563 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251209 08:59:27.960440 31563 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765270668420570-20677-0/minicluster-data/master-0/data/instance:
uuid: "857fa13249e74b4ab1d1b80601ef3690"
format_stamp: "Formatted at 2025-12-09 08:59:27 on dist-test-slave-8w6c"
server_key: "f9150181745a8fd1a07f179bf7f8286d"
server_key_iv: "9dbd3a6486871c50f7ebaaa4cbe59766"
server_key_version: "encryptionkey@0"
I20251209 08:59:27.960945 31563 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765270668420570-20677-0/minicluster-data/master-0/wal/instance:
uuid: "857fa13249e74b4ab1d1b80601ef3690"
format_stamp: "Formatted at 2025-12-09 08:59:27 on dist-test-slave-8w6c"
server_key: "f9150181745a8fd1a07f179bf7f8286d"
server_key_iv: "9dbd3a6486871c50f7ebaaa4cbe59766"
server_key_version: "encryptionkey@0"
I20251209 08:59:27.964349 31563 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20251209 08:59:27.966517 31578 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251209 08:59:27.967528 31563 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20251209 08:59:27.967641 31563 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765270668420570-20677-0/minicluster-data/master-0/data,/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765270668420570-20677-0/minicluster-data/master-0/wal
uuid: "857fa13249e74b4ab1d1b80601ef3690"
format_stamp: "Formatted at 2025-12-09 08:59:27 on dist-test-slave-8w6c"
server_key: "f9150181745a8fd1a07f179bf7f8286d"
server_key_iv: "9dbd3a6486871c50f7ebaaa4cbe59766"
server_key_version: "encryptionkey@0"
I20251209 08:59:27.967733 31563 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765270668420570-20677-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765270668420570-20677-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765270668420570-20677-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251209 08:59:27.987972 31563 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251209 08:59:27.990803 31563 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251209 08:59:27.991007 31563 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251209 08:59:27.998540 31563 rpc_server.cc:307] RPC server started. Bound to: 127.20.49.126:39485
I20251209 08:59:27.998543 31630 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.49.126:39485 every 8 connection(s)
I20251209 08:59:27.999630 31563 server_base.cc:1179] Dumped server information to /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765270668420570-20677-0/minicluster-data/master-0/data/info.pb
I20251209 08:59:28.002441 31631 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251209 08:59:28.008088 20677 external_mini_cluster.cc:1428] Started /tmp/dist-test-task8XuDZV/build/debug/bin/kudu as pid 31563
I20251209 08:59:28.008101 31631 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 857fa13249e74b4ab1d1b80601ef3690: Bootstrap starting.
I20251209 08:59:28.008268 20677 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765270668420570-20677-0/minicluster-data/master-0/wal/instance
I20251209 08:59:28.008544 20677 external_mini_cluster.cc:1468] Setting key d33f2bab5e70a5fb8a553db1ddd20247
I20251209 08:59:28.010663 31631 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 857fa13249e74b4ab1d1b80601ef3690: Neither blocks nor log segments found. Creating new log.
I20251209 08:59:28.011413 31631 log.cc:826] T 00000000000000000000000000000000 P 857fa13249e74b4ab1d1b80601ef3690: Log is configured to *not* fsync() on all Append() calls
I20251209 08:59:28.013302 31631 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 857fa13249e74b4ab1d1b80601ef3690: No bootstrap required, opened a new log
Dec 09 08:59:28 dist-test-slave-8w6c krb5kdc[31547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765270767, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.49.126@KRBTEST.COM
I20251209 08:59:28.016628 31631 raft_consensus.cc:359] T 00000000000000000000000000000000 P 857fa13249e74b4ab1d1b80601ef3690 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "857fa13249e74b4ab1d1b80601ef3690" member_type: VOTER last_known_addr { host: "127.20.49.126" port: 39485 } }
I20251209 08:59:28.016829 31631 raft_consensus.cc:385] T 00000000000000000000000000000000 P 857fa13249e74b4ab1d1b80601ef3690 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251209 08:59:28.016896 31631 raft_consensus.cc:740] T 00000000000000000000000000000000 P 857fa13249e74b4ab1d1b80601ef3690 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 857fa13249e74b4ab1d1b80601ef3690, State: Initialized, Role: FOLLOWER
I20251209 08:59:28.017349 31631 consensus_queue.cc:260] T 00000000000000000000000000000000 P 857fa13249e74b4ab1d1b80601ef3690 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "857fa13249e74b4ab1d1b80601ef3690" member_type: VOTER last_known_addr { host: "127.20.49.126" port: 39485 } }
I20251209 08:59:28.017494 31631 raft_consensus.cc:399] T 00000000000000000000000000000000 P 857fa13249e74b4ab1d1b80601ef3690 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251209 08:59:28.017555 31631 raft_consensus.cc:493] T 00000000000000000000000000000000 P 857fa13249e74b4ab1d1b80601ef3690 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251209 08:59:28.017649 31631 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 857fa13249e74b4ab1d1b80601ef3690 [term 0 FOLLOWER]: Advancing to term 1
I20251209 08:59:28.018484 31631 raft_consensus.cc:515] T 00000000000000000000000000000000 P 857fa13249e74b4ab1d1b80601ef3690 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "857fa13249e74b4ab1d1b80601ef3690" member_type: VOTER last_known_addr { host: "127.20.49.126" port: 39485 } }
I20251209 08:59:28.018812 31631 leader_election.cc:304] T 00000000000000000000000000000000 P 857fa13249e74b4ab1d1b80601ef3690 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 857fa13249e74b4ab1d1b80601ef3690; no voters:
I20251209 08:59:28.019130 31631 leader_election.cc:290] T 00000000000000000000000000000000 P 857fa13249e74b4ab1d1b80601ef3690 [CANDIDATE]: Term 1 election: Requested vote from peers
I20251209 08:59:28.019533 31636 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 857fa13249e74b4ab1d1b80601ef3690 [term 1 FOLLOWER]: Leader election won for term 1
I20251209 08:59:28.019703 31636 raft_consensus.cc:697] T 00000000000000000000000000000000 P 857fa13249e74b4ab1d1b80601ef3690 [term 1 LEADER]: Becoming Leader. State: Replica: 857fa13249e74b4ab1d1b80601ef3690, State: Running, Role: LEADER
I20251209 08:59:28.019981 31636 consensus_queue.cc:237] T 00000000000000000000000000000000 P 857fa13249e74b4ab1d1b80601ef3690 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "857fa13249e74b4ab1d1b80601ef3690" member_type: VOTER last_known_addr { host: "127.20.49.126" port: 39485 } }
I20251209 08:59:28.020404 31631 sys_catalog.cc:565] T 00000000000000000000000000000000 P 857fa13249e74b4ab1d1b80601ef3690 [sys.catalog]: configured and running, proceeding with master startup.
I20251209 08:59:28.021493 31637 sys_catalog.cc:455] T 00000000000000000000000000000000 P 857fa13249e74b4ab1d1b80601ef3690 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "857fa13249e74b4ab1d1b80601ef3690" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "857fa13249e74b4ab1d1b80601ef3690" member_type: VOTER last_known_addr { host: "127.20.49.126" port: 39485 } } }
I20251209 08:59:28.021610 31637 sys_catalog.cc:458] T 00000000000000000000000000000000 P 857fa13249e74b4ab1d1b80601ef3690 [sys.catalog]: This master's current role is: LEADER
I20251209 08:59:28.022389 31634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:59:28.009802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55976 (local address 127.20.49.126:39485)
1209 08:59:28.010207 (+ 405us) server_negotiation.cc:207] Beginning negotiation
1209 08:59:28.010217 (+ 10us) server_negotiation.cc:400] Waiting for connection header
1209 08:59:28.010253 (+ 36us) server_negotiation.cc:408] Connection header received
1209 08:59:28.010825 (+ 572us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1209 08:59:28.010846 (+ 21us) server_negotiation.cc:462] Received NEGOTIATE request from client
1209 08:59:28.011144 (+ 298us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1209 08:59:28.011426 (+ 282us) server_negotiation.cc:227] Negotiated authn=SASL
1209 08:59:28.012391 (+ 965us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:59:28.013137 (+ 746us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1209 08:59:28.014017 (+ 880us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:59:28.014231 (+ 214us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:59:28.016763 (+ 2532us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1209 08:59:28.016782 (+ 19us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1209 08:59:28.016794 (+ 12us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1209 08:59:28.016829 (+ 35us) server_negotiation.cc:923] calling sasl_server_start()
1209 08:59:28.018872 (+ 2043us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1209 08:59:28.019370 (+ 498us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1209 08:59:28.019376 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1209 08:59:28.019382 (+ 6us) server_negotiation.cc:968] calling sasl_server_step()
1209 08:59:28.019489 (+ 107us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1209 08:59:28.019882 (+ 393us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1209 08:59:28.019884 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1209 08:59:28.019886 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1209 08:59:28.020364 (+ 478us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1209 08:59:28.020640 (+ 276us) server_negotiation.cc:1036] Waiting for connection context
1209 08:59:28.020873 (+ 233us) server_negotiation.cc:300] Negotiation successful
1209 08:59:28.021325 (+ 452us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":270,"thread_start_us":120,"threads_started":1}
I20251209 08:59:28.023663 31638 sys_catalog.cc:455] T 00000000000000000000000000000000 P 857fa13249e74b4ab1d1b80601ef3690 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 857fa13249e74b4ab1d1b80601ef3690. Latest consensus state: current_term: 1 leader_uuid: "857fa13249e74b4ab1d1b80601ef3690" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "857fa13249e74b4ab1d1b80601ef3690" member_type: VOTER last_known_addr { host: "127.20.49.126" port: 39485 } } }
I20251209 08:59:28.024087 31638 sys_catalog.cc:458] T 00000000000000000000000000000000 P 857fa13249e74b4ab1d1b80601ef3690 [sys.catalog]: This master's current role is: LEADER
W20251209 08:59:28.026127 31651 catalog_manager.cc:1568] T 00000000000000000000000000000000 P 857fa13249e74b4ab1d1b80601ef3690: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20251209 08:59:28.026206 31651 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20251209 08:59:28.026305 31645 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251209 08:59:28.027063 31645 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251209 08:59:28.032186 31645 catalog_manager.cc:1357] Generated new cluster ID: 817069bad57244bc8cbc1ca0cb34fc8f
I20251209 08:59:28.032253 31645 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251209 08:59:28.049978 31645 catalog_manager.cc:1380] Generated new certificate authority record
I20251209 08:59:28.050959 31645 catalog_manager.cc:1514] Loading token signing keys...
I20251209 08:59:28.058769 31645 catalog_manager.cc:6027] T 00000000000000000000000000000000 P 857fa13249e74b4ab1d1b80601ef3690: Generated new TSK 0
I20251209 08:59:28.059731 31645 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.20.49.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.49.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.49.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765270668420570-20677-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.20.49.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765270668420570-20677-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.20.49.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.20.49.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.20.49.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765270668420570-20677-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.20.49.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765270668420570-20677-0/krb5kdc/kudu.keytab.
I20251209 08:59:28.113232 20677 external_mini_cluster.cc:1366] Running /tmp/dist-test-task8XuDZV/build/debug/bin/kudu
/tmp/dist-test-task8XuDZV/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765270668420570-20677-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765270668420570-20677-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765270668420570-20677-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765270668420570-20677-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.49.65:0
--local_ip_for_outbound_sockets=127.20.49.65
--webserver_interface=127.20.49.65
--webserver_port=0
--tserver_master_addrs=127.20.49.126:39485
--keytab_file=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765270668420570-20677-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.49.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.20.49.84:38885
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765270668420570-20677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765270668420570-20677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765270668420570-20677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251209 08:59:28.224040 31659 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251209 08:59:28.224285 31659 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251209 08:59:28.224344 31659 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251209 08:59:28.227805 31659 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251209 08:59:28.227878 31659 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251209 08:59:28.227959 31659 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.49.65
I20251209 08:59:28.232534 31659 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.49.84:38885
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765270668420570-20677-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765270668420570-20677-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765270668420570-20677-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.49.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.49.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765270668420570-20677-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.49.65
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.20.49.126:39485
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31659
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.49.65
--log_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765270668420570-20677-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 468897755969d686fb08386f42f7835685e7953a
build type DEBUG
built by None at 09 Dec 2025 08:43:15 UTC on 24a791456cd2
build id 9426
I20251209 08:59:28.233651 31659 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251209 08:59:28.234560 31659 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251209 08:59:28.241328 31664 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251209 08:59:28.241289 31665 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251209 08:59:28.241421 31659 server_base.cc:1047] running on GCE node
W20251209 08:59:28.241561 31667 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251209 08:59:28.241966 31659 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251209 08:59:28.242522 31659 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251209 08:59:28.243708 31659 hybrid_clock.cc:648] HybridClock initialized: now 1765270768243682 us; error 43 us; skew 500 ppm
Dec 09 08:59:28 dist-test-slave-8w6c krb5kdc[31547](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765270768, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.49.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251209 08:59:28.246475 31659 init.cc:377] Logged in from keytab as kudu/127.20.49.65@KRBTEST.COM (short username kudu)
I20251209 08:59:28.247575 31659 webserver.cc:492] Webserver started at http://127.20.49.65:36947/ using document root <none> and password file <none>
I20251209 08:59:28.248126 31659 fs_manager.cc:362] Metadata directory not provided
I20251209 08:59:28.248184 31659 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251209 08:59:28.248347 31659 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251209 08:59:28.250142 31659 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765270668420570-20677-0/minicluster-data/ts-0/data/instance:
uuid: "94130c207124419ba606b0a370ab351a"
format_stamp: "Formatted at 2025-12-09 08:59:28 on dist-test-slave-8w6c"
server_key: "76795ecb040951b7f8d1a474675703e8"
server_key_iv: "314b387e4d79e7a09877de9ef995a3cf"
server_key_version: "encryptionkey@0"
I20251209 08:59:28.250595 31659 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765270668420570-20677-0/minicluster-data/ts-0/wal/instance:
uuid: "94130c207124419ba606b0a370ab351a"
format_stamp: "Formatted at 2025-12-09 08:59:28 on dist-test-slave-8w6c"
server_key: "76795ecb040951b7f8d1a474675703e8"
server_key_iv: "314b387e4d79e7a09877de9ef995a3cf"
server_key_version: "encryptionkey@0"
I20251209 08:59:28.254087 31659 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20251209 08:59:28.256332 31674 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251209 08:59:28.257364 31659 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20251209 08:59:28.257463 31659 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765270668420570-20677-0/minicluster-data/ts-0/data,/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765270668420570-20677-0/minicluster-data/ts-0/wal
uuid: "94130c207124419ba606b0a370ab351a"
format_stamp: "Formatted at 2025-12-09 08:59:28 on dist-test-slave-8w6c"
server_key: "76795ecb040951b7f8d1a474675703e8"
server_key_iv: "314b387e4d79e7a09877de9ef995a3cf"
server_key_version: "encryptionkey@0"
I20251209 08:59:28.257571 31659 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765270668420570-20677-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765270668420570-20677-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765270668420570-20677-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251209 08:59:28.274183 31659 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251209 08:59:28.280522 31659 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251209 08:59:28.280715 31659 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251209 08:59:28.281317 31659 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251209 08:59:28.282259 31659 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251209 08:59:28.282308 31659 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251209 08:59:28.282351 31659 ts_tablet_manager.cc:616] Registered 0 tablets
I20251209 08:59:28.282384 31659 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251209 08:59:28.291848 31659 rpc_server.cc:307] RPC server started. Bound to: 127.20.49.65:40727
I20251209 08:59:28.291877 31787 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.49.65:40727 every 8 connection(s)
I20251209 08:59:28.292811 31659 server_base.cc:1179] Dumped server information to /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765270668420570-20677-0/minicluster-data/ts-0/data/info.pb
I20251209 08:59:28.300072 20677 external_mini_cluster.cc:1428] Started /tmp/dist-test-task8XuDZV/build/debug/bin/kudu as pid 31659
I20251209 08:59:28.300230 20677 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765270668420570-20677-0/minicluster-data/ts-0/wal/instance
I20251209 08:59:28.300516 20677 external_mini_cluster.cc:1468] Setting key 5c5374e12e237b9dd2fb8e5e4d7d29c2
Dec 09 08:59:28 dist-test-slave-8w6c krb5kdc[31547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765270768, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.49.65@KRBTEST.COM for kudu/127.20.49.126@KRBTEST.COM
I20251209 08:59:28.305261 31634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:59:28.294571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.49.65:51933 (local address 127.20.49.126:39485)
1209 08:59:28.294745 (+ 174us) server_negotiation.cc:207] Beginning negotiation
1209 08:59:28.294750 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1209 08:59:28.295429 (+ 679us) server_negotiation.cc:408] Connection header received
1209 08:59:28.296278 (+ 849us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1209 08:59:28.296282 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1209 08:59:28.296337 (+ 55us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1209 08:59:28.296415 (+ 78us) server_negotiation.cc:227] Negotiated authn=SASL
1209 08:59:28.298152 (+ 1737us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:59:28.298669 (+ 517us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1209 08:59:28.299277 (+ 608us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:59:28.299411 (+ 134us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:59:28.301830 (+ 2419us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1209 08:59:28.301850 (+ 20us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1209 08:59:28.301852 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1209 08:59:28.301879 (+ 27us) server_negotiation.cc:923] calling sasl_server_start()
1209 08:59:28.303347 (+ 1468us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1209 08:59:28.303843 (+ 496us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1209 08:59:28.303846 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1209 08:59:28.303847 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1209 08:59:28.303892 (+ 45us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1209 08:59:28.304177 (+ 285us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1209 08:59:28.304180 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1209 08:59:28.304183 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
1209 08:59:28.304344 (+ 161us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1209 08:59:28.304436 (+ 92us) server_negotiation.cc:1036] Waiting for connection context
1209 08:59:28.304938 (+ 502us) server_negotiation.cc:300] Negotiation successful
1209 08:59:28.305094 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20251209 08:59:28.305994 31790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:59:28.294839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.49.126:39485 (local address 127.20.49.65:51933)
1209 08:59:28.295285 (+ 446us) negotiation.cc:107] Waiting for socket to connect
1209 08:59:28.295334 (+ 49us) client_negotiation.cc:174] Beginning negotiation
1209 08:59:28.296095 (+ 761us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1209 08:59:28.296556 (+ 461us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1209 08:59:28.296563 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
1209 08:59:28.296948 (+ 385us) client_negotiation.cc:189] Negotiated authn=SASL
1209 08:59:28.298010 (+ 1062us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:59:28.298026 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:59:28.298763 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1209 08:59:28.298766 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1209 08:59:28.299180 (+ 414us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:59:28.299188 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:59:28.299359 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:59:28.299961 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1209 08:59:28.300024 (+ 63us) client_negotiation.cc:656] calling sasl_client_start()
1209 08:59:28.301647 (+ 1623us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1209 08:59:28.303480 (+ 1833us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1209 08:59:28.303486 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1209 08:59:28.303499 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
1209 08:59:28.303743 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1209 08:59:28.303977 (+ 234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1209 08:59:28.303980 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1209 08:59:28.303982 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1209 08:59:28.304088 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1209 08:59:28.304447 (+ 359us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1209 08:59:28.304456 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1209 08:59:28.304697 (+ 241us) client_negotiation.cc:769] Sending connection context
1209 08:59:28.304895 (+ 198us) client_negotiation.cc:240] Negotiation successful
1209 08:59:28.305184 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":105,"threads_started":1}
I20251209 08:59:28.307214 31788 heartbeater.cc:344] Connected to a master server at 127.20.49.126:39485
I20251209 08:59:28.307518 31788 heartbeater.cc:461] Registering TS with master...
I20251209 08:59:28.308194 31788 heartbeater.cc:507] Master 127.20.49.126:39485 requested a full tablet report, sending...
I20251209 08:59:28.309990 31595 ts_manager.cc:194] Registered new tserver with Master: 94130c207124419ba606b0a370ab351a (127.20.49.65:40727)
WARNING: no policy specified for kudu/127.20.49.66@KRBTEST.COM; defaulting to no policy
I20251209 08:59:28.311303 31595 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.49.65@KRBTEST.COM'} at 127.20.49.65:51933
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.49.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.49.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765270668420570-20677-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.20.49.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765270668420570-20677-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.20.49.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.20.49.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.20.49.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765270668420570-20677-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.20.49.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765270668420570-20677-0/krb5kdc/kudu.keytab.
I20251209 08:59:28.352608 20677 external_mini_cluster.cc:1366] Running /tmp/dist-test-task8XuDZV/build/debug/bin/kudu
/tmp/dist-test-task8XuDZV/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765270668420570-20677-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765270668420570-20677-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765270668420570-20677-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765270668420570-20677-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.49.66:0
--local_ip_for_outbound_sockets=127.20.49.66
--webserver_interface=127.20.49.66
--webserver_port=0
--tserver_master_addrs=127.20.49.126:39485
--keytab_file=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765270668420570-20677-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.49.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.20.49.84:38885
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765270668420570-20677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765270668420570-20677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765270668420570-20677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251209 08:59:28.465052 31795 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251209 08:59:28.465291 31795 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251209 08:59:28.465351 31795 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251209 08:59:28.468827 31795 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251209 08:59:28.468891 31795 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251209 08:59:28.468981 31795 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.49.66
I20251209 08:59:28.473323 31795 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.49.84:38885
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765270668420570-20677-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765270668420570-20677-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765270668420570-20677-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.49.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.49.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765270668420570-20677-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.49.66
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.20.49.126:39485
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31795
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.49.66
--log_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765270668420570-20677-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 468897755969d686fb08386f42f7835685e7953a
build type DEBUG
built by None at 09 Dec 2025 08:43:15 UTC on 24a791456cd2
build id 9426
I20251209 08:59:28.474416 31795 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251209 08:59:28.475222 31795 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251209 08:59:28.481778 31800 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251209 08:59:28.481782 31803 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251209 08:59:28.481781 31801 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251209 08:59:28.482066 31795 server_base.cc:1047] running on GCE node
I20251209 08:59:28.482443 31795 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251209 08:59:28.483004 31795 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251209 08:59:28.484163 31795 hybrid_clock.cc:648] HybridClock initialized: now 1765270768484150 us; error 45 us; skew 500 ppm
Dec 09 08:59:28 dist-test-slave-8w6c krb5kdc[31547](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765270768, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.49.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251209 08:59:28.486984 31795 init.cc:377] Logged in from keytab as kudu/127.20.49.66@KRBTEST.COM (short username kudu)
I20251209 08:59:28.488094 31795 webserver.cc:492] Webserver started at http://127.20.49.66:33589/ using document root <none> and password file <none>
I20251209 08:59:28.488649 31795 fs_manager.cc:362] Metadata directory not provided
I20251209 08:59:28.488698 31795 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251209 08:59:28.488883 31795 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251209 08:59:28.490657 31795 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765270668420570-20677-0/minicluster-data/ts-1/data/instance:
uuid: "86220bf3e5b9406fa77c716ccad135e5"
format_stamp: "Formatted at 2025-12-09 08:59:28 on dist-test-slave-8w6c"
server_key: "f1423739407d1b78de5a3be1a3efbcf4"
server_key_iv: "8b68f9582a5c02798448597b513252fc"
server_key_version: "encryptionkey@0"
I20251209 08:59:28.491250 31795 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765270668420570-20677-0/minicluster-data/ts-1/wal/instance:
uuid: "86220bf3e5b9406fa77c716ccad135e5"
format_stamp: "Formatted at 2025-12-09 08:59:28 on dist-test-slave-8w6c"
server_key: "f1423739407d1b78de5a3be1a3efbcf4"
server_key_iv: "8b68f9582a5c02798448597b513252fc"
server_key_version: "encryptionkey@0"
I20251209 08:59:28.495298 31795 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20251209 08:59:28.497788 31810 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251209 08:59:28.498813 31795 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20251209 08:59:28.498942 31795 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765270668420570-20677-0/minicluster-data/ts-1/data,/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765270668420570-20677-0/minicluster-data/ts-1/wal
uuid: "86220bf3e5b9406fa77c716ccad135e5"
format_stamp: "Formatted at 2025-12-09 08:59:28 on dist-test-slave-8w6c"
server_key: "f1423739407d1b78de5a3be1a3efbcf4"
server_key_iv: "8b68f9582a5c02798448597b513252fc"
server_key_version: "encryptionkey@0"
I20251209 08:59:28.499042 31795 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765270668420570-20677-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765270668420570-20677-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765270668420570-20677-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251209 08:59:28.514794 31795 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251209 08:59:28.517611 31795 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251209 08:59:28.517828 31795 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251209 08:59:28.518393 31795 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251209 08:59:28.519268 31795 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251209 08:59:28.519320 31795 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251209 08:59:28.519380 31795 ts_tablet_manager.cc:616] Registered 0 tablets
I20251209 08:59:28.519409 31795 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251209 08:59:28.528715 31795 rpc_server.cc:307] RPC server started. Bound to: 127.20.49.66:40925
I20251209 08:59:28.528751 31923 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.49.66:40925 every 8 connection(s)
I20251209 08:59:28.529747 31795 server_base.cc:1179] Dumped server information to /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765270668420570-20677-0/minicluster-data/ts-1/data/info.pb
Dec 09 08:59:28 dist-test-slave-8w6c krb5kdc[31547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765270768, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.49.66@KRBTEST.COM for kudu/127.20.49.126@KRBTEST.COM
I20251209 08:59:28.539701 20677 external_mini_cluster.cc:1428] Started /tmp/dist-test-task8XuDZV/build/debug/bin/kudu as pid 31795
I20251209 08:59:28.539860 20677 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765270668420570-20677-0/minicluster-data/ts-1/wal/instance
I20251209 08:59:28.540181 20677 external_mini_cluster.cc:1468] Setting key db681d136a573152f47011cb89c596de
I20251209 08:59:28.542829 31634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:59:28.531619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.49.66:54101 (local address 127.20.49.126:39485)
1209 08:59:28.531762 (+ 143us) server_negotiation.cc:207] Beginning negotiation
1209 08:59:28.531766 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1209 08:59:28.532454 (+ 688us) server_negotiation.cc:408] Connection header received
1209 08:59:28.533306 (+ 852us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1209 08:59:28.533311 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1209 08:59:28.533376 (+ 65us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1209 08:59:28.533464 (+ 88us) server_negotiation.cc:227] Negotiated authn=SASL
1209 08:59:28.535211 (+ 1747us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:59:28.535713 (+ 502us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1209 08:59:28.536368 (+ 655us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:59:28.536555 (+ 187us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:59:28.538964 (+ 2409us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1209 08:59:28.538986 (+ 22us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1209 08:59:28.538988 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1209 08:59:28.539015 (+ 27us) server_negotiation.cc:923] calling sasl_server_start()
1209 08:59:28.540722 (+ 1707us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1209 08:59:28.541285 (+ 563us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1209 08:59:28.541288 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1209 08:59:28.541290 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1209 08:59:28.541338 (+ 48us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1209 08:59:28.541667 (+ 329us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1209 08:59:28.541671 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1209 08:59:28.541673 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1209 08:59:28.541859 (+ 186us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1209 08:59:28.541943 (+ 84us) server_negotiation.cc:1036] Waiting for connection context
1209 08:59:28.542598 (+ 655us) server_negotiation.cc:300] Negotiation successful
1209 08:59:28.542709 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20251209 08:59:28.543968 31926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:59:28.531868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.49.126:39485 (local address 127.20.49.66:54101)
1209 08:59:28.532314 (+ 446us) negotiation.cc:107] Waiting for socket to connect
1209 08:59:28.532351 (+ 37us) client_negotiation.cc:174] Beginning negotiation
1209 08:59:28.533100 (+ 749us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1209 08:59:28.533645 (+ 545us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1209 08:59:28.533653 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
1209 08:59:28.534088 (+ 435us) client_negotiation.cc:189] Negotiated authn=SASL
1209 08:59:28.535046 (+ 958us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:59:28.535060 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:59:28.535841 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1209 08:59:28.535844 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1209 08:59:28.536238 (+ 394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:59:28.536245 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:59:28.536485 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:59:28.537148 (+ 663us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1209 08:59:28.537173 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
1209 08:59:28.538790 (+ 1617us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1209 08:59:28.540842 (+ 2052us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1209 08:59:28.540847 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1209 08:59:28.540861 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
1209 08:59:28.541180 (+ 319us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1209 08:59:28.541430 (+ 250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1209 08:59:28.541433 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1209 08:59:28.541435 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1209 08:59:28.541579 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1209 08:59:28.541967 (+ 388us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1209 08:59:28.541976 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1209 08:59:28.542316 (+ 340us) client_negotiation.cc:769] Sending connection context
1209 08:59:28.542567 (+ 251us) client_negotiation.cc:240] Negotiation successful
1209 08:59:28.542883 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":105,"threads_started":1}
I20251209 08:59:28.545173 31924 heartbeater.cc:344] Connected to a master server at 127.20.49.126:39485
I20251209 08:59:28.545437 31924 heartbeater.cc:461] Registering TS with master...
I20251209 08:59:28.546036 31924 heartbeater.cc:507] Master 127.20.49.126:39485 requested a full tablet report, sending...
I20251209 08:59:28.547118 31595 ts_manager.cc:194] Registered new tserver with Master: 86220bf3e5b9406fa77c716ccad135e5 (127.20.49.66:40925)
I20251209 08:59:28.547672 31595 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.49.66@KRBTEST.COM'} at 127.20.49.66:54101
WARNING: no policy specified for kudu/127.20.49.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.49.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.49.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765270668420570-20677-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.20.49.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765270668420570-20677-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.20.49.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.20.49.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.20.49.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765270668420570-20677-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.20.49.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765270668420570-20677-0/krb5kdc/kudu.keytab.
I20251209 08:59:28.590864 20677 external_mini_cluster.cc:1366] Running /tmp/dist-test-task8XuDZV/build/debug/bin/kudu
/tmp/dist-test-task8XuDZV/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765270668420570-20677-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765270668420570-20677-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765270668420570-20677-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765270668420570-20677-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.49.67:0
--local_ip_for_outbound_sockets=127.20.49.67
--webserver_interface=127.20.49.67
--webserver_port=0
--tserver_master_addrs=127.20.49.126:39485
--keytab_file=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765270668420570-20677-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.49.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.20.49.84:38885
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765270668420570-20677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765270668420570-20677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765270668420570-20677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251209 08:59:28.699509 31931 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251209 08:59:28.699774 31931 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251209 08:59:28.699833 31931 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251209 08:59:28.703281 31931 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251209 08:59:28.703356 31931 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251209 08:59:28.703438 31931 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.49.67
I20251209 08:59:28.707758 31931 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.49.84:38885
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765270668420570-20677-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765270668420570-20677-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765270668420570-20677-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.49.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.49.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765270668420570-20677-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.49.67
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.20.49.126:39485
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31931
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.49.67
--log_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765270668420570-20677-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 468897755969d686fb08386f42f7835685e7953a
build type DEBUG
built by None at 09 Dec 2025 08:43:15 UTC on 24a791456cd2
build id 9426
I20251209 08:59:28.708879 31931 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251209 08:59:28.709779 31931 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251209 08:59:28.716322 31937 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251209 08:59:28.716413 31931 server_base.cc:1047] running on GCE node
W20251209 08:59:28.716328 31936 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251209 08:59:28.716317 31939 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251209 08:59:28.717042 31931 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251209 08:59:28.717671 31931 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251209 08:59:28.718900 31931 hybrid_clock.cc:648] HybridClock initialized: now 1765270768718878 us; error 36 us; skew 500 ppm
Dec 09 08:59:28 dist-test-slave-8w6c krb5kdc[31547](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765270768, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.49.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251209 08:59:28.721820 31931 init.cc:377] Logged in from keytab as kudu/127.20.49.67@KRBTEST.COM (short username kudu)
I20251209 08:59:28.723022 31931 webserver.cc:492] Webserver started at http://127.20.49.67:38427/ using document root <none> and password file <none>
I20251209 08:59:28.723582 31931 fs_manager.cc:362] Metadata directory not provided
I20251209 08:59:28.723629 31931 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251209 08:59:28.723790 31931 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251209 08:59:28.725526 31931 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765270668420570-20677-0/minicluster-data/ts-2/data/instance:
uuid: "c4e57711af184378903bab4a9e856c6f"
format_stamp: "Formatted at 2025-12-09 08:59:28 on dist-test-slave-8w6c"
server_key: "14173d9bc878b850dea9d50c40c35d97"
server_key_iv: "e52b42f63d91570a4acd996d1ae77f87"
server_key_version: "encryptionkey@0"
I20251209 08:59:28.726037 31931 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765270668420570-20677-0/minicluster-data/ts-2/wal/instance:
uuid: "c4e57711af184378903bab4a9e856c6f"
format_stamp: "Formatted at 2025-12-09 08:59:28 on dist-test-slave-8w6c"
server_key: "14173d9bc878b850dea9d50c40c35d97"
server_key_iv: "e52b42f63d91570a4acd996d1ae77f87"
server_key_version: "encryptionkey@0"
I20251209 08:59:28.729420 31931 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20251209 08:59:28.731725 31946 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251209 08:59:28.732721 31931 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20251209 08:59:28.732815 31931 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765270668420570-20677-0/minicluster-data/ts-2/data,/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765270668420570-20677-0/minicluster-data/ts-2/wal
uuid: "c4e57711af184378903bab4a9e856c6f"
format_stamp: "Formatted at 2025-12-09 08:59:28 on dist-test-slave-8w6c"
server_key: "14173d9bc878b850dea9d50c40c35d97"
server_key_iv: "e52b42f63d91570a4acd996d1ae77f87"
server_key_version: "encryptionkey@0"
I20251209 08:59:28.732894 31931 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765270668420570-20677-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765270668420570-20677-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765270668420570-20677-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251209 08:59:28.751747 31931 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251209 08:59:28.754805 31931 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251209 08:59:28.754969 31931 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251209 08:59:28.755489 31931 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251209 08:59:28.756379 31931 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251209 08:59:28.756428 31931 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251209 08:59:28.756469 31931 ts_tablet_manager.cc:616] Registered 0 tablets
I20251209 08:59:28.756491 31931 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251209 08:59:28.765933 31931 rpc_server.cc:307] RPC server started. Bound to: 127.20.49.67:44593
I20251209 08:59:28.765976 32059 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.49.67:44593 every 8 connection(s)
I20251209 08:59:28.766937 31931 server_base.cc:1179] Dumped server information to /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765270668420570-20677-0/minicluster-data/ts-2/data/info.pb
Dec 09 08:59:28 dist-test-slave-8w6c krb5kdc[31547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765270768, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.49.67@KRBTEST.COM for kudu/127.20.49.126@KRBTEST.COM
I20251209 08:59:28.777184 20677 external_mini_cluster.cc:1428] Started /tmp/dist-test-task8XuDZV/build/debug/bin/kudu as pid 31931
I20251209 08:59:28.777331 20677 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765270668420570-20677-0/minicluster-data/ts-2/wal/instance
I20251209 08:59:28.777588 20677 external_mini_cluster.cc:1468] Setting key 3e3d17b1e252927af483ff266ae977bd
I20251209 08:59:28.780032 31634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:59:28.768829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.49.67:44703 (local address 127.20.49.126:39485)
1209 08:59:28.769020 (+ 191us) server_negotiation.cc:207] Beginning negotiation
1209 08:59:28.769025 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1209 08:59:28.769743 (+ 718us) server_negotiation.cc:408] Connection header received
1209 08:59:28.770617 (+ 874us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1209 08:59:28.770621 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1209 08:59:28.770679 (+ 58us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1209 08:59:28.770762 (+ 83us) server_negotiation.cc:227] Negotiated authn=SASL
1209 08:59:28.772640 (+ 1878us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:59:28.773158 (+ 518us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1209 08:59:28.773784 (+ 626us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:59:28.773926 (+ 142us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:59:28.776357 (+ 2431us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1209 08:59:28.776378 (+ 21us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1209 08:59:28.776381 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1209 08:59:28.776410 (+ 29us) server_negotiation.cc:923] calling sasl_server_start()
1209 08:59:28.777916 (+ 1506us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1209 08:59:28.778452 (+ 536us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1209 08:59:28.778455 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1209 08:59:28.778456 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1209 08:59:28.778506 (+ 50us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1209 08:59:28.778837 (+ 331us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1209 08:59:28.778841 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1209 08:59:28.778843 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1209 08:59:28.779077 (+ 234us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1209 08:59:28.779160 (+ 83us) server_negotiation.cc:1036] Waiting for connection context
1209 08:59:28.779790 (+ 630us) server_negotiation.cc:300] Negotiation successful
1209 08:59:28.779911 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20251209 08:59:28.780737 32062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:59:28.769146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.49.126:39485 (local address 127.20.49.67:44703)
1209 08:59:28.769584 (+ 438us) negotiation.cc:107] Waiting for socket to connect
1209 08:59:28.769623 (+ 39us) client_negotiation.cc:174] Beginning negotiation
1209 08:59:28.770414 (+ 791us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1209 08:59:28.770919 (+ 505us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1209 08:59:28.770927 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
1209 08:59:28.771362 (+ 435us) client_negotiation.cc:189] Negotiated authn=SASL
1209 08:59:28.772478 (+ 1116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:59:28.772496 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:59:28.773277 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1209 08:59:28.773280 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1209 08:59:28.773662 (+ 382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:59:28.773668 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:59:28.773867 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:59:28.774520 (+ 653us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1209 08:59:28.774545 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
1209 08:59:28.776188 (+ 1643us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1209 08:59:28.778039 (+ 1851us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1209 08:59:28.778048 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1209 08:59:28.778064 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
1209 08:59:28.778332 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1209 08:59:28.778606 (+ 274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1209 08:59:28.778608 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1209 08:59:28.778610 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1209 08:59:28.778733 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1209 08:59:28.779187 (+ 454us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1209 08:59:28.779194 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1209 08:59:28.779501 (+ 307us) client_negotiation.cc:769] Sending connection context
1209 08:59:28.779733 (+ 232us) client_negotiation.cc:240] Negotiation successful
1209 08:59:28.780014 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":105,"threads_started":1}
I20251209 08:59:28.781764 32060 heartbeater.cc:344] Connected to a master server at 127.20.49.126:39485
I20251209 08:59:28.782054 32060 heartbeater.cc:461] Registering TS with master...
I20251209 08:59:28.782573 32060 heartbeater.cc:507] Master 127.20.49.126:39485 requested a full tablet report, sending...
I20251209 08:59:28.783656 31595 ts_manager.cc:194] Registered new tserver with Master: c4e57711af184378903bab4a9e856c6f (127.20.49.67:44593)
I20251209 08:59:28.784216 31595 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.49.67@KRBTEST.COM'} at 127.20.49.67:44703
I20251209 08:59:28.791589 20677 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20251209 08:59:28.799906 31634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:59:28.793410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58650 (local address 127.20.49.126:39485)
1209 08:59:28.793531 (+ 121us) server_negotiation.cc:207] Beginning negotiation
1209 08:59:28.793535 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1209 08:59:28.793723 (+ 188us) server_negotiation.cc:408] Connection header received
1209 08:59:28.793845 (+ 122us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1209 08:59:28.793848 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1209 08:59:28.793892 (+ 44us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1209 08:59:28.793966 (+ 74us) server_negotiation.cc:227] Negotiated authn=SASL
1209 08:59:28.794778 (+ 812us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:59:28.795270 (+ 492us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1209 08:59:28.795874 (+ 604us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:59:28.796009 (+ 135us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:59:28.797149 (+ 1140us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1209 08:59:28.797165 (+ 16us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1209 08:59:28.797167 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1209 08:59:28.797193 (+ 26us) server_negotiation.cc:923] calling sasl_server_start()
1209 08:59:28.798670 (+ 1477us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1209 08:59:28.799035 (+ 365us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1209 08:59:28.799038 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1209 08:59:28.799040 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1209 08:59:28.799077 (+ 37us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1209 08:59:28.799226 (+ 149us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1209 08:59:28.799228 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1209 08:59:28.799229 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1209 08:59:28.799409 (+ 180us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1209 08:59:28.799518 (+ 109us) server_negotiation.cc:1036] Waiting for connection context
1209 08:59:28.799663 (+ 145us) server_negotiation.cc:300] Negotiation successful
1209 08:59:28.799780 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20251209 08:59:28.802484 31595 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58650:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251209 08:59:28.804908 31595 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251209 08:59:28.818035 32071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:59:28.813473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.49.65:40727 (local address 127.0.0.1:38878)
1209 08:59:28.813845 (+ 372us) negotiation.cc:107] Waiting for socket to connect
1209 08:59:28.813889 (+ 44us) client_negotiation.cc:174] Beginning negotiation
1209 08:59:28.814034 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1209 08:59:28.814611 (+ 577us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1209 08:59:28.814616 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
1209 08:59:28.814635 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1209 08:59:28.814958 (+ 323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:59:28.814970 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:59:28.816515 (+ 1545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1209 08:59:28.816519 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1209 08:59:28.817448 (+ 929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:59:28.817462 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:59:28.817578 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:59:28.817642 (+ 64us) client_negotiation.cc:769] Sending connection context
1209 08:59:28.817762 (+ 120us) client_negotiation.cc:240] Negotiation successful
1209 08:59:28.817869 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":89,"threads_started":1}
I20251209 08:59:28.818235 32073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:59:28.814189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.49.66:40925 (local address 127.0.0.1:43852)
1209 08:59:28.814533 (+ 344us) negotiation.cc:107] Waiting for socket to connect
1209 08:59:28.814550 (+ 17us) client_negotiation.cc:174] Beginning negotiation
1209 08:59:28.814622 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1209 08:59:28.815572 (+ 950us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1209 08:59:28.815575 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1209 08:59:28.815595 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1209 08:59:28.815841 (+ 246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:59:28.815849 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:59:28.816955 (+ 1106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1209 08:59:28.816960 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1209 08:59:28.817796 (+ 836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:59:28.817806 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:59:28.817913 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:59:28.817931 (+ 18us) client_negotiation.cc:769] Sending connection context
1209 08:59:28.817979 (+ 48us) client_negotiation.cc:240] Negotiation successful
1209 08:59:28.818029 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":106,"threads_started":1}
I20251209 08:59:28.818650 32075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:59:28.814518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.49.67:44593 (local address 127.0.0.1:41998)
1209 08:59:28.814948 (+ 430us) negotiation.cc:107] Waiting for socket to connect
1209 08:59:28.814962 (+ 14us) client_negotiation.cc:174] Beginning negotiation
1209 08:59:28.815060 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1209 08:59:28.815790 (+ 730us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1209 08:59:28.815794 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
1209 08:59:28.815810 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1209 08:59:28.816031 (+ 221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:59:28.816037 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:59:28.817404 (+ 1367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1209 08:59:28.817408 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1209 08:59:28.818284 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:59:28.818293 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:59:28.818407 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:59:28.818431 (+ 24us) client_negotiation.cc:769] Sending connection context
1209 08:59:28.818477 (+ 46us) client_negotiation.cc:240] Negotiation successful
1209 08:59:28.818530 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":365,"thread_start_us":90,"threads_started":1}
I20251209 08:59:28.819432 32074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:59:28.814340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43852 (local address 127.20.49.66:40925)
1209 08:59:28.814762 (+ 422us) server_negotiation.cc:207] Beginning negotiation
1209 08:59:28.814768 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1209 08:59:28.814790 (+ 22us) server_negotiation.cc:408] Connection header received
1209 08:59:28.815293 (+ 503us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1209 08:59:28.815297 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1209 08:59:28.815424 (+ 127us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1209 08:59:28.815562 (+ 138us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1209 08:59:28.815961 (+ 399us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:59:28.816844 (+ 883us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1209 08:59:28.818488 (+ 1644us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:59:28.819118 (+ 630us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:59:28.819200 (+ 82us) server_negotiation.cc:1036] Waiting for connection context
1209 08:59:28.819254 (+ 54us) server_negotiation.cc:300] Negotiation successful
1209 08:59:28.819325 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":303,"thread_start_us":106,"threads_started":1}
I20251209 08:59:28.819442 32076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:59:28.814895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41998 (local address 127.20.49.67:44593)
1209 08:59:28.815382 (+ 487us) server_negotiation.cc:207] Beginning negotiation
1209 08:59:28.815389 (+ 7us) server_negotiation.cc:400] Waiting for connection header
1209 08:59:28.815410 (+ 21us) server_negotiation.cc:408] Connection header received
1209 08:59:28.815487 (+ 77us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1209 08:59:28.815492 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1209 08:59:28.815643 (+ 151us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1209 08:59:28.815779 (+ 136us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1209 08:59:28.816157 (+ 378us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:59:28.817153 (+ 996us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1209 08:59:28.818425 (+ 1272us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:59:28.819005 (+ 580us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:59:28.819127 (+ 122us) server_negotiation.cc:1036] Waiting for connection context
1209 08:59:28.819206 (+ 79us) server_negotiation.cc:300] Negotiation successful
1209 08:59:28.819303 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":384,"thread_start_us":108,"threads_started":1}
I20251209 08:59:28.819512 32072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:59:28.813647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38878 (local address 127.20.49.65:40727)
1209 08:59:28.813994 (+ 347us) server_negotiation.cc:207] Beginning negotiation
1209 08:59:28.814002 (+ 8us) server_negotiation.cc:400] Waiting for connection header
1209 08:59:28.814026 (+ 24us) server_negotiation.cc:408] Connection header received
1209 08:59:28.814219 (+ 193us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1209 08:59:28.814225 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
1209 08:59:28.814432 (+ 207us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1209 08:59:28.815115 (+ 683us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1209 08:59:28.815266 (+ 151us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:59:28.816396 (+ 1130us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1209 08:59:28.818495 (+ 2099us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:59:28.819141 (+ 646us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:59:28.819230 (+ 89us) server_negotiation.cc:1036] Waiting for connection context
1209 08:59:28.819286 (+ 56us) server_negotiation.cc:300] Negotiation successful
1209 08:59:28.819366 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":237,"thread_start_us":108,"threads_started":1}
I20251209 08:59:28.822422 31994 tablet_service.cc:1505] Processing CreateTablet for tablet c9e1dee3a83a4992b70370576ade8298 (DEFAULT_TABLE table=test-table [id=502f8706423b482192755c2bf69634e7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251209 08:59:28.822641 31722 tablet_service.cc:1505] Processing CreateTablet for tablet c9e1dee3a83a4992b70370576ade8298 (DEFAULT_TABLE table=test-table [id=502f8706423b482192755c2bf69634e7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251209 08:59:28.823504 31722 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c9e1dee3a83a4992b70370576ade8298. 1 dirs total, 0 dirs full, 0 dirs failed
I20251209 08:59:28.823505 31994 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c9e1dee3a83a4992b70370576ade8298. 1 dirs total, 0 dirs full, 0 dirs failed
I20251209 08:59:28.828016 31858 tablet_service.cc:1505] Processing CreateTablet for tablet c9e1dee3a83a4992b70370576ade8298 (DEFAULT_TABLE table=test-table [id=502f8706423b482192755c2bf69634e7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251209 08:59:28.829067 31858 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c9e1dee3a83a4992b70370576ade8298. 1 dirs total, 0 dirs full, 0 dirs failed
I20251209 08:59:28.830312 32077 tablet_bootstrap.cc:492] T c9e1dee3a83a4992b70370576ade8298 P 94130c207124419ba606b0a370ab351a: Bootstrap starting.
I20251209 08:59:28.832609 32077 tablet_bootstrap.cc:654] T c9e1dee3a83a4992b70370576ade8298 P 94130c207124419ba606b0a370ab351a: Neither blocks nor log segments found. Creating new log.
I20251209 08:59:28.832950 32078 tablet_bootstrap.cc:492] T c9e1dee3a83a4992b70370576ade8298 P c4e57711af184378903bab4a9e856c6f: Bootstrap starting.
I20251209 08:59:28.833473 32077 log.cc:826] T c9e1dee3a83a4992b70370576ade8298 P 94130c207124419ba606b0a370ab351a: Log is configured to *not* fsync() on all Append() calls
I20251209 08:59:28.834861 32080 tablet_bootstrap.cc:492] T c9e1dee3a83a4992b70370576ade8298 P 86220bf3e5b9406fa77c716ccad135e5: Bootstrap starting.
I20251209 08:59:28.835161 32078 tablet_bootstrap.cc:654] T c9e1dee3a83a4992b70370576ade8298 P c4e57711af184378903bab4a9e856c6f: Neither blocks nor log segments found. Creating new log.
I20251209 08:59:28.835621 32077 tablet_bootstrap.cc:492] T c9e1dee3a83a4992b70370576ade8298 P 94130c207124419ba606b0a370ab351a: No bootstrap required, opened a new log
I20251209 08:59:28.835812 32077 ts_tablet_manager.cc:1403] T c9e1dee3a83a4992b70370576ade8298 P 94130c207124419ba606b0a370ab351a: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20251209 08:59:28.835944 32078 log.cc:826] T c9e1dee3a83a4992b70370576ade8298 P c4e57711af184378903bab4a9e856c6f: Log is configured to *not* fsync() on all Append() calls
I20251209 08:59:28.837579 32078 tablet_bootstrap.cc:492] T c9e1dee3a83a4992b70370576ade8298 P c4e57711af184378903bab4a9e856c6f: No bootstrap required, opened a new log
I20251209 08:59:28.837611 32080 tablet_bootstrap.cc:654] T c9e1dee3a83a4992b70370576ade8298 P 86220bf3e5b9406fa77c716ccad135e5: Neither blocks nor log segments found. Creating new log.
I20251209 08:59:28.837788 32078 ts_tablet_manager.cc:1403] T c9e1dee3a83a4992b70370576ade8298 P c4e57711af184378903bab4a9e856c6f: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20251209 08:59:28.838445 32080 log.cc:826] T c9e1dee3a83a4992b70370576ade8298 P 86220bf3e5b9406fa77c716ccad135e5: Log is configured to *not* fsync() on all Append() calls
I20251209 08:59:28.839038 32077 raft_consensus.cc:359] T c9e1dee3a83a4992b70370576ade8298 P 94130c207124419ba606b0a370ab351a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94130c207124419ba606b0a370ab351a" member_type: VOTER last_known_addr { host: "127.20.49.65" port: 40727 } } peers { permanent_uuid: "86220bf3e5b9406fa77c716ccad135e5" member_type: VOTER last_known_addr { host: "127.20.49.66" port: 40925 } } peers { permanent_uuid: "c4e57711af184378903bab4a9e856c6f" member_type: VOTER last_known_addr { host: "127.20.49.67" port: 44593 } }
I20251209 08:59:28.839221 32077 raft_consensus.cc:385] T c9e1dee3a83a4992b70370576ade8298 P 94130c207124419ba606b0a370ab351a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251209 08:59:28.839288 32077 raft_consensus.cc:740] T c9e1dee3a83a4992b70370576ade8298 P 94130c207124419ba606b0a370ab351a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94130c207124419ba606b0a370ab351a, State: Initialized, Role: FOLLOWER
I20251209 08:59:28.839740 32077 consensus_queue.cc:260] T c9e1dee3a83a4992b70370576ade8298 P 94130c207124419ba606b0a370ab351a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94130c207124419ba606b0a370ab351a" member_type: VOTER last_known_addr { host: "127.20.49.65" port: 40727 } } peers { permanent_uuid: "86220bf3e5b9406fa77c716ccad135e5" member_type: VOTER last_known_addr { host: "127.20.49.66" port: 40925 } } peers { permanent_uuid: "c4e57711af184378903bab4a9e856c6f" member_type: VOTER last_known_addr { host: "127.20.49.67" port: 44593 } }
I20251209 08:59:28.840471 31788 heartbeater.cc:499] Master 127.20.49.126:39485 was elected leader, sending a full tablet report...
I20251209 08:59:28.840631 32077 ts_tablet_manager.cc:1434] T c9e1dee3a83a4992b70370576ade8298 P 94130c207124419ba606b0a370ab351a: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20251209 08:59:28.841106 32078 raft_consensus.cc:359] T c9e1dee3a83a4992b70370576ade8298 P c4e57711af184378903bab4a9e856c6f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94130c207124419ba606b0a370ab351a" member_type: VOTER last_known_addr { host: "127.20.49.65" port: 40727 } } peers { permanent_uuid: "86220bf3e5b9406fa77c716ccad135e5" member_type: VOTER last_known_addr { host: "127.20.49.66" port: 40925 } } peers { permanent_uuid: "c4e57711af184378903bab4a9e856c6f" member_type: VOTER last_known_addr { host: "127.20.49.67" port: 44593 } }
I20251209 08:59:28.841274 32078 raft_consensus.cc:385] T c9e1dee3a83a4992b70370576ade8298 P c4e57711af184378903bab4a9e856c6f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251209 08:59:28.841351 32078 raft_consensus.cc:740] T c9e1dee3a83a4992b70370576ade8298 P c4e57711af184378903bab4a9e856c6f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4e57711af184378903bab4a9e856c6f, State: Initialized, Role: FOLLOWER
I20251209 08:59:28.841367 32080 tablet_bootstrap.cc:492] T c9e1dee3a83a4992b70370576ade8298 P 86220bf3e5b9406fa77c716ccad135e5: No bootstrap required, opened a new log
I20251209 08:59:28.841550 32080 ts_tablet_manager.cc:1403] T c9e1dee3a83a4992b70370576ade8298 P 86220bf3e5b9406fa77c716ccad135e5: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.004s
I20251209 08:59:28.841850 32078 consensus_queue.cc:260] T c9e1dee3a83a4992b70370576ade8298 P c4e57711af184378903bab4a9e856c6f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94130c207124419ba606b0a370ab351a" member_type: VOTER last_known_addr { host: "127.20.49.65" port: 40727 } } peers { permanent_uuid: "86220bf3e5b9406fa77c716ccad135e5" member_type: VOTER last_known_addr { host: "127.20.49.66" port: 40925 } } peers { permanent_uuid: "c4e57711af184378903bab4a9e856c6f" member_type: VOTER last_known_addr { host: "127.20.49.67" port: 44593 } }
I20251209 08:59:28.842725 32078 ts_tablet_manager.cc:1434] T c9e1dee3a83a4992b70370576ade8298 P c4e57711af184378903bab4a9e856c6f: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20251209 08:59:28.843076 32060 heartbeater.cc:499] Master 127.20.49.126:39485 was elected leader, sending a full tablet report...
I20251209 08:59:28.845259 32080 raft_consensus.cc:359] T c9e1dee3a83a4992b70370576ade8298 P 86220bf3e5b9406fa77c716ccad135e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94130c207124419ba606b0a370ab351a" member_type: VOTER last_known_addr { host: "127.20.49.65" port: 40727 } } peers { permanent_uuid: "86220bf3e5b9406fa77c716ccad135e5" member_type: VOTER last_known_addr { host: "127.20.49.66" port: 40925 } } peers { permanent_uuid: "c4e57711af184378903bab4a9e856c6f" member_type: VOTER last_known_addr { host: "127.20.49.67" port: 44593 } }
I20251209 08:59:28.845441 32080 raft_consensus.cc:385] T c9e1dee3a83a4992b70370576ade8298 P 86220bf3e5b9406fa77c716ccad135e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251209 08:59:28.845504 32080 raft_consensus.cc:740] T c9e1dee3a83a4992b70370576ade8298 P 86220bf3e5b9406fa77c716ccad135e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86220bf3e5b9406fa77c716ccad135e5, State: Initialized, Role: FOLLOWER
I20251209 08:59:28.845937 32080 consensus_queue.cc:260] T c9e1dee3a83a4992b70370576ade8298 P 86220bf3e5b9406fa77c716ccad135e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94130c207124419ba606b0a370ab351a" member_type: VOTER last_known_addr { host: "127.20.49.65" port: 40727 } } peers { permanent_uuid: "86220bf3e5b9406fa77c716ccad135e5" member_type: VOTER last_known_addr { host: "127.20.49.66" port: 40925 } } peers { permanent_uuid: "c4e57711af184378903bab4a9e856c6f" member_type: VOTER last_known_addr { host: "127.20.49.67" port: 44593 } }
I20251209 08:59:28.846558 31924 heartbeater.cc:499] Master 127.20.49.126:39485 was elected leader, sending a full tablet report...
I20251209 08:59:28.846756 32080 ts_tablet_manager.cc:1434] T c9e1dee3a83a4992b70370576ade8298 P 86220bf3e5b9406fa77c716ccad135e5: Time spent starting tablet: real 0.005s user 0.003s sys 0.002s
I20251209 08:59:28.931195 32083 raft_consensus.cc:493] T c9e1dee3a83a4992b70370576ade8298 P 94130c207124419ba606b0a370ab351a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251209 08:59:28.931399 32083 raft_consensus.cc:515] T c9e1dee3a83a4992b70370576ade8298 P 94130c207124419ba606b0a370ab351a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94130c207124419ba606b0a370ab351a" member_type: VOTER last_known_addr { host: "127.20.49.65" port: 40727 } } peers { permanent_uuid: "86220bf3e5b9406fa77c716ccad135e5" member_type: VOTER last_known_addr { host: "127.20.49.66" port: 40925 } } peers { permanent_uuid: "c4e57711af184378903bab4a9e856c6f" member_type: VOTER last_known_addr { host: "127.20.49.67" port: 44593 } }
I20251209 08:59:28.932554 32083 leader_election.cc:290] T c9e1dee3a83a4992b70370576ade8298 P 94130c207124419ba606b0a370ab351a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 86220bf3e5b9406fa77c716ccad135e5 (127.20.49.66:40925), c4e57711af184378903bab4a9e856c6f (127.20.49.67:44593)
I20251209 08:59:28.937042 32086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:59:28.932862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.49.66:40925 (local address 127.20.49.65:35981)
1209 08:59:28.933216 (+ 354us) negotiation.cc:107] Waiting for socket to connect
1209 08:59:28.933233 (+ 17us) client_negotiation.cc:174] Beginning negotiation
1209 08:59:28.933352 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1209 08:59:28.933809 (+ 457us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1209 08:59:28.933811 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1209 08:59:28.933828 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1209 08:59:28.934498 (+ 670us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:59:28.934505 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:59:28.935575 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1209 08:59:28.935580 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1209 08:59:28.936575 (+ 995us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:59:28.936585 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:59:28.936713 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:59:28.936736 (+ 23us) client_negotiation.cc:769] Sending connection context
1209 08:59:28.936792 (+ 56us) client_negotiation.cc:240] Negotiation successful
1209 08:59:28.936855 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":170,"threads_started":1}
I20251209 08:59:28.937042 32087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:59:28.933346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.49.67:44593 (local address 127.20.49.65:33827)
1209 08:59:28.933675 (+ 329us) negotiation.cc:107] Waiting for socket to connect
1209 08:59:28.933711 (+ 36us) client_negotiation.cc:174] Beginning negotiation
1209 08:59:28.933802 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1209 08:59:28.934084 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1209 08:59:28.934087 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1209 08:59:28.934107 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1209 08:59:28.934479 (+ 372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:59:28.934488 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:59:28.935575 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1209 08:59:28.935580 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1209 08:59:28.936575 (+ 995us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 08:59:28.936585 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 08:59:28.936712 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:59:28.936736 (+ 24us) client_negotiation.cc:769] Sending connection context
1209 08:59:28.936792 (+ 56us) client_negotiation.cc:240] Negotiation successful
1209 08:59:28.936855 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":95,"threads_started":1}
I20251209 08:59:28.937511 32074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:59:28.932955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.49.65:35981 (local address 127.20.49.66:40925)
1209 08:59:28.933096 (+ 141us) server_negotiation.cc:207] Beginning negotiation
1209 08:59:28.933100 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1209 08:59:28.933323 (+ 223us) server_negotiation.cc:408] Connection header received
1209 08:59:28.933507 (+ 184us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1209 08:59:28.933511 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1209 08:59:28.933574 (+ 63us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1209 08:59:28.933633 (+ 59us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1209 08:59:28.934618 (+ 985us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:59:28.935423 (+ 805us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1209 08:59:28.936716 (+ 1293us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:59:28.937228 (+ 512us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:59:28.937291 (+ 63us) server_negotiation.cc:1036] Waiting for connection context
1209 08:59:28.937334 (+ 43us) server_negotiation.cc:300] Negotiation successful
1209 08:59:28.937397 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20251209 08:59:28.937511 32076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:59:28.933546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.49.65:33827 (local address 127.20.49.67:44593)
1209 08:59:28.933720 (+ 174us) server_negotiation.cc:207] Beginning negotiation
1209 08:59:28.933724 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1209 08:59:28.933761 (+ 37us) server_negotiation.cc:408] Connection header received
1209 08:59:28.933903 (+ 142us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1209 08:59:28.933907 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1209 08:59:28.933958 (+ 51us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1209 08:59:28.934052 (+ 94us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1209 08:59:28.934618 (+ 566us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:59:28.935432 (+ 814us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1209 08:59:28.936768 (+ 1336us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:59:28.937228 (+ 460us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:59:28.937269 (+ 41us) server_negotiation.cc:1036] Waiting for connection context
1209 08:59:28.937334 (+ 65us) server_negotiation.cc:300] Negotiation successful
1209 08:59:28.937397 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20251209 08:59:28.938158 31878 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c9e1dee3a83a4992b70370576ade8298" candidate_uuid: "94130c207124419ba606b0a370ab351a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "86220bf3e5b9406fa77c716ccad135e5" is_pre_election: true
I20251209 08:59:28.938162 32013 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c9e1dee3a83a4992b70370576ade8298" candidate_uuid: "94130c207124419ba606b0a370ab351a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4e57711af184378903bab4a9e856c6f" is_pre_election: true
I20251209 08:59:28.938418 32013 raft_consensus.cc:2468] T c9e1dee3a83a4992b70370576ade8298 P c4e57711af184378903bab4a9e856c6f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 94130c207124419ba606b0a370ab351a in term 0.
I20251209 08:59:28.938417 31878 raft_consensus.cc:2468] T c9e1dee3a83a4992b70370576ade8298 P 86220bf3e5b9406fa77c716ccad135e5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 94130c207124419ba606b0a370ab351a in term 0.
I20251209 08:59:28.938889 31675 leader_election.cc:304] T c9e1dee3a83a4992b70370576ade8298 P 94130c207124419ba606b0a370ab351a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 86220bf3e5b9406fa77c716ccad135e5, 94130c207124419ba606b0a370ab351a; no voters:
I20251209 08:59:28.939174 32083 raft_consensus.cc:2804] T c9e1dee3a83a4992b70370576ade8298 P 94130c207124419ba606b0a370ab351a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251209 08:59:28.939247 32083 raft_consensus.cc:493] T c9e1dee3a83a4992b70370576ade8298 P 94130c207124419ba606b0a370ab351a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251209 08:59:28.939278 32083 raft_consensus.cc:3060] T c9e1dee3a83a4992b70370576ade8298 P 94130c207124419ba606b0a370ab351a [term 0 FOLLOWER]: Advancing to term 1
I20251209 08:59:28.940265 32083 raft_consensus.cc:515] T c9e1dee3a83a4992b70370576ade8298 P 94130c207124419ba606b0a370ab351a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94130c207124419ba606b0a370ab351a" member_type: VOTER last_known_addr { host: "127.20.49.65" port: 40727 } } peers { permanent_uuid: "86220bf3e5b9406fa77c716ccad135e5" member_type: VOTER last_known_addr { host: "127.20.49.66" port: 40925 } } peers { permanent_uuid: "c4e57711af184378903bab4a9e856c6f" member_type: VOTER last_known_addr { host: "127.20.49.67" port: 44593 } }
I20251209 08:59:28.940629 32083 leader_election.cc:290] T c9e1dee3a83a4992b70370576ade8298 P 94130c207124419ba606b0a370ab351a [CANDIDATE]: Term 1 election: Requested vote from peers 86220bf3e5b9406fa77c716ccad135e5 (127.20.49.66:40925), c4e57711af184378903bab4a9e856c6f (127.20.49.67:44593)
I20251209 08:59:28.941023 31878 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c9e1dee3a83a4992b70370576ade8298" candidate_uuid: "94130c207124419ba606b0a370ab351a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "86220bf3e5b9406fa77c716ccad135e5"
I20251209 08:59:28.941033 32013 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c9e1dee3a83a4992b70370576ade8298" candidate_uuid: "94130c207124419ba606b0a370ab351a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4e57711af184378903bab4a9e856c6f"
I20251209 08:59:28.941154 32013 raft_consensus.cc:3060] T c9e1dee3a83a4992b70370576ade8298 P c4e57711af184378903bab4a9e856c6f [term 0 FOLLOWER]: Advancing to term 1
I20251209 08:59:28.941154 31878 raft_consensus.cc:3060] T c9e1dee3a83a4992b70370576ade8298 P 86220bf3e5b9406fa77c716ccad135e5 [term 0 FOLLOWER]: Advancing to term 1
I20251209 08:59:28.942444 31878 raft_consensus.cc:2468] T c9e1dee3a83a4992b70370576ade8298 P 86220bf3e5b9406fa77c716ccad135e5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94130c207124419ba606b0a370ab351a in term 1.
I20251209 08:59:28.942444 32013 raft_consensus.cc:2468] T c9e1dee3a83a4992b70370576ade8298 P c4e57711af184378903bab4a9e856c6f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94130c207124419ba606b0a370ab351a in term 1.
I20251209 08:59:28.942778 31675 leader_election.cc:304] T c9e1dee3a83a4992b70370576ade8298 P 94130c207124419ba606b0a370ab351a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 94130c207124419ba606b0a370ab351a, c4e57711af184378903bab4a9e856c6f; no voters:
I20251209 08:59:28.942940 32083 raft_consensus.cc:2804] T c9e1dee3a83a4992b70370576ade8298 P 94130c207124419ba606b0a370ab351a [term 1 FOLLOWER]: Leader election won for term 1
I20251209 08:59:28.943166 32083 raft_consensus.cc:697] T c9e1dee3a83a4992b70370576ade8298 P 94130c207124419ba606b0a370ab351a [term 1 LEADER]: Becoming Leader. State: Replica: 94130c207124419ba606b0a370ab351a, State: Running, Role: LEADER
I20251209 08:59:28.943490 32083 consensus_queue.cc:237] T c9e1dee3a83a4992b70370576ade8298 P 94130c207124419ba606b0a370ab351a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94130c207124419ba606b0a370ab351a" member_type: VOTER last_known_addr { host: "127.20.49.65" port: 40727 } } peers { permanent_uuid: "86220bf3e5b9406fa77c716ccad135e5" member_type: VOTER last_known_addr { host: "127.20.49.66" port: 40925 } } peers { permanent_uuid: "c4e57711af184378903bab4a9e856c6f" member_type: VOTER last_known_addr { host: "127.20.49.67" port: 44593 } }
I20251209 08:59:28.946580 31594 catalog_manager.cc:5654] T c9e1dee3a83a4992b70370576ade8298 P 94130c207124419ba606b0a370ab351a reported cstate change: term changed from 0 to 1, leader changed from <none> to 94130c207124419ba606b0a370ab351a (127.20.49.65). New cstate: current_term: 1 leader_uuid: "94130c207124419ba606b0a370ab351a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94130c207124419ba606b0a370ab351a" member_type: VOTER last_known_addr { host: "127.20.49.65" port: 40727 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "86220bf3e5b9406fa77c716ccad135e5" member_type: VOTER last_known_addr { host: "127.20.49.66" port: 40925 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c4e57711af184378903bab4a9e856c6f" member_type: VOTER last_known_addr { host: "127.20.49.67" port: 44593 } health_report { overall_health: UNKNOWN } } }
I20251209 08:59:28.971395 32072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:59:28.968382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38882 (local address 127.20.49.65:40727)
1209 08:59:28.968521 (+ 139us) server_negotiation.cc:207] Beginning negotiation
1209 08:59:28.968525 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1209 08:59:28.968540 (+ 15us) server_negotiation.cc:408] Connection header received
1209 08:59:28.968646 (+ 106us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1209 08:59:28.968649 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1209 08:59:28.968705 (+ 56us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1209 08:59:28.968787 (+ 82us) server_negotiation.cc:227] Negotiated authn=TOKEN
1209 08:59:28.969097 (+ 310us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:59:28.969648 (+ 551us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1209 08:59:28.970330 (+ 682us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 08:59:28.970498 (+ 168us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 08:59:28.970615 (+ 117us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1209 08:59:28.971061 (+ 446us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1209 08:59:28.971159 (+ 98us) server_negotiation.cc:1036] Waiting for connection context
1209 08:59:28.971237 (+ 78us) server_negotiation.cc:300] Negotiation successful
1209 08:59:28.971284 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20251209 08:59:28.977288 32013 raft_consensus.cc:1275] T c9e1dee3a83a4992b70370576ade8298 P c4e57711af184378903bab4a9e856c6f [term 1 FOLLOWER]: Refusing update from remote peer 94130c207124419ba606b0a370ab351a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251209 08:59:28.977308 31878 raft_consensus.cc:1275] T c9e1dee3a83a4992b70370576ade8298 P 86220bf3e5b9406fa77c716ccad135e5 [term 1 FOLLOWER]: Refusing update from remote peer 94130c207124419ba606b0a370ab351a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251209 08:59:28.978106 32088 consensus_queue.cc:1048] T c9e1dee3a83a4992b70370576ade8298 P 94130c207124419ba606b0a370ab351a [LEADER]: Connected to new peer: Peer: permanent_uuid: "86220bf3e5b9406fa77c716ccad135e5" member_type: VOTER last_known_addr { host: "127.20.49.66" port: 40925 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251209 08:59:28.978298 32083 consensus_queue.cc:1048] T c9e1dee3a83a4992b70370576ade8298 P 94130c207124419ba606b0a370ab351a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4e57711af184378903bab4a9e856c6f" member_type: VOTER last_known_addr { host: "127.20.49.67" port: 44593 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251209 08:59:28.988332 32093 mvcc.cc:204] Tried to move back new op lower bound from 7230549069722968064 to 7230549069595144192. Current Snapshot: MvccSnapshot[applied={T|T < 7230549069722968064}]
I20251209 08:59:28.994781 31594 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58650:
table { table_name: "test-table" } modify_external_catalogs: true
I20251209 08:59:28.994952 31594 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58650:
table { table_name: "test-table" } modify_external_catalogs: true
I20251209 08:59:28.997406 31594 catalog_manager.cc:5941] T 00000000000000000000000000000000 P 857fa13249e74b4ab1d1b80601ef3690: Sending DeleteTablet for 3 replicas of tablet c9e1dee3a83a4992b70370576ade8298
I20251209 08:59:28.998164 31722 tablet_service.cc:1552] Processing DeleteTablet for tablet c9e1dee3a83a4992b70370576ade8298 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-09 08:59:28 UTC) from {username='kudu', principal='kudu/127.20.49.126@KRBTEST.COM'} at 127.0.0.1:38878
I20251209 08:59:28.998246 31858 tablet_service.cc:1552] Processing DeleteTablet for tablet c9e1dee3a83a4992b70370576ade8298 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-09 08:59:28 UTC) from {username='kudu', principal='kudu/127.20.49.126@KRBTEST.COM'} at 127.0.0.1:43852
I20251209 08:59:28.998370 31994 tablet_service.cc:1552] Processing DeleteTablet for tablet c9e1dee3a83a4992b70370576ade8298 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-12-09 08:59:28 UTC) from {username='kudu', principal='kudu/127.20.49.126@KRBTEST.COM'} at 127.0.0.1:41998
I20251209 08:59:28.998893 32103 tablet_replica.cc:333] T c9e1dee3a83a4992b70370576ade8298 P c4e57711af184378903bab4a9e856c6f: stopping tablet replica
I20251209 08:59:28.998885 32102 tablet_replica.cc:333] T c9e1dee3a83a4992b70370576ade8298 P 94130c207124419ba606b0a370ab351a: stopping tablet replica
I20251209 08:59:28.999192 32103 raft_consensus.cc:2243] T c9e1dee3a83a4992b70370576ade8298 P c4e57711af184378903bab4a9e856c6f [term 1 FOLLOWER]: Raft consensus shutting down.
I20251209 08:59:28.999192 32102 raft_consensus.cc:2243] T c9e1dee3a83a4992b70370576ade8298 P 94130c207124419ba606b0a370ab351a [term 1 LEADER]: Raft consensus shutting down.
I20251209 08:59:28.999545 32103 raft_consensus.cc:2272] T c9e1dee3a83a4992b70370576ade8298 P c4e57711af184378903bab4a9e856c6f [term 1 FOLLOWER]: Raft consensus is shut down!
I20251209 08:59:28.999545 32102 raft_consensus.cc:2272] T c9e1dee3a83a4992b70370576ade8298 P 94130c207124419ba606b0a370ab351a [term 1 FOLLOWER]: Raft consensus is shut down!
I20251209 08:59:28.999933 32101 tablet_replica.cc:333] T c9e1dee3a83a4992b70370576ade8298 P 86220bf3e5b9406fa77c716ccad135e5: stopping tablet replica
I20251209 08:59:29.000211 32101 raft_consensus.cc:2243] T c9e1dee3a83a4992b70370576ade8298 P 86220bf3e5b9406fa77c716ccad135e5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251209 08:59:29.000285 20677 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task8XuDZV/build/debug/bin/kudu with pid 31659
I20251209 08:59:29.000526 32101 raft_consensus.cc:2272] T c9e1dee3a83a4992b70370576ade8298 P 86220bf3e5b9406fa77c716ccad135e5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251209 08:59:29.001187 32103 ts_tablet_manager.cc:1916] T c9e1dee3a83a4992b70370576ade8298 P c4e57711af184378903bab4a9e856c6f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251209 08:59:29.002280 32101 ts_tablet_manager.cc:1916] T c9e1dee3a83a4992b70370576ade8298 P 86220bf3e5b9406fa77c716ccad135e5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251209 08:59:29.004087 32103 ts_tablet_manager.cc:1929] T c9e1dee3a83a4992b70370576ade8298 P c4e57711af184378903bab4a9e856c6f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251209 08:59:29.004184 32103 log.cc:1199] T c9e1dee3a83a4992b70370576ade8298 P c4e57711af184378903bab4a9e856c6f: Deleting WAL directory at /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765270668420570-20677-0/minicluster-data/ts-2/wal/wals/c9e1dee3a83a4992b70370576ade8298
I20251209 08:59:29.004550 32103 ts_tablet_manager.cc:1950] T c9e1dee3a83a4992b70370576ade8298 P c4e57711af184378903bab4a9e856c6f: Deleting consensus metadata
I20251209 08:59:29.005178 32101 ts_tablet_manager.cc:1929] T c9e1dee3a83a4992b70370576ade8298 P 86220bf3e5b9406fa77c716ccad135e5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251209 08:59:29.005275 32101 log.cc:1199] T c9e1dee3a83a4992b70370576ade8298 P 86220bf3e5b9406fa77c716ccad135e5: Deleting WAL directory at /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1765270668420570-20677-0/minicluster-data/ts-1/wal/wals/c9e1dee3a83a4992b70370576ade8298
I20251209 08:59:29.005621 32101 ts_tablet_manager.cc:1950] T c9e1dee3a83a4992b70370576ade8298 P 86220bf3e5b9406fa77c716ccad135e5: Deleting consensus metadata
I20251209 08:59:29.006063 31579 catalog_manager.cc:4985] TS c4e57711af184378903bab4a9e856c6f (127.20.49.67:44593): tablet c9e1dee3a83a4992b70370576ade8298 (table test-table [id=502f8706423b482192755c2bf69634e7]) successfully deleted
I20251209 08:59:29.006794 31579 catalog_manager.cc:4985] TS 86220bf3e5b9406fa77c716ccad135e5 (127.20.49.66:40925): tablet c9e1dee3a83a4992b70370576ade8298 (table test-table [id=502f8706423b482192755c2bf69634e7]) successfully deleted
W20251209 08:59:29.012154 31579 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.20.49.65:40727 (error 108)
I20251209 08:59:29.012439 20677 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task8XuDZV/build/debug/bin/kudu with pid 31795
I20251209 08:59:29.014840 32075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 08:59:29.014443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.49.65:40727 (local address 127.0.0.1:38888)
1209 08:59:29.014642 (+ 199us) negotiation.cc:107] Waiting for socket to connect
1209 08:59:29.014724 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.49.65:40727: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20251209 08:59:29.015053 31579 catalog_manager.cc:4712] TS 94130c207124419ba606b0a370ab351a (127.20.49.65:40727): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c9e1dee3a83a4992b70370576ade8298: Network error: Client connection negotiation failed: client connection to 127.20.49.65:40727: connect: Connection refused (error 111)
I20251209 08:59:29.019200 20677 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task8XuDZV/build/debug/bin/kudu with pid 31931
I20251209 08:59:29.025525 20677 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task8XuDZV/build/debug/bin/kudu with pid 31563
2025-12-09T08:59:29Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (3351 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765270668420570-20677-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Dec 09 08:59:29 dist-test-slave-8w6c krb5kdc[32107](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Dec 09 08:59:29 dist-test-slave-8w6c krb5kdc[32107](info): set up 2 sockets
Dec 09 08:59:29 dist-test-slave-8w6c krb5kdc[32107](info): commencing operation
krb5kdc: starting...
W20251209 08:59:31.022706 20677 mini_kdc.cc:121] Time spent starting KDC: real 1.973s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765270668420570-20677-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765270668420570-20677-0/krb5kdc/test-user.keytab.
Dec 09 08:59:31 dist-test-slave-8w6c krb5kdc[32107](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765270771, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-12-09T08:59:31Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-12-09T08:59:31Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.20.49.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.20.49.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.20.49.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765270668420570-20677-0/krb5kdc/rangeradmin_127.20.49.84@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.20.49.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765270668420570-20677-0/krb5kdc/rangeradmin_127.20.49.84@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.20.49.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.20.49.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.20.49.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765270668420570-20677-0/krb5kdc/rangerlookup_127.20.49.84@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.20.49.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765270668420570-20677-0/krb5kdc/rangerlookup_127.20.49.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.20.49.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.20.49.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.20.49.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765270668420570-20677-0/krb5kdc/HTTP_127.20.49.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.20.49.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765270668420570-20677-0/krb5kdc/HTTP_127.20.49.84@KRBTEST.COM.keytab.
I20251209 08:59:31.192143 20677 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765270668420570-20677-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default "max_connections" ... 100
selecting default "shared_buffers" ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
initdb: hint: You can change this by editing pg_hba.conf or using the option -A, or --auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-task8XuDZV/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765270668420570-20677-0/minicluster-data/postgres -l logfile start
2025-12-09 08:59:33.650 UTC [32138] LOG: starting PostgreSQL 17.2 on x86_64-pc-linux-gnu, compiled by gcc (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0, 64-bit
2025-12-09 08:59:33.651 UTC [32138] LOG: listening on IPv4 address "127.20.49.84", port 51715
2025-12-09 08:59:33.654 UTC [32138] LOG: listening on Unix socket "/tmp/.s.PGSQL.51715"
2025-12-09 08:59:33.660 UTC [32143] LOG: database system was shut down at 2025-12-09 08:59:32 UTC
2025-12-09 08:59:33.665 UTC [32138] LOG: database system is ready to accept connections
I20251209 08:59:35.570497 20677 mini_postgres.cc:96] Postgres bound to 51715
2025-12-09 08:59:35.577 UTC [32149] FATAL: database "slave" does not exist
127.20.49.84:51715 - accepting connections
I20251209 08:59:35.578104 20677 mini_ranger.cc:161] Starting Ranger...
I20251209 08:59:35.596948 20677 mini_ranger.cc:84] Created miniranger Postgres user
I20251209 08:59:35.671033 20677 mini_ranger.cc:87] Created ranger Postgres database
I20251209 08:59:35.671193 20677 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765270668420570-20677-0/minicluster-data/ranger-admin
2025-12-09 08:59:35,965 [I] DB FLAVOR :POSTGRES
2025-12-09 08:59:35,965 [I] --------- Verifying Ranger DB connection ---------
2025-12-09 08:59:35,965 [I] Checking connection..
2025-12-09 08:59:35,966 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8XuDZV/build/debug/bin/postgresql.jar:/tmp/dist-test-task8XuDZV/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.49.84:51715/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2025-12-09 08:59:36,272 [I] Checking connection passed.
2025-12-09 08:59:36,273 [I] --------- Verifying version history table ---------
2025-12-09 08:59:36,273 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8XuDZV/build/debug/bin/postgresql.jar:/tmp/dist-test-task8XuDZV/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.49.84:51715/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2025-12-09 08:59:36,566 [I] Table x_db_version_h does not exist in database ranger
2025-12-09 08:59:36,567 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8XuDZV/build/debug/bin/postgresql.jar:/tmp/dist-test-task8XuDZV/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.49.84:51715/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2025-12-09 08:59:36,894 [I] Table x_db_version_h does not exist in database ranger
2025-12-09 08:59:36,894 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2025-12-09 08:59:36,894 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8XuDZV/build/debug/bin/postgresql.jar:/tmp/dist-test-task8XuDZV/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.49.84:51715/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task8XuDZV/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2025-12-09 08:59:37.199 UTC [32216] WARNING: there is no transaction in progress
2025-12-09 08:59:37,206 [I] create_dbversion_catalog.sql file imported successfully
2025-12-09 08:59:37,207 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8XuDZV/build/debug/bin/postgresql.jar:/tmp/dist-test-task8XuDZV/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.49.84:51715/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2025-12-09 08:59:37,522 [I] Table x_db_version_h already exists in database 'ranger'
2025-12-09 08:59:37,522 [I] --------- Importing Ranger Core DB Schema ---------
2025-12-09 08:59:37,522 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8XuDZV/build/debug/bin/postgresql.jar:/tmp/dist-test-task8XuDZV/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.49.84:51715/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2025-12-09 08:59:37,827 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8XuDZV/build/debug/bin/postgresql.jar:/tmp/dist-test-task8XuDZV/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.49.84:51715/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2025-12-09 08:59:38,145 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8XuDZV/build/debug/bin/postgresql.jar:/tmp/dist-test-task8XuDZV/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.49.84:51715/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-8w6c.c.gcp-upstream.internal','N') ;"
2025-12-09 08:59:38,459 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8XuDZV/build/debug/bin/postgresql.jar:/tmp/dist-test-task8XuDZV/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.49.84:51715/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2025-12-09 08:59:38,776 [I] Table x_portal_user does not exist in database ranger
2025-12-09 08:59:38,776 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8XuDZV/build/debug/bin/postgresql.jar:/tmp/dist-test-task8XuDZV/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.49.84:51715/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2025-12-09 08:59:39,080 [I] Table x_policy_ref_group does not exist in database ranger
2025-12-09 08:59:39,080 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8XuDZV/build/debug/bin/postgresql.jar:/tmp/dist-test-task8XuDZV/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.49.84:51715/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2025-12-09 08:59:39,376 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8XuDZV/build/debug/bin/postgresql.jar:/tmp/dist-test-task8XuDZV/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.49.84:51715/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2025-12-09 08:59:39,713 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2025-12-09 08:59:39,713 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8XuDZV/build/debug/bin/postgresql.jar:/tmp/dist-test-task8XuDZV/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.49.84:51715/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task8XuDZV/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2025-12-09 08:59:40.555 UTC [32333] WARNING: there is no transaction in progress
2025-12-09 08:59:40.570 UTC [32333] WARNING: there is no transaction in progress
2025-12-09 08:59:40.582 UTC [32333] WARNING: there is no transaction in progress
2025-12-09 08:59:40.592 UTC [32333] WARNING: there is no transaction in progress
2025-12-09 08:59:40.604 UTC [32333] WARNING: there is no transaction in progress
2025-12-09 08:59:40.614 UTC [32333] WARNING: there is no transaction in progress
2025-12-09 08:59:40.660 UTC [32333] WARNING: there is no transaction in progress
2025-12-09 08:59:40.665 UTC [32333] WARNING: there is no transaction in progress
2025-12-09 08:59:40.674 UTC [32333] WARNING: there is no transaction in progress
2025-12-09 08:59:40.684 UTC [32333] WARNING: there is no transaction in progress
2025-12-09 08:59:40.693 UTC [32333] WARNING: there is no transaction in progress
2025-12-09 08:59:40.702 UTC [32333] WARNING: there is no transaction in progress
2025-12-09 08:59:40.711 UTC [32333] WARNING: there is no transaction in progress
2025-12-09 08:59:40.716 UTC [32333] WARNING: there is no transaction in progress
2025-12-09 08:59:40.719 UTC [32333] WARNING: there is no transaction in progress
2025-12-09 08:59:40.723 UTC [32333] WARNING: there is no transaction in progress
2025-12-09 08:59:41,140 [I] ranger_core_db_postgres.sql file imported successfully
2025-12-09 08:59:41,141 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8XuDZV/build/debug/bin/postgresql.jar:/tmp/dist-test-task8XuDZV/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.49.84:51715/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2025-12-09 08:59:41,422 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2025-12-09 08:59:41,423 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8XuDZV/build/debug/bin/postgresql.jar:/tmp/dist-test-task8XuDZV/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.49.84:51715/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2025-12-09 08:59:41,733 [I] Table x_portal_user already exists in database 'ranger'
2025-12-09 08:59:41,733 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8XuDZV/build/debug/bin/postgresql.jar:/tmp/dist-test-task8XuDZV/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.49.84:51715/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2025-12-09 08:59:42,039 [I] Table x_policy_ref_group already exists in database 'ranger'
2025-12-09 08:59:42,039 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8XuDZV/build/debug/bin/postgresql.jar:/tmp/dist-test-task8XuDZV/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.49.84:51715/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2025-12-09 08:59:42,342 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8XuDZV/build/debug/bin/postgresql.jar:/tmp/dist-test-task8XuDZV/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.49.84:51715/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2025-12-09 08:59:42,643 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8XuDZV/build/debug/bin/postgresql.jar:/tmp/dist-test-task8XuDZV/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.49.84:51715/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-8w6c.c.gcp-upstream.internal';"
2025-12-09 08:59:42,937 [I] CORE_DB_SCHEMA import status has been updated
2025-12-09 08:59:42,937 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8XuDZV/build/debug/bin/postgresql.jar:/tmp/dist-test-task8XuDZV/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.49.84:51715/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2025-12-09 08:59:43,253 [I] DB_PATCHES have already been applied
I20251209 08:59:43.258723 20677 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765270668420570-20677-0/minicluster-data/ranger-admin:/tmp/dist-test-task8XuDZV/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-task8XuDZV/thirdparty/src/hadoop-3.4.1/*:/tmp/dist-test-task8XuDZV/build/debug/bin/postgresql.jar:/tmp/dist-test-task8XuDZV/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20251209 08:59:43.258798 20677 mini_ranger.cc:193] Using host: 127.20.49.84
I20251209 08:59:43.262298 20677 mini_ranger.cc:227] Ranger admin URL: http://127.20.49.84:49909
Dec 09, 2025 8:59:43 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Dec 09, 2025 8:59:43 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Dec 09, 2025 8:59:43 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765270668420570-20677-0/minicluster-data/ranger-admin/ews/webapp
Dec 09, 2025 8:59:43 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765270668420570-20677-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Dec 09, 2025 8:59:43 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765270668420570-20677-0/minicluster-data/ranger-admin/ews/webapp] .....
Dec 09, 2025 8:59:44 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765270668420570-20677-0/minicluster-data/ranger-admin/ews/webapp].
Dec 09, 2025 8:59:44 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20251209 08:59:45.476723 20677 mini_ranger.cc:160] Time spent starting Ranger: real 9.899s user 0.004s sys 0.008s
Dec 09, 2025 9:00:03 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Dec 09, 2025 9:00:03 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.AssetREST
Dec 09, 2025 9:00:03 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Dec 09, 2025 9:00:03 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Dec 09, 2025 9:00:03 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Dec 09, 2025 9:00:03 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Dec 09, 2025 9:00:03 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Dec 09, 2025 9:00:03 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Dec 09, 2025 9:00:03 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Dec 09, 2025 9:00:03 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Dec 09, 2025 9:00:03 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Dec 09, 2025 9:00:03 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Dec 09, 2025 9:00:03 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Dec 09, 2025 9:00:03 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Dec 09, 2025 9:00:03 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Dec 09, 2025 9:00:03 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Dec 09, 2025 9:00:03 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Dec 09, 2025 9:00:04 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20251209 09:00:05.123184 20677 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.20.49.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.20.49.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.20.49.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765270668420570-20677-0/krb5kdc/rangerkms_127.20.49.84@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.20.49.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765270668420570-20677-0/krb5kdc/rangerkms_127.20.49.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.20.49.84@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.20.49.84@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.20.49.84@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765270668420570-20677-0/krb5kdc/HTTP_127.20.49.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.20.49.84@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765270668420570-20677-0/krb5kdc/HTTP_127.20.49.84@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Dec 09 09:00:05 dist-test-slave-8w6c krb5kdc[32107](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765270805, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20251209 09:00:05.201748 20677 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20251209 09:00:05.211666 20677 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20251209 09:00:05.317698 20677 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20251209 09:00:05.317806 20677 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765270668420570-20677-0/minicluster-data/ranger-kms
I20251209 09:00:05.317852 20677 mini_ranger_kms.cc:227] Using postgres at 127.20.49.84:51715
2025-12-09 09:00:05,497 [I] DB FLAVOR :POSTGRES
2025-12-09 09:00:05,497 [I] --------- Verifying Ranger DB connection ---------
2025-12-09 09:00:05,497 [I] Checking connection
2025-12-09 09:00:05,497 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8XuDZV/build/debug/bin/postgresql.jar:/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765270668420570-20677-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.49.84:51715/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2025-12-09 09:00:05,803 [I] connection success
2025-12-09 09:00:05,803 [I] --------- Verifying Ranger DB tables ---------
2025-12-09 09:00:05,803 [I] Verifying table ranger_masterkey in database rangerkms
2025-12-09 09:00:05,803 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8XuDZV/build/debug/bin/postgresql.jar:/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765270668420570-20677-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.49.84:51715/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2025-12-09 09:00:06,107 [I] Table ranger_masterkey does not exist in database rangerkms
2025-12-09 09:00:06,108 [I] --------- Importing Ranger Core DB Schema ---------
2025-12-09 09:00:06,108 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2025-12-09 09:00:06,108 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8XuDZV/build/debug/bin/postgresql.jar:/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765270668420570-20677-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.49.84:51715/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765270668420570-20677-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2025-12-09 09:00:06,436 [I] kms_core_db_postgres.sql DB schema imported successfully
I20251209 09:00:06.901906 20677 mini_ranger_kms.cc:310] Created kms service
I20251209 09:00:07.200580 20677 mini_ranger_kms.cc:326] Created kudu user
I20251209 09:00:07.244668 20677 mini_ranger_kms.cc:343] Created rangerkms user
Dec 09 09:00:07 dist-test-slave-8w6c krb5kdc[32107](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765270805, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.20.49.84@KRBTEST.COM
I20251209 09:00:07.621469 20677 mini_ranger_kms.cc:384] Added ranger policy
I20251209 09:00:07.621708 20677 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765270668420570-20677-0/minicluster-data/ranger-kms:/tmp/dist-test-task8XuDZV/build/debug/bin/postgresql.jar:/tmp/dist-test-task8XuDZV/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-task8XuDZV/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20251209 09:00:07.621747 20677 mini_ranger_kms.cc:240] Using host: 127.20.49.84
I20251209 09:00:07.625622 20677 mini_ranger_kms.cc:276] Ranger KMS PID: 32525
I20251209 09:00:07.625748 20677 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.20.49.84:41639
Dec 09, 2025 9:00:08 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Dec 09, 2025 9:00:08 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Dec 09, 2025 9:00:08 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765270668420570-20677-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Dec 09, 2025 9:00:08 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765270668420570-20677-0/minicluster-data/ranger-kms/ews/webapp] .....
Dec 09, 2025 9:00:09 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765270668420570-20677-0/minicluster-data/ranger-kms/ews/webapp].
Dec 09, 2025 9:00:09 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Dec 09 09:00:09 dist-test-slave-8w6c krb5kdc[32107](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1765270809, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.20.49.84@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251209 09:00:09.855214 20677 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 4.653s user 0.019s sys 0.144s
I20251209 09:00:09.855358 20677 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20251209 09:00:09.855443 20677 mini_ranger_kms.cc:401] 127.20.49.84:41639/kms/v1/keys
Dec 09 09:00:10 dist-test-slave-8w6c krb5kdc[32107](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765270809, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.20.49.84@KRBTEST.COM for HTTP/127.20.49.84@KRBTEST.COM
Dec 09 09:00:10 dist-test-slave-8w6c krb5kdc[32107](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765270809, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.20.49.84@KRBTEST.COM for HTTP/127.20.49.84@KRBTEST.COM
Dec 09 09:00:12 dist-test-slave-8w6c krb5kdc[32107](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765270812, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.20.49.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.49.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.49.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765270668420570-20677-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.20.49.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765270668420570-20677-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.20.49.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.20.49.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.20.49.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765270668420570-20677-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.20.49.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765270668420570-20677-0/krb5kdc/kudu.keytab.
I20251209 09:00:12.764591 20677 external_mini_cluster.cc:1366] Running /tmp/dist-test-task8XuDZV/build/debug/bin/kudu
/tmp/dist-test-task8XuDZV/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765270668420570-20677-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765270668420570-20677-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765270668420570-20677-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765270668420570-20677-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.49.126:36427
--webserver_interface=127.20.49.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765270668420570-20677-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.49.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.20.49.84:44227
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.49.126:36427
--ranger_config_path=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765270668420570-20677-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.20.49.84:41639/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765270668420570-20677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765270668420570-20677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765270668420570-20677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251209 09:00:12.873879 32578 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251209 09:00:12.874125 32578 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251209 09:00:12.874183 32578 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251209 09:00:12.877421 32578 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251209 09:00:12.877488 32578 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251209 09:00:12.877511 32578 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251209 09:00:12.877529 32578 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251209 09:00:12.877549 32578 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251209 09:00:12.882165 32578 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.49.84:44227
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765270668420570-20677-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765270668420570-20677-0/minicluster-data/master-0/wal
--ranger_kms_url=127.20.49.84:41639/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.20.49.126:36427
--ranger_config_path=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765270668420570-20677-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765270668420570-20677-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.49.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.49.126:36427
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765270668420570-20677-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.49.126
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32578
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765270668420570-20677-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 468897755969d686fb08386f42f7835685e7953a
build type DEBUG
built by None at 09 Dec 2025 08:43:15 UTC on 24a791456cd2
build id 9426
I20251209 09:00:12.883455 32578 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251209 09:00:12.884306 32578 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251209 09:00:12.889926 32583 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251209 09:00:12.890050 32578 server_base.cc:1047] running on GCE node
W20251209 09:00:12.890162 32586 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251209 09:00:12.889925 32584 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251209 09:00:12.890857 32578 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251209 09:00:12.892002 32578 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251209 09:00:12.893186 32578 hybrid_clock.cc:648] HybridClock initialized: now 1765270812893149 us; error 50 us; skew 500 ppm
Dec 09 09:00:12 dist-test-slave-8w6c krb5kdc[32107](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765270812, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.49.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251209 09:00:12.896353 32578 init.cc:377] Logged in from keytab as kudu/127.20.49.126@KRBTEST.COM (short username kudu)
I20251209 09:00:12.897604 32578 webserver.cc:492] Webserver started at http://127.20.49.126:42595/ using document root <none> and password file <none>
I20251209 09:00:12.898278 32578 fs_manager.cc:362] Metadata directory not provided
I20251209 09:00:12.898347 32578 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251209 09:00:12.898554 32578 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
Dec 09 09:00:12 dist-test-slave-8w6c krb5kdc[32107](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765270812, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.49.126@KRBTEST.COM for HTTP/127.20.49.84@KRBTEST.COM
I20251209 09:00:12.950465 32578 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765270668420570-20677-0/minicluster-data/master-0/data/instance:
uuid: "2cccbab6e8414e1681e5b5aa746ede4b"
format_stamp: "Formatted at 2025-12-09 09:00:12 on dist-test-slave-8w6c"
server_key: "e559c7e4f3289a7ec31bd1d5fc238d81"
server_key_iv: "faa9a8cac5d1b885c64ca202c240a826"
server_key_version: "kuduclusterkey@0"
I20251209 09:00:12.951028 32578 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765270668420570-20677-0/minicluster-data/master-0/wal/instance:
uuid: "2cccbab6e8414e1681e5b5aa746ede4b"
format_stamp: "Formatted at 2025-12-09 09:00:12 on dist-test-slave-8w6c"
server_key: "e559c7e4f3289a7ec31bd1d5fc238d81"
server_key_iv: "faa9a8cac5d1b885c64ca202c240a826"
server_key_version: "kuduclusterkey@0"
I20251209 09:00:12.954599 32578 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20251209 09:00:12.971752 32593 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251209 09:00:12.972889 32578 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20251209 09:00:12.972998 32578 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765270668420570-20677-0/minicluster-data/master-0/data,/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765270668420570-20677-0/minicluster-data/master-0/wal
uuid: "2cccbab6e8414e1681e5b5aa746ede4b"
format_stamp: "Formatted at 2025-12-09 09:00:12 on dist-test-slave-8w6c"
server_key: "e559c7e4f3289a7ec31bd1d5fc238d81"
server_key_iv: "faa9a8cac5d1b885c64ca202c240a826"
server_key_version: "kuduclusterkey@0"
I20251209 09:00:12.973116 32578 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765270668420570-20677-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765270668420570-20677-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765270668420570-20677-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251209 09:00:12.984814 32578 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251209 09:00:12.991333 32578 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251209 09:00:12.991540 32578 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251209 09:00:12.999325 32578 rpc_server.cc:307] RPC server started. Bound to: 127.20.49.126:36427
I20251209 09:00:12.999308 32645 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.49.126:36427 every 8 connection(s)
I20251209 09:00:13.000363 32578 server_base.cc:1179] Dumped server information to /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765270668420570-20677-0/minicluster-data/master-0/data/info.pb
I20251209 09:00:13.001880 20677 external_mini_cluster.cc:1428] Started /tmp/dist-test-task8XuDZV/build/debug/bin/kudu as pid 32578
I20251209 09:00:13.002012 20677 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765270668420570-20677-0/minicluster-data/master-0/wal/instance
I20251209 09:00:13.002274 20677 external_mini_cluster.cc:1468] Setting key cf73edced902b054e931fbffd609a7ab
I20251209 09:00:13.003692 32646 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Dec 09 09:00:13 dist-test-slave-8w6c krb5kdc[32107](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765270812, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.49.126@KRBTEST.COM
I20251209 09:00:13.009719 32646 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2cccbab6e8414e1681e5b5aa746ede4b: Bootstrap starting.
I20251209 09:00:13.012220 32646 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2cccbab6e8414e1681e5b5aa746ede4b: Neither blocks nor log segments found. Creating new log.
I20251209 09:00:13.013134 32646 log.cc:826] T 00000000000000000000000000000000 P 2cccbab6e8414e1681e5b5aa746ede4b: Log is configured to *not* fsync() on all Append() calls
I20251209 09:00:13.014967 32649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 09:00:13.003695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34116 (local address 127.20.49.126:36427)
1209 09:00:13.004091 (+ 396us) server_negotiation.cc:207] Beginning negotiation
1209 09:00:13.004101 (+ 10us) server_negotiation.cc:400] Waiting for connection header
1209 09:00:13.004134 (+ 33us) server_negotiation.cc:408] Connection header received
1209 09:00:13.004749 (+ 615us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1209 09:00:13.004766 (+ 17us) server_negotiation.cc:462] Received NEGOTIATE request from client
1209 09:00:13.005033 (+ 267us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1209 09:00:13.005328 (+ 295us) server_negotiation.cc:227] Negotiated authn=SASL
1209 09:00:13.006271 (+ 943us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 09:00:13.007005 (+ 734us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1209 09:00:13.007674 (+ 669us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 09:00:13.007966 (+ 292us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 09:00:13.010210 (+ 2244us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1209 09:00:13.010265 (+ 55us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1209 09:00:13.010275 (+ 10us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1209 09:00:13.010305 (+ 30us) server_negotiation.cc:923] calling sasl_server_start()
1209 09:00:13.011925 (+ 1620us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1209 09:00:13.012336 (+ 411us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1209 09:00:13.012340 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1209 09:00:13.012344 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
1209 09:00:13.012407 (+ 63us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1209 09:00:13.012624 (+ 217us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1209 09:00:13.012627 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1209 09:00:13.012629 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1209 09:00:13.012944 (+ 315us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1209 09:00:13.013137 (+ 193us) server_negotiation.cc:1036] Waiting for connection context
1209 09:00:13.013739 (+ 602us) server_negotiation.cc:300] Negotiation successful
1209 09:00:13.013943 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":253,"thread_start_us":116,"threads_started":1}
I20251209 09:00:13.015290 32646 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2cccbab6e8414e1681e5b5aa746ede4b: No bootstrap required, opened a new log
I20251209 09:00:13.017951 32646 raft_consensus.cc:359] T 00000000000000000000000000000000 P 2cccbab6e8414e1681e5b5aa746ede4b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2cccbab6e8414e1681e5b5aa746ede4b" member_type: VOTER last_known_addr { host: "127.20.49.126" port: 36427 } }
I20251209 09:00:13.018116 32646 raft_consensus.cc:385] T 00000000000000000000000000000000 P 2cccbab6e8414e1681e5b5aa746ede4b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251209 09:00:13.018163 32646 raft_consensus.cc:740] T 00000000000000000000000000000000 P 2cccbab6e8414e1681e5b5aa746ede4b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2cccbab6e8414e1681e5b5aa746ede4b, State: Initialized, Role: FOLLOWER
I20251209 09:00:13.018556 32646 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2cccbab6e8414e1681e5b5aa746ede4b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2cccbab6e8414e1681e5b5aa746ede4b" member_type: VOTER last_known_addr { host: "127.20.49.126" port: 36427 } }
I20251209 09:00:13.018666 32646 raft_consensus.cc:399] T 00000000000000000000000000000000 P 2cccbab6e8414e1681e5b5aa746ede4b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251209 09:00:13.018733 32646 raft_consensus.cc:493] T 00000000000000000000000000000000 P 2cccbab6e8414e1681e5b5aa746ede4b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251209 09:00:13.018822 32646 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 2cccbab6e8414e1681e5b5aa746ede4b [term 0 FOLLOWER]: Advancing to term 1
I20251209 09:00:13.019624 32646 raft_consensus.cc:515] T 00000000000000000000000000000000 P 2cccbab6e8414e1681e5b5aa746ede4b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2cccbab6e8414e1681e5b5aa746ede4b" member_type: VOTER last_known_addr { host: "127.20.49.126" port: 36427 } }
I20251209 09:00:13.019941 32646 leader_election.cc:304] T 00000000000000000000000000000000 P 2cccbab6e8414e1681e5b5aa746ede4b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2cccbab6e8414e1681e5b5aa746ede4b; no voters:
I20251209 09:00:13.020203 32646 leader_election.cc:290] T 00000000000000000000000000000000 P 2cccbab6e8414e1681e5b5aa746ede4b [CANDIDATE]: Term 1 election: Requested vote from peers
I20251209 09:00:13.020315 32651 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 2cccbab6e8414e1681e5b5aa746ede4b [term 1 FOLLOWER]: Leader election won for term 1
I20251209 09:00:13.020531 32651 raft_consensus.cc:697] T 00000000000000000000000000000000 P 2cccbab6e8414e1681e5b5aa746ede4b [term 1 LEADER]: Becoming Leader. State: Replica: 2cccbab6e8414e1681e5b5aa746ede4b, State: Running, Role: LEADER
I20251209 09:00:13.020857 32651 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2cccbab6e8414e1681e5b5aa746ede4b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2cccbab6e8414e1681e5b5aa746ede4b" member_type: VOTER last_known_addr { host: "127.20.49.126" port: 36427 } }
I20251209 09:00:13.021291 32646 sys_catalog.cc:565] T 00000000000000000000000000000000 P 2cccbab6e8414e1681e5b5aa746ede4b [sys.catalog]: configured and running, proceeding with master startup.
I20251209 09:00:13.022013 32651 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2cccbab6e8414e1681e5b5aa746ede4b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2cccbab6e8414e1681e5b5aa746ede4b. Latest consensus state: current_term: 1 leader_uuid: "2cccbab6e8414e1681e5b5aa746ede4b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2cccbab6e8414e1681e5b5aa746ede4b" member_type: VOTER last_known_addr { host: "127.20.49.126" port: 36427 } } }
I20251209 09:00:13.022202 32651 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2cccbab6e8414e1681e5b5aa746ede4b [sys.catalog]: This master's current role is: LEADER
I20251209 09:00:13.022452 32653 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2cccbab6e8414e1681e5b5aa746ede4b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2cccbab6e8414e1681e5b5aa746ede4b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2cccbab6e8414e1681e5b5aa746ede4b" member_type: VOTER last_known_addr { host: "127.20.49.126" port: 36427 } } }
I20251209 09:00:13.022617 32653 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2cccbab6e8414e1681e5b5aa746ede4b [sys.catalog]: This master's current role is: LEADER
I20251209 09:00:13.023485 32665 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251209 09:00:13.023649 32646 ranger_client.cc:318] Using new properties file: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765270668420570-20677-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20251209 09:00:13.030228 32665 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251209 09:00:13.041186 32665 catalog_manager.cc:1357] Generated new cluster ID: 4650f05026714bbda467cb8b5494f9fb
I20251209 09:00:13.041334 32665 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251209 09:00:13.059538 32665 catalog_manager.cc:1380] Generated new certificate authority record
I20251209 09:00:13.060958 32665 catalog_manager.cc:1514] Loading token signing keys...
I20251209 09:00:13.077826 32665 catalog_manager.cc:6027] T 00000000000000000000000000000000 P 2cccbab6e8414e1681e5b5aa746ede4b: Generated new TSK 0
I20251209 09:00:13.078807 32665 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.20.49.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.49.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.49.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765270668420570-20677-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.20.49.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765270668420570-20677-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.20.49.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.20.49.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.20.49.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765270668420570-20677-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.20.49.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765270668420570-20677-0/krb5kdc/kudu.keytab.
I20251209 09:00:13.862532 20677 external_mini_cluster.cc:1366] Running /tmp/dist-test-task8XuDZV/build/debug/bin/kudu
/tmp/dist-test-task8XuDZV/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765270668420570-20677-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765270668420570-20677-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765270668420570-20677-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765270668420570-20677-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.20.49.84:41639/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.49.65:0
--local_ip_for_outbound_sockets=127.20.49.65
--webserver_interface=127.20.49.65
--webserver_port=0
--tserver_master_addrs=127.20.49.126:36427
--keytab_file=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765270668420570-20677-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.49.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.20.49.84:44227
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765270668420570-20677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765270668420570-20677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765270668420570-20677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251209 09:00:13.992934 32692 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251209 09:00:13.993232 32692 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251209 09:00:13.993314 32692 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251209 09:00:13.996745 32692 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251209 09:00:13.996812 32692 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251209 09:00:13.996908 32692 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.49.65
I20251209 09:00:14.001159 32692 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.49.84:44227
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765270668420570-20677-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765270668420570-20677-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.20.49.84:41639/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765270668420570-20677-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.49.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.49.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765270668420570-20677-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.49.65
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.20.49.126:36427
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32692
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.49.65
--log_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765270668420570-20677-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 468897755969d686fb08386f42f7835685e7953a
build type DEBUG
built by None at 09 Dec 2025 08:43:15 UTC on 24a791456cd2
build id 9426
I20251209 09:00:14.002254 32692 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251209 09:00:14.003093 32692 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251209 09:00:14.009577 32699 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251209 09:00:14.009666 32700 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251209 09:00:14.009572 32702 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251209 09:00:14.010856 32692 server_base.cc:1047] running on GCE node
I20251209 09:00:14.011270 32692 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251209 09:00:14.011929 32692 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251209 09:00:14.013132 32692 hybrid_clock.cc:648] HybridClock initialized: now 1765270814013120 us; error 35 us; skew 500 ppm
Dec 09 09:00:14 dist-test-slave-8w6c krb5kdc[32107](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765270814, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.49.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251209 09:00:14.016606 32692 init.cc:377] Logged in from keytab as kudu/127.20.49.65@KRBTEST.COM (short username kudu)
I20251209 09:00:14.018025 32692 webserver.cc:492] Webserver started at http://127.20.49.65:33519/ using document root <none> and password file <none>
I20251209 09:00:14.018800 32692 fs_manager.cc:362] Metadata directory not provided
I20251209 09:00:14.018874 32692 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251209 09:00:14.019104 32692 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
Dec 09 09:00:14 dist-test-slave-8w6c krb5kdc[32107](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765270814, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.49.65@KRBTEST.COM for HTTP/127.20.49.84@KRBTEST.COM
I20251209 09:00:14.035708 32692 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765270668420570-20677-0/minicluster-data/ts-0/data/instance:
uuid: "1b311d5d567d47a4814386b648d8721f"
format_stamp: "Formatted at 2025-12-09 09:00:14 on dist-test-slave-8w6c"
server_key: "88454020dcefce9eeb20a6ce5cc75d3d"
server_key_iv: "15aa98a579981cbab69f17d61872d9ad"
server_key_version: "kuduclusterkey@0"
I20251209 09:00:14.038167 32692 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765270668420570-20677-0/minicluster-data/ts-0/wal/instance:
uuid: "1b311d5d567d47a4814386b648d8721f"
format_stamp: "Formatted at 2025-12-09 09:00:14 on dist-test-slave-8w6c"
server_key: "88454020dcefce9eeb20a6ce5cc75d3d"
server_key_iv: "15aa98a579981cbab69f17d61872d9ad"
server_key_version: "kuduclusterkey@0"
I20251209 09:00:14.047161 32692 fs_manager.cc:696] Time spent creating directory manager: real 0.009s user 0.000s sys 0.003s
I20251209 09:00:14.069991 32709 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251209 09:00:14.071513 32692 fs_manager.cc:730] Time spent opening block manager: real 0.010s user 0.001s sys 0.000s
I20251209 09:00:14.071692 32692 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765270668420570-20677-0/minicluster-data/ts-0/data,/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765270668420570-20677-0/minicluster-data/ts-0/wal
uuid: "1b311d5d567d47a4814386b648d8721f"
format_stamp: "Formatted at 2025-12-09 09:00:14 on dist-test-slave-8w6c"
server_key: "88454020dcefce9eeb20a6ce5cc75d3d"
server_key_iv: "15aa98a579981cbab69f17d61872d9ad"
server_key_version: "kuduclusterkey@0"
I20251209 09:00:14.071834 32692 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765270668420570-20677-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765270668420570-20677-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765270668420570-20677-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251209 09:00:14.093739 32692 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251209 09:00:14.097295 32692 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251209 09:00:14.097499 32692 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251209 09:00:14.098141 32692 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251209 09:00:14.099094 32692 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251209 09:00:14.099155 32692 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251209 09:00:14.099217 32692 ts_tablet_manager.cc:616] Registered 0 tablets
I20251209 09:00:14.099241 32692 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251209 09:00:14.113284 32692 rpc_server.cc:307] RPC server started. Bound to: 127.20.49.65:44853
I20251209 09:00:14.113845 356 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.49.65:44853 every 8 connection(s)
I20251209 09:00:14.114538 32692 server_base.cc:1179] Dumped server information to /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765270668420570-20677-0/minicluster-data/ts-0/data/info.pb
I20251209 09:00:14.123128 20677 external_mini_cluster.cc:1428] Started /tmp/dist-test-task8XuDZV/build/debug/bin/kudu as pid 32692
I20251209 09:00:14.123266 20677 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765270668420570-20677-0/minicluster-data/ts-0/wal/instance
I20251209 09:00:14.123543 20677 external_mini_cluster.cc:1468] Setting key a26f6a0af6c5e4b4c10a8ce476ed7717
Dec 09 09:00:14 dist-test-slave-8w6c krb5kdc[32107](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765270814, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.49.65@KRBTEST.COM for kudu/127.20.49.126@KRBTEST.COM
I20251209 09:00:14.132028 359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 09:00:14.116874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.49.126:36427 (local address 127.20.49.65:40173)
1209 09:00:14.118216 (+ 1342us) negotiation.cc:107] Waiting for socket to connect
1209 09:00:14.118263 (+ 47us) client_negotiation.cc:174] Beginning negotiation
1209 09:00:14.119262 (+ 999us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1209 09:00:14.120090 (+ 828us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1209 09:00:14.120101 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
1209 09:00:14.120467 (+ 366us) client_negotiation.cc:189] Negotiated authn=SASL
1209 09:00:14.121259 (+ 792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 09:00:14.121277 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 09:00:14.122525 (+ 1248us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1209 09:00:14.122529 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1209 09:00:14.123131 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1209 09:00:14.123142 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1209 09:00:14.123434 (+ 292us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 09:00:14.124193 (+ 759us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1209 09:00:14.124228 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
1209 09:00:14.126595 (+ 2367us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1209 09:00:14.129021 (+ 2426us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1209 09:00:14.129027 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1209 09:00:14.129041 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
1209 09:00:14.129364 (+ 323us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1209 09:00:14.129655 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1209 09:00:14.129659 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1209 09:00:14.129661 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1209 09:00:14.129836 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1209 09:00:14.130291 (+ 455us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1209 09:00:14.130298 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1209 09:00:14.130624 (+ 326us) client_negotiation.cc:769] Sending connection context
1209 09:00:14.130863 (+ 239us) client_negotiation.cc:240] Negotiation successful
1209 09:00:14.131143 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1122,"thread_start_us":119,"threads_started":1}
I20251209 09:00:14.132858 360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 09:00:14.118039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.49.65:40173 (local address 127.20.49.126:36427)
1209 09:00:14.118293 (+ 254us) server_negotiation.cc:207] Beginning negotiation
1209 09:00:14.118298 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1209 09:00:14.118355 (+ 57us) server_negotiation.cc:408] Connection header received
1209 09:00:14.119538 (+ 1183us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1209 09:00:14.119543 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1209 09:00:14.119599 (+ 56us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1209 09:00:14.119702 (+ 103us) server_negotiation.cc:227] Negotiated authn=SASL
1209 09:00:14.121445 (+ 1743us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 09:00:14.122363 (+ 918us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1209 09:00:14.124132 (+ 1769us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1209 09:00:14.124327 (+ 195us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1209 09:00:14.126775 (+ 2448us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1209 09:00:14.126795 (+ 20us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1209 09:00:14.126798 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1209 09:00:14.126826 (+ 28us) server_negotiation.cc:923] calling sasl_server_start()
1209 09:00:14.128874 (+ 2048us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1209 09:00:14.129499 (+ 625us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1209 09:00:14.129503 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1209 09:00:14.129505 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1209 09:00:14.129559 (+ 54us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1209 09:00:14.129948 (+ 389us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1209 09:00:14.129951 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1209 09:00:14.129953 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1209 09:00:14.130143 (+ 190us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1209 09:00:14.130236 (+ 93us) server_negotiation.cc:1036] Waiting for connection context
1209 09:00:14.132556 (+ 2320us) server_negotiation.cc:300] Negotiation successful
1209 09:00:14.132693 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":178,"thread_start_us":80,"threads_started":1}
I20251209 09:00:14.133945 357 heartbeater.cc:344] Connected to a master server at 127.20.49.126:36427
I20251209 09:00:14.134266 357 heartbeater.cc:461] Registering TS with master...
I20251209 09:00:14.134960 357 heartbeater.cc:507] Master 127.20.49.126:36427 requested a full tablet report, sending...
I20251209 09:00:14.136777 32610 ts_manager.cc:194] Registered new tserver with Master: 1b311d5d567d47a4814386b648d8721f (127.20.49.65:44853)
I20251209 09:00:14.138451 32610 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.49.65@KRBTEST.COM'} at 127.20.49.65:40173
WARNING: no policy specified for kudu/127.20.49.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.49.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.49.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765270668420570-20677-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.20.49.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765270668420570-20677-0/krb5kdc/kudu.keytab.
Exception in thread "main" org.apache.kudu.subprocess.KuduSubprocessException: Failed to login with Kudu principal/keytab
at org.apache.kudu.subprocess.ranger.authorization.RangerKuduAuthorizer.init(RangerKuduAuthorizer.java:87)
at org.apache.kudu.subprocess.ranger.RangerProtocolHandler.<init>(RangerProtocolHandler.java:45)
at org.apache.kudu.subprocess.ranger.RangerSubprocessMain.main(RangerSubprocessMain.java:39)
Caused by: org.apache.hadoop.security.KerberosAuthException: failure to login: for principal: kudu/127.20.49.126 from keytab /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765270668420570-20677-0/krb5kdc/kudu.keytab javax.security.auth.login.LoginException: Unable to obtain password from user
at org.apache.hadoop.security.UserGroupInformation.doSubjectLogin(UserGroupInformation.java:2064)
at org.apache.hadoop.security.UserGroupInformation.loginUserFromKeytabAndReturnUGI(UserGroupInformation.java:1398)
at org.apache.hadoop.security.UserGroupInformation.loginUserFromKeytab(UserGroupInformation.java:1136)
at org.apache.kudu.subprocess.ranger.authorization.RangerKuduAuthorizer.init(RangerKuduAuthorizer.java:85)
... 2 more
Caused by: javax.security.auth.login.LoginException: Unable to obtain password from user
at com.sun.security.auth.module.Krb5LoginModule.promptForPass(Krb5LoginModule.java:901)
at com.sun.security.auth.module.Krb5LoginModule.attemptAuthentication(Krb5LoginModule.java:764)
at com.sun.security.auth.module.Krb5LoginModule.login(Krb5LoginModule.java:618)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at javax.security.auth.login.LoginContext.invoke(LoginContext.java:755)
at javax.security.auth.login.LoginContext.access$000(LoginContext.java:195)
at javax.security.auth.login.LoginContext$4.run(LoginContext.java:682)
at javax.security.auth.login.LoginContext$4.run(LoginContext.java:680)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.login.LoginContext.invokePriv(LoginContext.java:680)
at javax.security.auth.login.LoginContext.login(LoginContext.java:587)
at org.apache.hadoop.security.UserGroupInformation$HadoopLoginContext.login(UserGroupInformation.java:2148)
at org.apache.hadoop.security.UserGroupInformation.doSubjectLogin(UserGroupInformation.java:2053)
... 5 more
WARNING: no policy specified for HTTP/127.20.49.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.20.49.66@KRBTEST.COM" created.
I20251209 09:00:14.178215 32684 server.cc:273] Received an EOF from the subprocess
F20251209 09:00:14.178906 32670 server.cc:406] The subprocess has exited with status 1
*** Check failure stack trace: ***
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.20.49.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765270668420570-20677-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.20.49.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765270668420570-20677-0/krb5kdc/kudu.keytab.
@ 0x7fd23cc66dcd google::LogMessage::Fail() at ??:0
I20251209 09:00:14.182456 20677 external_mini_cluster.cc:1366] Running /tmp/dist-test-task8XuDZV/build/debug/bin/kudu
/tmp/dist-test-task8XuDZV/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765270668420570-20677-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765270668420570-20677-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765270668420570-20677-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765270668420570-20677-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.20.49.84:41639/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.49.66:0
--local_ip_for_outbound_sockets=127.20.49.66
--webserver_interface=127.20.49.66
--webserver_port=0
--tserver_master_addrs=127.20.49.126:36427
--keytab_file=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765270668420570-20677-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.49.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.20.49.84:44227
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765270668420570-20677-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765270668420570-20677-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765270668420570-20677-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
@ 0x7fd23cc6ab93 google::LogMessage::SendToLog() at ??:0
@ 0x7fd23cc667cc google::LogMessage::Flush() at ??:0
@ 0x7fd23cc67f59 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fd23e91d2d7 kudu::subprocess::SubprocessServer::ExitCheckerThread() at ??:0
@ 0x7fd23e91a0d9 _ZZN4kudu10subprocess16SubprocessServer4InitEvENKUlvE0_clEv at ??:0
@ 0x7fd23e91e005 _ZNSt17_Function_handlerIFvvEZN4kudu10subprocess16SubprocessServer4InitEvEUlvE0_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x55eff1950a0a std::function<>::operator()() at ??:0
@ 0x7fd23e0c07dc kudu::Thread::SuperviseThread() at ??:0
@ 0x7fd23e9596db start_thread at ??:0
@ 0x7fd23be1d71f clone at ??:0
W20251209 09:00:14.313308 367 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251209 09:00:14.313531 367 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251209 09:00:14.313593 367 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251209 09:00:14.316864 367 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251209 09:00:14.316917 367 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251209 09:00:14.317028 367 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.49.66
I20251209 09:00:14.321146 367 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.49.84:44227
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765270668420570-20677-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765270668420570-20677-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.20.49.84:41639/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765270668420570-20677-0/krb5kdc/kudu.keytab
--principal=kudu/127.20.49.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.49.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765270668420570-20677-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.49.66
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.20.49.126:36427
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.367
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.49.66
--log_dir=/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765270668420570-20677-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 468897755969d686fb08386f42f7835685e7953a
build type DEBUG
built by None at 09 Dec 2025 08:43:15 UTC on 24a791456cd2
build id 9426
I20251209 09:00:14.322233 367 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251209 09:00:14.323048 367 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251209 09:00:14.329362 382 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251209 09:00:14.329375 385 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251209 09:00:14.329365 383 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251209 09:00:14.330068 367 server_base.cc:1047] running on GCE node
I20251209 09:00:14.330510 367 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251209 09:00:14.331172 367 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251209 09:00:14.332365 367 hybrid_clock.cc:648] HybridClock initialized: now 1765270814332342 us; error 40 us; skew 500 ppm
Dec 09 09:00:14 dist-test-slave-8w6c krb5kdc[32107](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765270814, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.49.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251209 09:00:14.335877 367 init.cc:377] Logged in from keytab as kudu/127.20.49.66@KRBTEST.COM (short username kudu)
I20251209 09:00:14.337137 367 webserver.cc:492] Webserver started at http://127.20.49.66:34413/ using document root <none> and password file <none>
I20251209 09:00:14.337863 367 fs_manager.cc:362] Metadata directory not provided
I20251209 09:00:14.337934 367 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251209 09:00:14.338158 367 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
Dec 09 09:00:14 dist-test-slave-8w6c krb5kdc[32107](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1765270814, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.49.66@KRBTEST.COM for HTTP/127.20.49.84@KRBTEST.COM
I20251209 09:00:14.352984 367 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765270668420570-20677-0/minicluster-data/ts-1/data/instance:
uuid: "a329397bf1b0491c9c1c7710d3828eca"
format_stamp: "Formatted at 2025-12-09 09:00:14 on dist-test-slave-8w6c"
server_key: "59b3fc9b0598e84e554b5431aa86418e"
server_key_iv: "a327aa353af8945755172e2ca24a4053"
server_key_version: "kuduclusterkey@0"
I20251209 09:00:14.353506 367 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765270668420570-20677-0/minicluster-data/ts-1/wal/instance:
uuid: "a329397bf1b0491c9c1c7710d3828eca"
format_stamp: "Formatted at 2025-12-09 09:00:14 on dist-test-slave-8w6c"
server_key: "59b3fc9b0598e84e554b5431aa86418e"
server_key_iv: "a327aa353af8945755172e2ca24a4053"
server_key_version: "kuduclusterkey@0"
I20251209 09:00:14.358729 367 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.001s sys 0.001s
I20251209 09:00:14.372808 392 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251209 09:00:14.374070 367 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20251209 09:00:14.374197 367 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765270668420570-20677-0/minicluster-data/ts-1/data,/tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765270668420570-20677-0/minicluster-data/ts-1/wal
uuid: "a329397bf1b0491c9c1c7710d3828eca"
format_stamp: "Formatted at 2025-12-09 09:00:14 on dist-test-slave-8w6c"
server_key: "59b3fc9b0598e84e554b5431aa86418e"
server_key_iv: "a327aa353af8945755172e2ca24a4053"
server_key_version: "kuduclusterkey@0"
I20251209 09:00:14.374315 367 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765270668420570-20677-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765270668420570-20677-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765270668420570-20677-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251209 09:00:14.408556 367 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251209 09:00:14.411892 367 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251209 09:00:14.412081 367 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251209 09:00:14.412627 367 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251209 09:00:14.413513 367 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251209 09:00:14.413565 367 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251209 09:00:14.413627 367 ts_tablet_manager.cc:616] Registered 0 tablets
I20251209 09:00:14.413652 367 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251209 09:00:14.423354 367 rpc_server.cc:307] RPC server started. Bound to: 127.20.49.66:37091
I20251209 09:00:14.423652 505 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.49.66:37091 every 8 connection(s)
I20251209 09:00:14.424326 367 server_base.cc:1179] Dumped server information to /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765270668420570-20677-0/minicluster-data/ts-1/data/info.pb
I20251209 09:00:14.427541 508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 09:00:14.426390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.49.126:36427 (local address 127.20.49.66:56345)
1209 09:00:14.426822 (+ 432us) negotiation.cc:107] Waiting for socket to connect
1209 09:00:14.426928 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.49.126:36427: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":94,"threads_started":1}
W20251209 09:00:14.428119 506 heartbeater.cc:646] Failed to heartbeat to 127.20.49.126:36427 (0 consecutive failures): Network error: Failed to ping master at 127.20.49.126:36427: Client connection negotiation failed: client connection to 127.20.49.126:36427: connect: Connection refused (error 111)
I20251209 09:00:14.428685 508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 09:00:14.428461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.49.126:36427 (local address 127.20.49.66:45971)
1209 09:00:14.428578 (+ 117us) negotiation.cc:107] Waiting for socket to connect
1209 09:00:14.428623 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.49.126:36427: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20251209 09:00:14.429239 508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 09:00:14.429062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.49.126:36427 (local address 127.20.49.66:44895)
1209 09:00:14.429153 (+ 91us) negotiation.cc:107] Waiting for socket to connect
1209 09:00:14.429191 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.49.126:36427: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20251209 09:00:14.429389 506 heartbeater.cc:412] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20251209 09:00:14.429932 20677 external_mini_cluster.cc:1428] Started /tmp/dist-test-task8XuDZV/build/debug/bin/kudu as pid 367
I20251209 09:00:14.430038 20677 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task8XuDZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1765270668420570-20677-0/minicluster-data/ts-1/wal/instance
I20251209 09:00:14.430246 20677 external_mini_cluster.cc:1468] Setting key 7399d6b12fb2c2647f617e1b80ac6ba4
I20251209 09:00:15.141665 357 heartbeater.cc:499] Master 127.20.49.126:36427 was elected leader, sending a full tablet report...
I20251209 09:00:15.142665 510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 09:00:15.142226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.49.126:36427 (local address 127.20.49.65:33991)
1209 09:00:15.142488 (+ 262us) negotiation.cc:107] Waiting for socket to connect
1209 09:00:15.142559 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.49.126:36427: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":74,"threads_started":1}
W20251209 09:00:15.142937 357 heartbeater.cc:646] Failed to heartbeat to 127.20.49.126:36427 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.20.49.126:36427: connect: Connection refused (error 111)
I20251209 09:00:15.430589 511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 09:00:15.430156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.49.126:36427 (local address 127.20.49.66:44407)
1209 09:00:15.430423 (+ 267us) negotiation.cc:107] Waiting for socket to connect
1209 09:00:15.430482 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.49.126:36427: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":74,"threads_started":1}
I20251209 09:00:16.144013 512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 09:00:16.143575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.49.126:36427 (local address 127.20.49.65:52059)
1209 09:00:16.143847 (+ 272us) negotiation.cc:107] Waiting for socket to connect
1209 09:00:16.143910 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.49.126:36427: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":81,"threads_started":1}
I20251209 09:00:16.431881 513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 09:00:16.431401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.49.126:36427 (local address 127.20.49.66:37369)
1209 09:00:16.431695 (+ 294us) negotiation.cc:107] Waiting for socket to connect
1209 09:00:16.431760 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.49.126:36427: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":96,"threads_started":1}
I20251209 09:00:17.145350 514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 09:00:17.144868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.49.126:36427 (local address 127.20.49.65:52043)
1209 09:00:17.145145 (+ 277us) negotiation.cc:107] Waiting for socket to connect
1209 09:00:17.145210 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.49.126:36427: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":86,"threads_started":1}
I20251209 09:00:17.433115 515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 09:00:17.432681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.49.126:36427 (local address 127.20.49.66:42053)
1209 09:00:17.432957 (+ 276us) negotiation.cc:107] Waiting for socket to connect
1209 09:00:17.433016 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.49.126:36427: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":85,"threads_started":1}
I20251209 09:00:18.146641 516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 09:00:18.146174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.49.126:36427 (local address 127.20.49.65:53803)
1209 09:00:18.146466 (+ 292us) negotiation.cc:107] Waiting for socket to connect
1209 09:00:18.146541 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.49.126:36427: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":80,"threads_started":1}
I20251209 09:00:18.434298 517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 09:00:18.433885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.49.126:36427 (local address 127.20.49.66:45321)
1209 09:00:18.434150 (+ 265us) negotiation.cc:107] Waiting for socket to connect
1209 09:00:18.434202 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.49.126:36427: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":83,"threads_started":1}
I20251209 09:00:19.147939 518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 09:00:19.147481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.49.126:36427 (local address 127.20.49.65:50139)
1209 09:00:19.147781 (+ 300us) negotiation.cc:107] Waiting for socket to connect
1209 09:00:19.147842 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.49.126:36427: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":83,"threads_started":1}
I20251209 09:00:19.435611 519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1209 09:00:19.435137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.49.126:36427 (local address 127.20.49.66:36019)
1209 09:00:19.435417 (+ 280us) negotiation.cc:107] Waiting for socket to connect
1209 09:00:19.435500 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.49.126:36427: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":87,"threads_started":1}
F20251209 09:00:19.472028 20677 external_mini_cluster.cc:650] Check failed: leader_master() != nullptr Must have started at least 1 master before adding tablet servers
*** Check failure stack trace: ***
*** Aborted at 1765270819 (unix time) try "date -d @1765270819" if you are using GNU date ***
PC: @ 0x0 (unknown)
*** SIGABRT (@0x3e8000050c5) received by PID 20677 (TID 0x7f0660756d80) from PID 20677; stack trace: ***
@ 0x7f066c3f2980 (unknown) at ??:0
@ 0x7f0669a29fb7 gsignal at ??:0
@ 0x7f0669a2b921 abort at ??:0
@ 0x7f066ac97dcd google::LogMessage::Fail() at ??:0
@ 0x7f066ac9bb93 google::LogMessage::SendToLog() at ??:0
@ 0x7f066ac977cc google::LogMessage::Flush() at ??:0
@ 0x7f066ac98f59 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f066da23cab kudu::cluster::ExternalMiniCluster::AddTabletServer() at ??:0
@ 0x7f066da209c6 kudu::cluster::ExternalMiniCluster::Start() at ??:0
@ 0x55706fc9b92f kudu::SecurityITest::StartCluster() at ??:0
@ 0x55706fc83792 kudu::SecurityITest_TestEncryptionWithKMSIntegration_Test::TestBody() at ??:0
@ 0x7f066bca70ed testing::internal::HandleExceptionsInMethodIfSupported<>() at ??:0
@ 0x7f066bc9bbdb testing::Test::Run() at ??:0
@ 0x7f066bc9bd9d testing::TestInfo::Run() at ??:0
@ 0x7f066bc9c377 testing::TestSuite::Run() at ??:0
@ 0x7f066bc9c77c testing::internal::UnitTestImpl::RunAllTests() at ??:0
@ 0x7f066bca760d testing::internal::HandleExceptionsInMethodIfSupported<>() at ??:0
@ 0x7f066bc9be63 testing::UnitTest::Run() at ??:0
@ 0x55706fcd3c7e RUN_ALL_TESTS() at ??:0
@ 0x55706fcd2e6b main at ??:0
@ 0x7f0669a0cbf7 __libc_start_main at ??:0
@ 0x55706fc6b06a _start at ??:0
2025-12-09 09:00:19.677 UTC [32556] FATAL: terminating connection due to unexpected postmaster exit
2025-12-09 09:00:19.678 UTC [32452] FATAL: terminating connection due to unexpected postmaster exit
2025-12-09 09:00:19.680 UTC [32450] FATAL: terminating connection due to unexpected postmaster exit
2025-12-09 09:00:19.681 UTC [32449] FATAL: terminating connection due to unexpected postmaster exit
2025-12-09 09:00:19.681 UTC [32453] FATAL: terminating connection due to unexpected postmaster exit
2025-12-09 09:00:19.681 UTC [32448] FATAL: terminating connection due to unexpected postmaster exit